You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Keys Botzum <kb...@maprtech.com> on 2012/03/30 20:07:49 UTC

Accumulo on MapR Continued

I've been building on the work of Todd Stavish to put Accumulo onto MapR:

http://t.co/RJJ8Ht4B
http://mail-archives.apache.org/mod_mbox/accumulo-user/201203.mbox/%3CCADKA3WiSRDrTw_vKcB9zrP0oyq01jQfZOBGQDRnujEjQ3kHd2w@mail.gmail.com%3E 

I've installed Accumulo on a 5 node MapR cluster (real hardware) basing my work on what Todd posted earlier. I'm using Accumulo 1.4.0 RC3 (downloaded from http://people.apache.org/~kturner/1.4.0rc3/) which contains the file system fix (https://issues.apache.org/jira/browse/ACCUMULO-476).

The install went fine. I can start the Accumulo shell and create tables and such without difficulty. 

I decided then to do a more complete test using the Accumulo built in tests found under ACCUMULO_HOME/test and this is where I started encountering problems. It may be that I just don't understand how to configure and run the tests properly or perhaps something deeper. Any help would be greatly appreciated.

I decided to run the tests under the auto directory as those seemed promising (was that a good idea?). As instructed in the README I can start tests in local mode (no MapReduce job) using ./run.py -t testname or leaving out the -t to run all tests. When I run the tests (all at once or individually) while the majority succeed a small subset consistently fail. They are:

FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
FAIL: runTest (simple.bulkSplitOptimization.BulkSplitOptimizationTest)
FAIL: runTest (simple.deleterows.DeleteRowsSplitTest)
FAIL: runTest (simple.dynamic.DynamicClassloader)
FAIL: runTest (simple.largeRow.LargeRowTest)
FAIL: runTest (simple.merge.MergeTest)
FAIL: runTest (simple.zooCacheTest.ZooCacheTest)

Some fail because of timeouts, other fail with assertion errors. Rather than dumping every issue into the forum, I thought I'd just start by focusing on the first test - simple.batchScanSplit.BatchScanSplitTest. When I run this test by itself, this is the output:

./run.py -t ulkSplitOptimizationTest -d -v 10
…
DEBUG:test.auto:out:        1,010 records written |
DEBUG:test.auto:out:   72,142 records/sec |       92,920 bytes written | 6,637,142 bytes/sec |
DEBUG:test.auto:out:  0.014 secs
FAIL
======================================================================
FAIL: runTest (simple.bulkSplitOptimization.BulkSplitOptimizationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 56, in runTest
    self.waitForStop(handle, self.maxRuntime)
  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 371, in waitForStop
    self.fail("Process failed to finish in %s seconds" % secs)
AssertionError: Process failed to finish in 200 seconds


======================================================================
FAIL: runTest (simple.bulkSplitOptimization.BulkSplitOptimizationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 56, in runTest
    self.waitForStop(handle, self.maxRuntime)
  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 371, in waitForStop
    self.fail("Process failed to finish in %s seconds" % secs)
AssertionError: Process failed to finish in 200 seconds

----------------------------------------------------------------------
Ran 1 test in 208.163s

FAILED (failures=1)

Since I set the test to preserve the logs, I look at the logs and found only one thing that jumps out at me as the problem. I see this message repeated over and over while the test is running:

30 11:00:14,587 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:45335 !0 2 entries in 0.00 secs, nbTimes = [1 1 1.00 1] 
30 11:00:15,092 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:45335 !0 2 entries in 0.00 secs, nbTimes = [0 0 0.00 1] 
30 11:00:15,499 [file.FileUtil] DEBUG: Too many indexes (100) to open at once for null null, reducing in tmpDir = /user/mapr/accumulo-SE-test-04-17598/tmp/idxReduce_1785031962
30 11:00:15,552 [tabletserver.TabletServer] DEBUG: gc PS Scavenge=4.22(+0.00) secs PS MarkSweep=0.18(+0.00) secs freemem=74,816,456(+5,884,960) totalmem=110,100,480
30 11:00:15,596 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:45335 !0 2 entries in 0.00 secs, nbTimes = [1 1 1.00 1] 
30 11:00:15,643 [file.FileUtil] DEBUG: Finished reducing indexes for null null in   0.14 secs
30 11:00:15,654 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.01 secs.

30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey Seeking beyond EOF, filelen 733, wantpos 733


I suspect the last line is the cause but I do not understand what it means or how to investigate this further. Of course I can provide any additional information you need but I didn't want to make this post even longer at first.

Help is appreciated.

Thanks!
Keys

p.s. As a sanity test I installed Accumulo a 2nd time on a MapR M5 VM with one CPU and the same tests fail.
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com




Re: Accumulo on MapR - BatchScanSplitTest

Posted by Keys Botzum <kb...@maprtech.com>.
Eric,

You nailed it. That was the issue. Not only does the test complete, it now completes much more quickly. Huge improvement. However, in my defense, Accumulo was configured to use those arguments in conf/accumulo-env.sh. The issue seems to be that the test driver didn't honor those settings. I think I fixed this via a hack to TestUtils.py. I edited this section (new stuff in bold/red):


    def runOn(self, host, cmd, **opts):
        cmd = map(str, cmd)
        log.debug('%s: %s', host, ' '.join(cmd))
        if host == 'localhost':
            os.environ['ACCUMULO_TSERVER_OPTS']='-Xmx700m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75'
            os.environ['ACCUMULO_GENERAL_OPTS']='-Dorg.apache.accumulo.config.file=%s ' % SITE
            os.environ['ACCUMULO_LOG_DIR']= ACCUMULO_HOME + '/logs/' + ID
            return Popen(cmd, stdout=PIPE, stderr=PIPE, **opts)


At least for me the test now seems to run successfully so I fixed at least that path through the code.  Obviously, that's a quick and dirty hack. I'm sure someone more familiar with the framework can do a cleaner fix.

As soon as I have a moment I'm going to go back and rerun all of the tests. I suspect they will all run much faster because of this change and the timeout issues I was seeing may simply go away.

Once again, thanks!
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 11, 2012, at 1:12 PM, Eric Newton wrote:

> Your tablet server crashed (the master reported the loss of the server).  My guess is that it had a stop-the-world gc that lasted longer than the zookeeper timeout.
> 
> Is there some reason why you aren't using the suggested
> 
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
> 
> jvm arguments?
> 
> -Eric
> 
> On Tue, Apr 10, 2012 at 2:21 PM, Keys Botzum <kb...@maprtech.com> wrote:
> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: simple.batchScanSplit.BatchScanSplitTest
> 
> When I run it, this is the output I see:
> ./run.py -t batchandsplittest -d -v10
> ….
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-17205 org.apache.accumulo.server.test.functional.BatchScanSplitTest run
> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-17205 org.apache.accumulo.server.test.functional.BatchScanSplitTest run to stop in 240 secs
> DEBUG:test.auto:out: 10 11:01:17,382 [admin.TableOperations] INFO : Problem with metadata table, first entry for table 1- 1<;00002 - has non null prev end row ... retrying ...
> DEBUG:test.auto:out: splits : [0000019cd, 0000026b4, 000004082, 000005a50, 000006737, 00000741e, 000007a0f, 000008, 000008dec, 000009ad3, 00000b4a1, 00000ba5, 00000c, 00000ce6f, 00000db56, 00000f524, 00000fa92, 00001, 000010ef2, 000011bd9, 0000135a7, 000013ad4, 000014, 000014f75, 000015c5c, 00001762a, 000017b15, 000018, 000018ff8, 000019cdf, 00001b6ad, 00001bb57, 00001c, 00001d07b, 00001dd62, 00001f730, 00001fb98, 00002, 0000210fe, 000021de5, 0000237b3, 000023bda, 000024, 000025181, 000025e68, 000027836, 000027c1b, 000028, 000029204, 000029eeb, 00002b8b9, 00002bc5d, 00002c, 00002d287, 00002df6e, 00002f93c, 00002fc9e, 00003, 00003130a, 000031ff1, 0000339bf, 000033ce, 000034, 00003538d, 000036074, 0000374b5, 000037a5b, 000038, 000038e83, 000039b6a, 00003b538, 00003c, 00003dbed, 00003e8d4]
> DEBUG:test.auto:out:
> DEBUG:test.auto:out: rate :
> DEBUG:test.auto:out: 205.34
> DEBUG:test.auto:out: 10 11:03:01,983 [impl.ThriftScanner] WARN : Security Violation in scan request to 10.250.99.204:39256: ThriftSecurityException(user:root, code:null)
> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
> DEBUG:test.auto:err:    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> DEBUG:test.auto:err:    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> DEBUG:test.auto:err: Caused by: java.lang.NullPointerException
> DEBUG:test.auto:err:    at org.apache.accumulo.core.client.AccumuloSecurityException.getDefaultErrorMessage(AccumuloSecurityException.java:30)
>        at org.apache.accumulo.core.client.AccumuloSecurityException.<init>(AccumuloSecurityException.java:70)
>        at org.apache.accumulo.core.client.impl.ThriftScanner.getBatchFromServer(ThriftScanner.java:153)
>        at org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:88)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:393)
> DEBUG:test.auto:err:    at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:115)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:370)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:390)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:215)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:288)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:236)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:301)
> DEBUG:test.auto:err:    at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.access$900(TabletServerBatchReaderIterator.java:73)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:374)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        ... 1 more
> DEBUG:test.auto:Output from command: 10 11:01:17,382 [admin.TableOperations] INFO : Problem with metadata table, first entry for table 1- 1<;00002 - has non null prev end row ... retrying ...
> splits : [0000019cd, 0000026b4, 000004082, 000005a50, 000006737, 00000741e, 000007a0f, 000008, 000008dec, 000009ad3, 00000b4a1, 00000ba5, 00000c, 00000ce6f, 00000db56, 00000f524, 00000fa92, 00001, 000010ef2, 000011bd9, 0000135a7, 000013ad4, 000014, 000014f75, 000015c5c, 00001762a, 000017b15, 000018, 000018ff8, 000019cdf, 00001b6ad, 00001bb57, 00001c, 00001d07b, 00001dd62, 00001f730, 00001fb98, 00002, 0000210fe, 000021de5, 0000237b3, 000023bda, 000024, 000025181, 000025e68, 000027836, 000027c1b, 000028, 000029204, 000029eeb, 00002b8b9, 00002bc5d, 00002c, 00002d287, 00002df6e, 00002f93c, 00002fc9e, 00003, 00003130a, 000031ff1, 0000339bf, 000033ce, 000034, 00003538d, 000036074, 0000374b5, 000037a5b, 000038, 000038e83, 000039b6a, 00003b538, 00003c, 00003dbed, 00003e8d4]
> rate : 205.34
> 10 11:03:01,983 [impl.ThriftScanner] WARN : Security Violation in scan request to 10.250.99.204:39256: ThriftSecurityException(user:root, code:null)
> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
>        at org.apache.accumulo.core.client.AccumuloSecurityException.getDefaultErrorMessage(AccumuloSecurityException.java:30)
>        at org.apache.accumulo.core.client.AccumuloSecurityException.<init>(AccumuloSecurityException.java:70)
>        at org.apache.accumulo.core.client.impl.ThriftScanner.getBatchFromServer(ThriftScanner.java:153)
>        at org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:88)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:393)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:115)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:370)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:390)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:215)
>        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:288)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:236)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:301)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.access$900(TabletServerBatchReaderIterator.java:73)
>        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:374)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        ... 1 more
> 
> FAIL
> ======================================================================
> FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>    self.waitForStop(handle, self.maxRuntime)
>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>    self.assert_(self.processResult(out, err, handle.returncode))
> AssertionError: False is not true
> 
> 
> ======================================================================
> FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>    self.waitForStop(handle, self.maxRuntime)
>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>    self.assert_(self.processResult(out, err, handle.returncode))
> AssertionError: False is not true
> 
> ----------------------------------------------------------------------
> Ran 1 test in 118.172s
> 
> FAILED (failures=1)
> 
> 
> In this case, I found nothing in the logs that seemed helpful, but perhaps I don't understand what I'm looking at. All I see in the tserver_xxx.debug.log prior to the error is this and the times are nearly 2 minutes before the failure.
> 
> 10 11:01:20,906 [tabletserver.Tablet] DEBUG: completeClose(saveState=true completeClose=false) 1;00001bb57;00001b6ad
> 10 11:01:20,908 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 10 11:01:20,911 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [0 0 0.00 1]
> 10 11:01:20,911 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.01 secs.
> 
> 10 11:01:20,913 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 10 11:01:20,914 [file.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between 00000fa92 and 00000fd49 for [/user/mapr/accumulo-SE-test-04-17205/tables/1/default_tablet/F0000000.rf]
> 10 11:01:20,930 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.02 secs.
> 
> 10 11:01:20,935 [file.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between 00000fd49 and 00001 for [/user/mapr/accumulo-SE-test-04-17205/tables/1/default_tablet/F0000000.rf]
> 10 11:01:20,952 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.02 secs.
> 
> There is no further output. Here is the complete tar and gzip of the logs:
> 
> 
> 
> 
> I really have no idea what is wrong here. I suspect some kind of security issue based on the stack trace, but its hard to tell. It looks like Accumulo was trying to report a security error but the error handling code had an issue. Is there some setting I need to provide to ensure that Accumulo has access to its error messages?
> 
> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
> 
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 
> 


Re: Accumulo on MapR - BatchScanSplitTest

Posted by Eric Newton <er...@gmail.com>.
Your tablet server crashed (the master reported the loss of the server).
 My guess is that it had a stop-the-world gc that lasted longer than the
zookeeper timeout.

Is there some reason why you aren't using the suggested

-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75

jvm arguments?

-Eric

On Tue, Apr 10, 2012 at 2:21 PM, Keys Botzum <kb...@maprtech.com> wrote:

> At this point all but two of the Accumulo test/system/auto tests have
> completed successfully. This test is failing and I'm not quite sure why:
> simple.batchScanSplit.BatchScanSplitTest
>
> When I run it, this is the output I see:
> ./run.py -t batchandsplittest -d -v10
> ….
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo
> org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u
> root -p secret -i SE-test-04-17205
> org.apache.accumulo.server.test.functional.BatchScanSplitTest run
> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo
> org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u
> root -p secret -i SE-test-04-17205
> org.apache.accumulo.server.test.functional.BatchScanSplitTest run to stop
> in 240 secs
> DEBUG:test.auto:out: 10 11:01:17,382 [admin.TableOperations] INFO :
> Problem with metadata table, first entry for table 1- 1<;00002 - has non
> null prev end row ... retrying ...
> DEBUG:test.auto:out: splits : [0000019cd, 0000026b4, 000004082, 000005a50,
> 000006737, 00000741e, 000007a0f, 000008, 000008dec, 000009ad3, 00000b4a1,
> 00000ba5, 00000c, 00000ce6f, 00000db56, 00000f524, 00000fa92, 00001,
> 000010ef2, 000011bd9, 0000135a7, 000013ad4, 000014, 000014f75, 000015c5c,
> 00001762a, 000017b15, 000018, 000018ff8, 000019cdf, 00001b6ad, 00001bb57,
> 00001c, 00001d07b, 00001dd62, 00001f730, 00001fb98, 00002, 0000210fe,
> 000021de5, 0000237b3, 000023bda, 000024, 000025181, 000025e68, 000027836,
> 000027c1b, 000028, 000029204, 000029eeb, 00002b8b9, 00002bc5d, 00002c,
> 00002d287, 00002df6e, 00002f93c, 00002fc9e, 00003, 00003130a, 000031ff1,
> 0000339bf, 000033ce, 000034, 00003538d, 000036074, 0000374b5, 000037a5b,
> 000038, 000038e83, 000039b6a, 00003b538, 00003c, 00003dbed, 00003e8d4]
> DEBUG:test.auto:out:
> DEBUG:test.auto:out: rate :
> DEBUG:test.auto:out: 205.34
> DEBUG:test.auto:out: 10 11:03:01,983 [impl.ThriftScanner] WARN : Security
> Violation in scan request to 10.250.99.204:39256:
> ThriftSecurityException(user:root, code:null)
> DEBUG:test.auto:err: Thread
> "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
> DEBUG:test.auto:err:    at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> DEBUG:test.auto:err:    at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> DEBUG:test.auto:err: Caused by: java.lang.NullPointerException
> DEBUG:test.auto:err:    at
> org.apache.accumulo.core.client.AccumuloSecurityException.getDefaultErrorMessage(AccumuloSecurityException.java:30)
>        at
> org.apache.accumulo.core.client.AccumuloSecurityException.<init>(AccumuloSecurityException.java:70)
>        at
> org.apache.accumulo.core.client.impl.ThriftScanner.getBatchFromServer(ThriftScanner.java:153)
>        at
> org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:88)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:393)
> DEBUG:test.auto:err:    at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at
> org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:115)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:370)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:390)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:215)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:288)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:236)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:301)
> DEBUG:test.auto:err:    at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.access$900(TabletServerBatchReaderIterator.java:73)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:374)
>        at
> org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        ... 1 more
> DEBUG:test.auto:Output from command: 10 11:01:17,382
> [admin.TableOperations] INFO : Problem with metadata table, first entry for
> table 1- 1<;00002 - has non null prev end row ... retrying ...
> splits : [0000019cd, 0000026b4, 000004082, 000005a50, 000006737,
> 00000741e, 000007a0f, 000008, 000008dec, 000009ad3, 00000b4a1, 00000ba5,
> 00000c, 00000ce6f, 00000db56, 00000f524, 00000fa92, 00001, 000010ef2,
> 000011bd9, 0000135a7, 000013ad4, 000014, 000014f75, 000015c5c, 00001762a,
> 000017b15, 000018, 000018ff8, 000019cdf, 00001b6ad, 00001bb57, 00001c,
> 00001d07b, 00001dd62, 00001f730, 00001fb98, 00002, 0000210fe, 000021de5,
> 0000237b3, 000023bda, 000024, 000025181, 000025e68, 000027836, 000027c1b,
> 000028, 000029204, 000029eeb, 00002b8b9, 00002bc5d, 00002c, 00002d287,
> 00002df6e, 00002f93c, 00002fc9e, 00003, 00003130a, 000031ff1, 0000339bf,
> 000033ce, 000034, 00003538d, 000036074, 0000374b5, 000037a5b, 000038,
> 000038e83, 000039b6a, 00003b538, 00003c, 00003dbed, 00003e8d4]
> rate : 205.34
> 10 11:03:01,983 [impl.ThriftScanner] WARN : Security Violation in scan
> request to 10.250.99.204:39256: ThriftSecurityException(user:root,
> code:null)
> ERROR:test.auto:This looks like a stack trace: Thread
> "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
>        at
> org.apache.accumulo.core.client.AccumuloSecurityException.getDefaultErrorMessage(AccumuloSecurityException.java:30)
>        at
> org.apache.accumulo.core.client.AccumuloSecurityException.<init>(AccumuloSecurityException.java:70)
>        at
> org.apache.accumulo.core.client.impl.ThriftScanner.getBatchFromServer(ThriftScanner.java:153)
>        at
> org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:88)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:393)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at
> org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:115)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:370)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:390)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:215)
>        at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:288)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:236)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:301)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.access$900(TabletServerBatchReaderIterator.java:73)
>        at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:374)
>        at
> org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        ... 1 more
>
> FAIL
> ======================================================================
> FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in
> runTest
>    self.waitForStop(handle, self.maxRuntime)
>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in
> waitForStop
>    self.assert_(self.processResult(out, err, handle.returncode))
> AssertionError: False is not true
>
>
> ======================================================================
> FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in
> runTest
>    self.waitForStop(handle, self.maxRuntime)
>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in
> waitForStop
>    self.assert_(self.processResult(out, err, handle.returncode))
> AssertionError: False is not true
>
> ----------------------------------------------------------------------
> Ran 1 test in 118.172s
>
> FAILED (failures=1)
>
>
> In this case, I found nothing in the logs that seemed helpful, but perhaps
> I don't understand what I'm looking at. All I see in the
> tserver_xxx.debug.log prior to the error is this and the times are nearly 2
> minutes before the failure.
>
> 10 11:01:20,906 [tabletserver.Tablet] DEBUG: completeClose(saveState=true
> completeClose=false) 1;00001bb57;00001b6ad
> 10 11:01:20,908 [tabletserver.TabletServer] DEBUG: ScanSess tid
> 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 10 11:01:20,911 [tabletserver.TabletServer] DEBUG: ScanSess tid
> 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [0 0 0.00 1]
> 10 11:01:20,911 [file.FileUtil] DEBUG: Found midPoint from indexes in
> 0.01 secs.
>
> 10 11:01:20,913 [tabletserver.TabletServer] DEBUG: ScanSess tid
> 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 10 11:01:20,914 [file.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> 00000fa92 and 00000fd49 for
> [/user/mapr/accumulo-SE-test-04-17205/tables/1/default_tablet/F0000000.rf]
> 10 11:01:20,930 [file.FileUtil] DEBUG: Found midPoint from indexes in
> 0.02 secs.
>
> 10 11:01:20,935 [file.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> 00000fd49 and 00001 for
> [/user/mapr/accumulo-SE-test-04-17205/tables/1/default_tablet/F0000000.rf]
> 10 11:01:20,952 [file.FileUtil] DEBUG: Found midPoint from indexes in
> 0.02 secs.
>
> There is no further output. Here is the complete tar and gzip of the logs:
>
>
>
>
> I really have no idea what is wrong here. I suspect some kind of security
> issue based on the stack trace, but its hard to tell. It looks like
> Accumulo was trying to report a security error but the error handling code
> had an issue. Is there some setting I need to provide to ensure that
> Accumulo has access to its error messages?
>
> If anyone has suggestions on how to look into this further from the
> Accumulo side, I'd really appreciate it.
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>

Re: Accumulo on MapR - BatchScanSplitTest

Posted by Keys Botzum <kb...@maprtech.com>.
At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: simple.batchScanSplit.BatchScanSplitTest

When I run it, this is the output I see:
./run.py -t batchandsplittest -d -v10
….
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-17205 org.apache.accumulo.server.test.functional.BatchScanSplitTest run
DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-17205 org.apache.accumulo.server.test.functional.BatchScanSplitTest run to stop in 240 secs
DEBUG:test.auto:out: 10 11:01:17,382 [admin.TableOperations] INFO : Problem with metadata table, first entry for table 1- 1<;00002 - has non null prev end row ... retrying ...
DEBUG:test.auto:out: splits : [0000019cd, 0000026b4, 000004082, 000005a50, 000006737, 00000741e, 000007a0f, 000008, 000008dec, 000009ad3, 00000b4a1, 00000ba5, 00000c, 00000ce6f, 00000db56, 00000f524, 00000fa92, 00001, 000010ef2, 000011bd9, 0000135a7, 000013ad4, 000014, 000014f75, 000015c5c, 00001762a, 000017b15, 000018, 000018ff8, 000019cdf, 00001b6ad, 00001bb57, 00001c, 00001d07b, 00001dd62, 00001f730, 00001fb98, 00002, 0000210fe, 000021de5, 0000237b3, 000023bda, 000024, 000025181, 000025e68, 000027836, 000027c1b, 000028, 000029204, 000029eeb, 00002b8b9, 00002bc5d, 00002c, 00002d287, 00002df6e, 00002f93c, 00002fc9e, 00003, 00003130a, 000031ff1, 0000339bf, 000033ce, 000034, 00003538d, 000036074, 0000374b5, 000037a5b, 000038, 000038e83, 000039b6a, 00003b538, 00003c, 00003dbed, 00003e8d4]
DEBUG:test.auto:out: 
DEBUG:test.auto:out: rate :
DEBUG:test.auto:out: 205.34
DEBUG:test.auto:out: 10 11:03:01,983 [impl.ThriftScanner] WARN : Security Violation in scan request to 10.250.99.204:39256: ThriftSecurityException(user:root, code:null)
DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
DEBUG:test.auto:err: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
DEBUG:test.auto:err: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
DEBUG:test.auto:err: Caused by: java.lang.NullPointerException
DEBUG:test.auto:err: 	at org.apache.accumulo.core.client.AccumuloSecurityException.getDefaultErrorMessage(AccumuloSecurityException.java:30)
	at org.apache.accumulo.core.client.AccumuloSecurityException.<init>(AccumuloSecurityException.java:70)
	at org.apache.accumulo.core.client.impl.ThriftScanner.getBatchFromServer(ThriftScanner.java:153)
	at org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:88)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:393)
DEBUG:test.auto:err: 	at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
	at org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:115)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:370)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:390)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:215)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:288)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:236)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:301)
DEBUG:test.auto:err: 	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.access$900(TabletServerBatchReaderIterator.java:73)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:374)
	at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	... 1 more
DEBUG:test.auto:Output from command: 10 11:01:17,382 [admin.TableOperations] INFO : Problem with metadata table, first entry for table 1- 1<;00002 - has non null prev end row ... retrying ...
splits : [0000019cd, 0000026b4, 000004082, 000005a50, 000006737, 00000741e, 000007a0f, 000008, 000008dec, 000009ad3, 00000b4a1, 00000ba5, 00000c, 00000ce6f, 00000db56, 00000f524, 00000fa92, 00001, 000010ef2, 000011bd9, 0000135a7, 000013ad4, 000014, 000014f75, 000015c5c, 00001762a, 000017b15, 000018, 000018ff8, 000019cdf, 00001b6ad, 00001bb57, 00001c, 00001d07b, 00001dd62, 00001f730, 00001fb98, 00002, 0000210fe, 000021de5, 0000237b3, 000023bda, 000024, 000025181, 000025e68, 000027836, 000027c1b, 000028, 000029204, 000029eeb, 00002b8b9, 00002bc5d, 00002c, 00002d287, 00002df6e, 00002f93c, 00002fc9e, 00003, 00003130a, 000031ff1, 0000339bf, 000033ce, 000034, 00003538d, 000036074, 0000374b5, 000037a5b, 000038, 000038e83, 000039b6a, 00003b538, 00003c, 00003dbed, 00003e8d4]
rate : 205.34
10 11:03:01,983 [impl.ThriftScanner] WARN : Security Violation in scan request to 10.250.99.204:39256: ThriftSecurityException(user:root, code:null)
ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
	at org.apache.accumulo.core.client.AccumuloSecurityException.getDefaultErrorMessage(AccumuloSecurityException.java:30)
	at org.apache.accumulo.core.client.AccumuloSecurityException.<init>(AccumuloSecurityException.java:70)
	at org.apache.accumulo.core.client.impl.ThriftScanner.getBatchFromServer(ThriftScanner.java:153)
	at org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:88)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:393)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
	at org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:115)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:370)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:390)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:536)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:215)
	at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:288)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:236)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:301)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.access$900(TabletServerBatchReaderIterator.java:73)
	at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:374)
	at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	... 1 more

FAIL
======================================================================
FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
    self.waitForStop(handle, self.maxRuntime)
  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
    self.assert_(self.processResult(out, err, handle.returncode))
AssertionError: False is not true


======================================================================
FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
    self.waitForStop(handle, self.maxRuntime)
  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
    self.assert_(self.processResult(out, err, handle.returncode))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 118.172s

FAILED (failures=1)


In this case, I found nothing in the logs that seemed helpful, but perhaps I don't understand what I'm looking at. All I see in the tserver_xxx.debug.log prior to the error is this and the times are nearly 2 minutes before the failure.

10 11:01:20,906 [tabletserver.Tablet] DEBUG: completeClose(saveState=true completeClose=false) 1;00001bb57;00001b6ad
10 11:01:20,908 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1] 
10 11:01:20,911 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [0 0 0.00 1] 
10 11:01:20,911 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.01 secs.

10 11:01:20,913 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:58769 !0 1 entries in 0.00 secs, nbTimes = [1 1 1.00 1] 
10 11:01:20,914 [file.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between 00000fa92 and 00000fd49 for [/user/mapr/accumulo-SE-test-04-17205/tables/1/default_tablet/F0000000.rf]
10 11:01:20,930 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.02 secs.

10 11:01:20,935 [file.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between 00000fd49 and 00001 for [/user/mapr/accumulo-SE-test-04-17205/tables/1/default_tablet/F0000000.rf]
10 11:01:20,952 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.02 secs.

There is no further output. Here is the complete tar and gzip of the logs:


Re: Accumulo on MapR - Compaction Test

Posted by Keys Botzum <kb...@maprtech.com>.
I need a bit more help. I really appreciate the help already provided by Kevin and Eric.

We've been testing Accumulo 1.4.0 on additional hardware platforms and have hit an unexpected issue. The compaction auto test (test/system/auth) fails. Interestingly, it fails every time on one machine and intermittently on another which makes me suspect it is some kind of race condition. At this point I can easily reproduce the problem and what I observe is that when the failure occurs, it always occurs in the same block of code but not on the same file. 

To be clear, when I run the following test:

	/run.py -t compact -d

I get this exception in the tserver log:

02 08:41:15,944 [tabletserver.TabletServer] WARN : exception while scanning tablet 1<<
java.io.IOException: invalid distance too far back
        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:63)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.accumulo.core.data.Value.readFields(Value.java:161)
        at org.apache.accumulo.core.file.map.MySequenceFile$Reader.getCurrentValue(MySequenceFile.java:1773)
        at org.apache.accumulo.core.file.map.MySequenceFile$Reader.next(MySequenceFile.java:1893)
        at org.apache.accumulo.core.file.map.MyMapFile$Reader.next(MyMapFile.java:678)
        at org.apache.accumulo.core.file.map.MyMapFile$Reader.next(MyMapFile.java:799)
        at org.apache.accumulo.core.file.map.MapFileOperations$RangeIterator.next(MapFileOperations.java:111)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.SkippingIterator.next(SkippingIterator.java:29)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:88)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Don't worry about the line numbers being slightly off from the official source, that's an artifact of my having added a lot of comments and trace code to figure out what is happening. I've added debug statements as follows to the code in MySequenceFile.java and Value.java.

Value
public void readFields(final DataInput in) throws IOException {
System.err.println(new java.util.Date() + "KDB: " + java.lang.Thread.currentThread().getId() + ": object id " + in.hashCode() +", readInt()...");
    int len;
    try {
        len = in.readInt();
        } catch (IOException e) {

System.err.println(new java.util.Date() + "KDB: " + java.lang.Thread.currentThread().getId() + ": readInt() resulted in FAILURE");
        throw e;
        }
System.err.println(new java.util.Date() + "KDB: " + java.lang.Thread.currentThread().getId() + ": readInt() resulted in SUCCESS");
    this.value = new byte[len];
System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": readFully() value of this many bytes: " + len);System.err.flush();
    in.readFully(this.value, 0, this.value.length);
  }

MySequenceFile
    /** Read a compressed buffer */
    private synchronized void readBuffer(DataInputBuffer buffer, CompressionInputStream filter) throws IOException {
      // Read data into a temporary buffer
      DataOutputBuffer dataBuffer = new DataOutputBuffer();

      try {
        int dataBufferLength = WritableUtils.readVInt(in);
        dataBuffer.write(in, dataBufferLength);

        // Set up 'buffer' connected to the input-stream:
        buffer.reset(dataBuffer.getData(), 0, dataBuffer.getLength());
System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": dataBuffer.getLength() " + dataBuffer.getLength() +  ", buffer.getLength() = " + buffer.getLength() + ", buffer.getPosition() = " + buffer.getPosition());

      } finally {
        dataBuffer.close();
      }

      // Reset the codec
System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": resetState()");
      filter.resetState();
    }

    private synchronized void seekToCurrentValue() throws IOException {
      if (!blockCompressed) {
        if (decompress) {
          valInFilter.resetState();
        }
        valBuffer.reset();
      } else {
        // Check if this is the first value in the 'block' to be read
        if (lazyDecompress && !valuesDecompressed) {
System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": seekToCurrentValue calling readBuffer ");
          // Read the value lengths and values
          readBuffer(valLenBuffer, valLenInFilter);
          readBuffer(valBuffer, valInFilter);
          noBufferedValues = noBufferedRecords;
          valuesDecompressed = true;
// KDB hack
          //valIn.reset();
        }

        // Calculate the no. of bytes to skip
        // Note: 'current' key has already been read!
        int skipValBytes = 0;
        int currentKey = noBufferedKeys + 1;
        if (noBufferedValues <= noBufferedKeys) {
          throw new IOException("Cannot seek to current value twice");
        }

        for (int i = noBufferedValues; i > currentKey; --i) {
          skipValBytes += WritableUtils.readVInt(valLenIn);
          --noBufferedValues;
        }
System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": seekToCurrentValue skipValBytes = " + skipValBytes +", valBuffer.getPosition() = " + valBuffer.getPosition());

        // Skip to the 'val' corresponding to 'current' key
        if (skipValBytes > 0) {
          if (valIn.skipBytes(skipValBytes) != skipValBytes) {
            throw new IOException("Failed to seek to " + currentKey + "(th) value!");
          }
        }
      }
System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": seekToCurrentValue valIn.available() = " + valIn.available() + ", valBuffer.getPosition() = " + valBuffer.getPosition() + ", getLength() = " + valBuffer.getLength());
    }

    public synchronized void getCurrentValue(Writable val) throws IOException {
      if (val instanceof Configurable) {
        ((Configurable) val).setConf(this.conf);
      }

      // Position stream to 'current' value
      seekToCurrentValue();

      if (!blockCompressed) {
        val.readFields(valIn);

        if (valIn.read() > 0) {
          LOG.info("available bytes: " + valIn.available());
          throw new IOException(val + " read " + (valBuffer.getPosition() - keyLength) + " bytes, should read " + (valBuffer.getLength() - keyLength));
        }
      } else {
        // Get the value
        int valLength = WritableUtils.readVInt(valLenIn);
        String str = "KDB: " + java.lang.Thread.currentThread().getId() + ": Attempt readFields(). valLength = " + valLength + ", valIn.available() is " + valIn.available() ;
        System.err.println(str);
        System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ": " + file);
        System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + " valBuffer.getPosition() = " + valBuffer.getPosition() + ", getLength() = " + valBuffer.getLength());

        val.readFields(new DataInputStream(valInFilter));

        //val.readFields(valIn);

        // Read another compressed 'value'
        --noBufferedValues;

        // Sanity check
        if (valLength < 0) {
          LOG.debug(val + " is a zero-length value");
        }
      }

    }



When run, this is the sequence that yields a failure. The thread and file change every time.

KDB: 195: dataBuffer.getLength() 18864, buffer.getLength() = 18864, buffer.getPosition() = 0
KDB: 195: resetState()
KDB: 195: seekToCurrentValue skipValBytes = 0, valBuffer.getPosition() = 0
KDB: 195: seekToCurrentValue valIn.available() = 1, valBuffer.getPosition() = 0, getLength() = 18864
Wed May 02 08:41:15 PDT 2012KDB: 174: object id 152878657, readInt()...
KDB: 195: Attempt readFields(). valLength = 54, valIn.available() is 1
KDB: 195: /user/mapr/accumulo-SE-test-04-19165/tables/1/b-0000000/I000001j.map/data
KDB: 195 valBuffer.getPosition() = 0, getLength() = 18864
Wed May 02 08:41:15 PDT 2012KDB: 174: readInt() resulted in SUCCESS
KDB: 174: readFully() value of this many bytes: 50
Wed May 02 08:41:15 PDT 2012KDB: 195: object id 1041146387, readInt()...
KDB: 174: call next(key)
KDB: 174: readVInt
KDB: 174: readFields KeyIn
KDB: 174: done with next(key), more = true key=row_0000387268 colf:col_00000 [L1&L2&G1&GROUP2] 1 false
KDB: 174: seekToCurrentValue skipValBytes = 0, valBuffer.getPosition() = 40960
KDB: 174: seekToCurrentValue valIn.available() = 1, valBuffer.getPosition() = 40960, getLength() = 42744
KDB: 174: Attempt readFields(). valLength = 54, valIn.available() is 1
KDB: 174: /user/mapr/accumulo-SE-test-04-19165/tables/1/b-0000000/I0000017.map/data
KDB: 174 valBuffer.getPosition() = 40960, getLength() = 42744
Wed May 02 08:41:15 PDT 2012KDB: 174: object id 850570553, readInt()...
Wed May 02 08:41:15 PDT 2012KDB: 174: readInt() resulted in SUCCESS
KDB: 174: readFully() value of this many bytes: 50
KDB: 174: call next(key)
KDB: 174: readVInt
KDB: 174: readFields KeyIn
KDB: 174: done with next(key), more = true key=row_0000387269 colf:col_00000 [L1&L2&G1&GROUP2] 1 false
KDB: 174: seekToCurrentValue skipValBytes = 0, valBuffer.getPosition() = 40960
KDB: 174: seekToCurrentValue valIn.available() = 1, valBuffer.getPosition() = 40960, getLength() = 42744
KDB: 174: Attempt readFields(). valLength = 54, valIn.available() is 1
KDB: 174: /user/mapr/accumulo-SE-test-04-19165/tables/1/b-0000000/I0000017.map/data
KDB: 174 valBuffer.getPosition() = 40960, getLength() = 42744
Wed May 02 08:41:15 PDT 2012KDB: 174: object id 1888129839, readInt()...
Wed May 02 08:41:15 PDT 2012KDB: 174: readInt() resulted in SUCCESS
KDB: 174: readFully() value of this many bytes: 50
KDB: 174: call next(key)
Wed May 02 08:41:15 PDT 2012KDB: 195: readInt() resulted in FAILURE

I'm trying to eliminate compression as part of the issue, but I haven't been able to. I disabled compression by editing TestUtils.py and I can see this in the tserver log:

02 08:40:48,103 [server.Accumulo] INFO : table.file.compress.blocksize = 100K
02 08:40:48,103 [server.Accumulo] INFO : table.file.compress.blocksize.index = 128K
02 08:40:48,103 [server.Accumulo] INFO : table.file.compress.type = none

But it's pretty clear the files are still compressed. I'd like to disable compression if you think it would be useful to get us a clearer picture of the issue. Is there a way to do that?

I've tried a number of things that have lead to dead ends, but basically I think this comes down to the following:
- next() is being used to get the next chunk of data
- we successfully obtain the keys
- we successfully read value data from the input stream as raw bytes (still compressed) into the internal buffer (valBuffer) which is indirectly referenced by the valIn stream
- as soon as we try to read the first integer from the valIn stream, we get the exception above.
- I can tell from my trace that the valBuffer has seems to have plenty of bytes, so I don't think this is something as simple as reading past the end of the buffer, but I could be wrong.
- the failure occurs "randomly" in the sense that the thread, buffer size, and even file being read when this fails, varies.

It's not clear to me if the problem is that the file is corrupted already or that the reading of the file is hitting a timing condition.


Any ideas much appreciated. I can of course provide any tracing you desire or make code changes.

Thank you,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com

Re: Accumulo on MapR - Compaction Test

Posted by Adam Fuchs <ad...@ugov.gov>.
Keys,

There's not really a way to change the compression type used by MyMapFile
within Accumulo -- it always uses block compression with the DefaultCodec
(gzip). However, if you want to write a separate standalone test that tries
to mimic this behavior then you can use one of the MyMapFile.Writer
constructors to specify the type of compression you want. Incidentally,
MyMapFile is legacy code and we're getting rid of it in version 1.5, but it
is well tested and we wouldn't expect to see this type of problem.

Cheers,
Adam



On Wed, May 2, 2012 at 1:35 PM, Keys Botzum <kb...@maprtech.com> wrote:

> I need a bit more help. I really appreciate the help already provided by
> Kevin and Eric.
>
> We've been testing Accumulo 1.4.0 on additional hardware platforms and
> have hit an unexpected issue. The compaction auto test (test/system/auth)
> fails. Interestingly, it fails every time on one machine and intermittently
> on another which makes me suspect it is some kind of race condition. At
> this point I can easily reproduce the problem and what I observe is that
> when the failure occurs, it always occurs in the same block of code but not
> on the same file.
>
> To be clear, when I run the following test:
>
> /run.py -t compact -d
>
> I get this exception in the tserver log:
>
> 02 08:41:15,944 [tabletserver.TabletServer] WARN : exception while
> scanning tablet 1<<
> java.io.IOException: invalid distance too far back
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
> Method)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>         at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>         at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>         at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:63)
> *        at java.io.DataInputStream.readInt(DataInputStream.java:370)*
> *        at org.apache.accumulo.core.data.Value.readFields(Value.java:161)
> *
> *        at
> org.apache.accumulo.core.file.map.MySequenceFile$Reader.getCurrentValue(MySequenceFile.java:1773)
> *
> *        at
> org.apache.accumulo.core.file.map.MySequenceFile$Reader.next(MySequenceFile.java:1893)
> *
>         at
> org.apache.accumulo.core.file.map.MyMapFile$Reader.next(MyMapFile.java:678)
>         at
> org.apache.accumulo.core.file.map.MyMapFile$Reader.next(MyMapFile.java:799)
>         at
> org.apache.accumulo.core.file.map.MapFileOperations$RangeIterator.next(MapFileOperations.java:111)
>         at
> org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>         at
> org.apache.accumulo.core.iterators.SkippingIterator.next(SkippingIterator.java:29)
>         at
> org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>         at
> org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:88)
>         at
> org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>         at
> org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>         at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>         at
> org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>         at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>         at
> org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>         at
> org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>         at
> org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>         at
> org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>         at
> org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>         at
> org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>         at
> org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>         at
> org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>         at
> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>         at
> org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> Don't worry about the line numbers being slightly off from the official
> source, that's an artifact of my having added a lot of comments and trace
> code to figure out what is happening. I've added debug statements as
> follows to the code in MySequenceFile.java and Value.java.
>
> *Value*
> public void readFields(final DataInput in) throws IOException {
> System.err.println(new java.util.Date() + "KDB: " +
> java.lang.Thread.currentThread().getId() + ": object id " + in.hashCode()
> +", readInt()...");
>     int len;
>     try {
>         len = in.readInt();
>         } catch (IOException e) {
>
> System.err.println(new java.util.Date() + "KDB: " +
> java.lang.Thread.currentThread().getId() + ": readInt() resulted in *
> FAILURE*");
>         throw e;
>         }
> System.err.println(new java.util.Date() + "KDB: " +
> java.lang.Thread.currentThread().getId() + ": readInt() resulted in
> SUCCESS");
>     this.value = new byte[len];
> System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ":
> readFully() value of this many bytes: " + len);System.err.flush();
>     in.readFully(this.value, 0, this.value.length);
>   }
>
> *MySequenceFile*
>     /** Read a compressed buffer */
>     private synchronized void readBuffer(DataInputBuffer buffer,
> CompressionInputStream filter) throws IOException {
>       // Read data into a temporary buffer
>       DataOutputBuffer dataBuffer = new DataOutputBuffer();
>
>       try {
>         int dataBufferLength = WritableUtils.readVInt(in);
>         dataBuffer.write(in, dataBufferLength);
>
>         // Set up 'buffer' connected to the input-stream:
>         buffer.reset(dataBuffer.getData(), 0, dataBuffer.getLength());
> System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ":
> dataBuffer.getLength() " + dataBuffer.getLength() +  ", buffer.getLength()
> = " + buffer.getLength() + ", buffer.getPosition() = " +
> buffer.getPosition());
>
>       } finally {
>         dataBuffer.close();
>       }
>
>       // Reset the codec
> System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ":
> resetState()");
>       filter.resetState();
>     }
>
>     private synchronized void seekToCurrentValue() throws IOException {
>       if (!blockCompressed) {
>         if (decompress) {
>           valInFilter.resetState();
>         }
>         valBuffer.reset();
>       } else {
>         // Check if this is the first value in the 'block' to be read
>         if (lazyDecompress && !valuesDecompressed) {
> System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ":
> seekToCurrentValue calling readBuffer ");
>           // Read the value lengths and values
>           readBuffer(valLenBuffer, valLenInFilter);
>           readBuffer(valBuffer, valInFilter);
>           noBufferedValues = noBufferedRecords;
>           valuesDecompressed = true;
> // KDB hack
>           //valIn.reset();
>         }
>
>         // Calculate the no. of bytes to skip
>         // Note: 'current' key has already been read!
>         int skipValBytes = 0;
>         int currentKey = noBufferedKeys + 1;
>         if (noBufferedValues <= noBufferedKeys) {
>           throw new IOException("Cannot seek to current value twice");
>         }
>
>         for (int i = noBufferedValues; i > currentKey; --i) {
>           skipValBytes += WritableUtils.readVInt(valLenIn);
>           --noBufferedValues;
>         }
> System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ":
> seekToCurrentValue skipValBytes = " + skipValBytes +",
> valBuffer.getPosition() = " + valBuffer.getPosition());
>
>         // Skip to the 'val' corresponding to 'current' key
>         if (skipValBytes > 0) {
>           if (valIn.skipBytes(skipValBytes) != skipValBytes) {
>             throw new IOException("Failed to seek to " + currentKey +
> "(th) value!");
>           }
>         }
>       }
> System.err.println("KDB: " + java.lang.Thread.currentThread().getId() + ":
> seekToCurrentValue valIn.available() = " + valIn.available() + ",
> valBuffer.getPosition() = " + valBuffer.getPosition() + ", getLength() = "
> + valBuffer.getLength());
>     }
>
>     public synchronized void getCurrentValue(Writable val) throws
> IOException {
>       if (val instanceof Configurable) {
>         ((Configurable) val).setConf(this.conf);
>       }
>
>       // Position stream to 'current' value
>       seekToCurrentValue();
>
>       if (!blockCompressed) {
>         val.readFields(valIn);
>
>         if (valIn.read() > 0) {
>           LOG.info("available bytes: " + valIn.available());
>           throw new IOException(val + " read " + (valBuffer.getPosition()
> - keyLength) + " bytes, should read " + (valBuffer.getLength() -
> keyLength));
>         }
>       } else {
>         // Get the value
>         int valLength = WritableUtils.readVInt(valLenIn);
>         String str = "KDB: " + java.lang.Thread.currentThread().getId() +
> ": Attempt readFields(). valLength = " + valLength + ", valIn.available()
> is " + valIn.available() ;
>         System.err.println(str);
>         System.err.println("KDB: " +
> java.lang.Thread.currentThread().getId() + ": " + file);
>         System.err.println("KDB: " +
> java.lang.Thread.currentThread().getId() + " valBuffer.getPosition() = " +
> valBuffer.getPosition() + ", getLength() = " + valBuffer.getLength());
>
>         val.readFields(new DataInputStream(valInFilter));
>
>         //val.readFields(valIn);
>
>         // Read another compressed 'value'
>         --noBufferedValues;
>
>         // Sanity check
>         if (valLength < 0) {
>           LOG.debug(val + " is a zero-length value");
>         }
>       }
>
>     }
>
>
>
> When run, this is the sequence that yields a failure. The thread and file
> change every time.
>
> KDB: 195: dataBuffer.getLength() 18864, buffer.getLength() = 18864,
> buffer.getPosition() = 0
> KDB: 195: resetState()
> KDB: 195: seekToCurrentValue skipValBytes = 0, valBuffer.getPosition() = 0
> KDB: 195: seekToCurrentValue valIn.available() = 1,
> valBuffer.getPosition() = 0, getLength() = 18864
> Wed May 02 08:41:15 PDT 2012KDB: 174: object id 152878657, readInt()...
> KDB: 195: Attempt readFields(). valLength = 54, valIn.available() is 1
> KDB: 195:
> /user/mapr/accumulo-SE-test-04-19165/tables/1/b-0000000/I000001j.map/data
> KDB: 195 valBuffer.getPosition() = 0, getLength() = 18864
> Wed May 02 08:41:15 PDT 2012KDB: 174: readInt() resulted in SUCCESS
> KDB: 174: readFully() value of this many bytes: 50
> Wed May 02 08:41:15 PDT 2012KDB: 195: object id 1041146387, readInt()...
> KDB: 174: call next(key)
> KDB: 174: readVInt
> KDB: 174: readFields KeyIn
> KDB: 174: done with next(key), more = true key=row_0000387268
> colf:col_00000 [L1&L2&G1&GROUP2] 1 false
> KDB: 174: seekToCurrentValue skipValBytes = 0, valBuffer.getPosition() =
> 40960
> KDB: 174: seekToCurrentValue valIn.available() = 1,
> valBuffer.getPosition() = 40960, getLength() = 42744
> KDB: 174: Attempt readFields(). valLength = 54, valIn.available() is 1
> KDB: 174:
> /user/mapr/accumulo-SE-test-04-19165/tables/1/b-0000000/I0000017.map/data
> KDB: 174 valBuffer.getPosition() = 40960, getLength() = 42744
> Wed May 02 08:41:15 PDT 2012KDB: 174: object id 850570553, readInt()...
> Wed May 02 08:41:15 PDT 2012KDB: 174: readInt() resulted in SUCCESS
> KDB: 174: readFully() value of this many bytes: 50
> KDB: 174: call next(key)
> KDB: 174: readVInt
> KDB: 174: readFields KeyIn
> KDB: 174: done with next(key), more = true key=row_0000387269
> colf:col_00000 [L1&L2&G1&GROUP2] 1 false
> KDB: 174: seekToCurrentValue skipValBytes = 0, valBuffer.getPosition() =
> 40960
> KDB: 174: seekToCurrentValue valIn.available() = 1,
> valBuffer.getPosition() = 40960, getLength() = 42744
> KDB: 174: Attempt readFields(). valLength = 54, valIn.available() is 1
> KDB: 174:
> /user/mapr/accumulo-SE-test-04-19165/tables/1/b-0000000/I0000017.map/data
> KDB: 174 valBuffer.getPosition() = 40960, getLength() = 42744
> Wed May 02 08:41:15 PDT 2012KDB: 174: object id 1888129839, readInt()...
> Wed May 02 08:41:15 PDT 2012KDB: 174: readInt() resulted in SUCCESS
> KDB: 174: readFully() value of this many bytes: 50
> KDB: 174: call next(key)
> Wed May 02 08:41:15 PDT 2012KDB: 195: readInt() resulted in *FAILURE*
>
> I'm trying to eliminate compression as part of the issue, but I haven't
> been able to. I disabled compression by editing TestUtils.py and I can see
> this in the tserver log:
>
> 02 08:40:48,103 [server.Accumulo] INFO : table.file.compress.blocksize =
> 100K
> 02 08:40:48,103 [server.Accumulo] INFO :
> table.file.compress.blocksize.index = 128K
> 02 08:40:48,103 [server.Accumulo] INFO : table.file.compress.type = none
>
> But it's pretty clear the files are still compressed. I'd like to disable
> compression if you think it would be useful to get us a clearer picture of
> the issue. Is there a way to do that?
>
> I've tried a number of things that have lead to dead ends, but basically I
> think this comes down to the following:
> - next() is being used to get the next chunk of data
> - we successfully obtain the keys
> - we successfully read value data from the input stream as raw bytes
> (still compressed) into the internal buffer (valBuffer) which is indirectly
> referenced by the valIn stream
> - as soon as we try to read the first integer from the valIn stream, we
> get the exception above.
> - I can tell from my trace that the valBuffer has seems to have plenty of
> bytes, so I don't think this is something as simple as reading past the end
> of the buffer, but I could be wrong.
> - the failure occurs "randomly" in the sense that the thread, buffer size,
> and even file being read when this fails, varies.
>
> It's not clear to me if the problem is that the file is corrupted already
> or that the reading of the file is hitting a timing condition.
>
>
> Any ideas much appreciated. I can of course provide any tracing you desire
> or make code changes.
>
> Thank you,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>

Re: Accumulo on MapR Continued

Posted by Keith Turner <ke...@deenlo.com>.
Another test that would be interesting to try is the continuous ingest
test under test/system/continuous.  There is a lot in the readme.  A
basic test would be to run ingest for a bit (a few hours or
overnight), then stop ingest and run the verification map reduce job.

Keith

On Fri, Mar 30, 2012 at 2:07 PM, Keys Botzum <kb...@maprtech.com> wrote:
> I've been building on the work of Todd Stavish to put Accumulo onto MapR:
>
> http://t.co/RJJ8Ht4B
> http://mail-archives.apache.org/mod_mbox/accumulo-user/201203.mbox/%3CCADKA3WiSRDrTw_vKcB9zrP0oyq01jQfZOBGQDRnujEjQ3kHd2w@mail.gmail.com%3E
>
> I've installed Accumulo on a 5 node MapR cluster (real hardware) basing my
> work on what Todd posted earlier. I'm using Accumulo 1.4.0 RC3 (downloaded
> from http://people.apache.org/~kturner/1.4.0rc3/) which contains the file
> system fix (https://issues.apache.org/jira/browse/ACCUMULO-476).
>
> The install went fine. I can start the Accumulo shell and create tables and
> such without difficulty.
>
> I decided then to do a more complete test using the Accumulo built in tests
> found under ACCUMULO_HOME/test and this is where I started encountering
> problems. It may be that I just don't understand how to configure and run
> the tests properly or perhaps something deeper. Any help would be greatly
> appreciated.
>
> I decided to run the tests under the auto directory as those seemed
> promising (was that a good idea?). As instructed in the README I can start
> tests in local mode (no MapReduce job) using ./run.py -t testname or leaving
> out the -t to run all tests. When I run the tests (all at once or
> individually) while the majority succeed a small subset consistently fail.
> They are:
>
> FAIL: runTest (simple.batchScanSplit.BatchScanSplitTest)
> FAIL: runTest (simple.bulkSplitOptimization.BulkSplitOptimizationTest)
> FAIL: runTest (simple.deleterows.DeleteRowsSplitTest)
> FAIL: runTest (simple.dynamic.DynamicClassloader)
> FAIL: runTest (simple.largeRow.LargeRowTest)
> FAIL: runTest (simple.merge.MergeTest)
> FAIL: runTest (simple.zooCacheTest.ZooCacheTest)
>
> Some fail because of timeouts, other fail with assertion errors. Rather than
> dumping every issue into the forum, I thought I'd just start by focusing on
> the first test - simple.batchScanSplit.BatchScanSplitTest. When I run this
> test by itself, this is the output:
>
> ./run.py -t ulkSplitOptimizationTest -d -v 10
> …
> DEBUG:test.auto:out:        1,010 records written |
> DEBUG:test.auto:out:   72,142 records/sec |       92,920 bytes written |
> 6,637,142 bytes/sec |
> DEBUG:test.auto:out:  0.014 secs
> FAIL
> ======================================================================
> FAIL: runTest (simple.bulkSplitOptimization.BulkSplitOptimizationTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 56, in
> runTest
>     self.waitForStop(handle, self.maxRuntime)
>   File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 371, in
> waitForStop
>     self.fail("Process failed to finish in %s seconds" % secs)
> AssertionError: Process failed to finish in 200 seconds
>
>
> ======================================================================
> FAIL: runTest (simple.bulkSplitOptimization.BulkSplitOptimizationTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 56, in
> runTest
>     self.waitForStop(handle, self.maxRuntime)
>   File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 371, in
> waitForStop
>     self.fail("Process failed to finish in %s seconds" % secs)
> AssertionError: Process failed to finish in 200 seconds
>
> ----------------------------------------------------------------------
> Ran 1 test in 208.163s
>
> FAILED (failures=1)
>
> Since I set the test to preserve the logs, I look at the logs and found only
> one thing that jumps out at me as the problem. I see this message repeated
> over and over while the test is running:
>
> 30 11:00:14,587 [tabletserver.TabletServer] DEBUG: ScanSess tid
> 10.250.99.204:45335 !0 2 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 30 11:00:15,092 [tabletserver.TabletServer] DEBUG: ScanSess tid
> 10.250.99.204:45335 !0 2 entries in 0.00 secs, nbTimes = [0 0 0.00 1]
> 30 11:00:15,499 [file.FileUtil] DEBUG: Too many indexes (100) to open at
> once for null null, reducing in tmpDir =
> /user/mapr/accumulo-SE-test-04-17598/tmp/idxReduce_1785031962
> 30 11:00:15,552 [tabletserver.TabletServer] DEBUG: gc PS
> Scavenge=4.22(+0.00) secs PS MarkSweep=0.18(+0.00) secs
> freemem=74,816,456(+5,884,960) totalmem=110,100,480
> 30 11:00:15,596 [tabletserver.TabletServer] DEBUG: ScanSess tid
> 10.250.99.204:45335 !0 2 entries in 0.00 secs, nbTimes = [1 1 1.00 1]
> 30 11:00:15,643 [file.FileUtil] DEBUG: Finished reducing indexes for null
> null in   0.14 secs
> 30 11:00:15,654 [file.FileUtil] DEBUG: Found midPoint from indexes in   0.01
> secs.
>
> 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey Seeking
> beyond EOF, filelen 733, wantpos 733
>
>
> I suspect the last line is the cause but I do not understand what it means
> or how to investigate this further. Of course I can provide any additional
> information you need but I didn't want to make this post even longer at
> first.
>
> Help is appreciated.
>
> Thanks!
> Keys
>
> p.s. As a sanity test I installed Accumulo a 2nd time on a MapR M5 VM with
> one CPU and the same tests fail.
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
Comments inline below

On Wed, Apr 11, 2012 at 9:48 AM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
>
> Thanks for the suggestion.  I made the change to the source as you suggested and rebuilt it using Maven (surprisingly easy).
>
> Here's the log from tserver now. Does this help at all? I can of course provide the complete log or logs if useful to you. I can also provide the temporary tables and such if that's useful.
>
>
> 10 15:44:07,786 [cache.LruBlockCache] DEBUG: Block cache LRU eviction completed. Freed 2494168 bytes.  Priority Sizes: Single=3.2550507MB (3413168), Multi=1
> 3.89547MB (14570456),Memory=0.0MB (0)
> 10 15:44:07,798 [rfile.RelativeKey] DEBUG: len : 131072
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s :OKl:2"cp>]yT(ZeP
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :

This looks like the data I would expect for this test, it reads a 128k
row, then a 0 len colfam, 0 len colqual, and 0 len colvis.

> 10 15:44:07,799 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:52572 2 1 entries in 0.03 secs, nbTimes = [25 25 25.00 1]
> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 65
> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.tP"RsUOI
> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 47

The reads above do not look right, the data looks like the data that
would be generated for the row.  However, the length is not right. Its
65 bytes instead of 128k.  Now it may not be trying to read the row,
there is not enough information here to be sure.  But its definitely
trying to read two fields that look rows in the test data, and the
lengths are not whats expected.  So the read is off, I suspect its not
starting read the key in the right place for some reason.  I am going
to look around in the code and see where the best places to add some
more debug might be.

> 10 15:44:07,833 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-SE-test-04-18004/tables/2/t-0000000/F000000p.rf
> 10 15:44:07,834 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... T
> RUNCATED<
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:381)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:135)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>
>
> It appears to be attempting to read 47 bytes but isn't succeeding. Out of curiosity I changed the code to read what it could and print a warning. Here's the new code version:
>
>
> And this is a snippet of the exception which occurs with that change. Everything else is the same. As you can see my hack gets us past the read of the key, but then the next read fails.
>
> 11 06:42:32,254 [rfile.RelativeKey] DEBUG: data :
> 11 06:42:32,254 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:47993 2 1 entries in 0.02
>  secs, nbTimes = [23 23 23.00 1]
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 65
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.t
> P"RsUOI
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 47
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: MISSING BYTES!!: read 45
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : )vRS>4 ?c>$Sgn#[QcscA!HAYcF;M_Jg3d&Jzc85$)6Y7^@^@
> 11 06:42:32,288 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-
> SE-test-04-32318/tables/2/t-0000000/F000000q.rf
> 11 06:42:32,289 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\
> ;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
> java.io.EOFException
>        at java.io.DataInputStream.readInt(DataInputStream.java:375)
>        at org.apache.accumulo.core.data.Value.readFields(Value.java:156)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:585)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.j
>       …..
>
> So it looks like we are missing quite a bit of data.
>
> Any help or ideas appreciated.
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 10, 2012, at 5:23 PM, Keith Turner wrote:
>
>> Keys,
>>
>> Looking at the test, it writes out random rows that 128k in len.  The
>> column family and column qualifier it writes out are 0 bytes long.
>> When the non compression test failed, it was trying to read a column
>> qualifier.  If we assume that it was reading a column qualifier from
>> the test table then it should be calling readFully() with a zero
>> length array.
>>
>> Trying to think how to debug this.  One way may be to change the code
>> in RelativeKey to the following and run the test.  This will show us
>> what its trying to do right before it hits the eof, but it will also
>> generate a lot of noise as things scan the metadata table.
>>
>>  private byte[] read(DataInput in) throws IOException {
>>    int len = WritableUtils.readVInt(in);
>>    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
>>    byte[] data = new byte[len];
>>    in.readFully(data);
>>    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
>> new String(data).substring(0, Math.min(data.length, 60)));
>>    return data;
>>  }
>>
>> Keith
>>
>> On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
>>> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>>>
>>> When I run it, this is the output I see:
>>> ./run.py -t largerowtest -d -v10
>>> ….
>>> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
>>> DEBUG:test.auto:{
>>> 'tserver.compaction.major.delay':'1',
>>> }
>>>
>>> DEBUG:test.auto:
>>> INFO:test.auto:killing accumulo processes everywhere
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
>>> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
>>> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
>>> DEBUG:test.auto:Exit code: 255
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
>>> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
>>> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
>>> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
>>> Instance name : SE-test-04-22187
>>> Enter initial password for root: ******
>>> Confirm initial password for root: ******
>>> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>>> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
>>> DEBUG:test.auto:Exit code: 0
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
>>> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
>>> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
>>> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
>>> DEBUG:test.auto:Exit code: 0
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
>>> DEBUG:test.auto:
>>> DEBUG:test.auto:
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
>>> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
>>> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>> DEBUG:test.auto:err:
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>        ... 6 more
>>> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>        ... 11 more
>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>> DEBUG:test.auto:err:
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>        at $Proxy1.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>        ... 13 more
>>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>> java.lang.reflect.InvocationTargetException
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>        ... 6 more
>>> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>        ... 11 more
>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>        at $Proxy1.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>        ... 13 more
>>>
>>> FAIL
>>> ======================================================================
>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>    self.waitForStop(handle, self.maxRuntime)
>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>> AssertionError: False is not true
>>>
>>>
>>> ======================================================================
>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>    self.waitForStop(handle, self.maxRuntime)
>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>> AssertionError: False is not true
>>>
>>> ----------------------------------------------------------------------
>>> Ran 1 test in 43.014s
>>>
>>> FAILED (failures=1)
>>>
>>>
>>> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
>>> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>>>
>>>
>>> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
>>> java.io.IOException: invalid distance too far back
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>        at $Proxy0.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>        ... 15 more
>>> Caused by: java.io.IOException: invalid distance too far back
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        ... 1 more
>>>
>>>
>>> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>>>
>>> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>>>
>>> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
>>> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
>>> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>> java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>        at $Proxy0.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>        ... 15 more
>>> Caused by: java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>        at $Proxy0.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>        ... 15 more
>>> Caused by: java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        ... 1 more
>>>
>>>
>>> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
>>> 1) the file was originally written incorrectly by the writer,
>>> 2) the reader is reading too far
>>>
>>> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>>>
>>> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>>>
>>> Thanks,
>>> Keys
>>> ________________________________
>>> Keys Botzum
>>> Senior Principal Technologist
>>> WW Systems Engineering
>>> kbotzum@maprtech.com
>>> 443-718-0098
>>> MapR Technologies
>>> http://www.mapr.com
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Eric Newton <er...@gmail.com>.
The latelastcontact test was recently re-written, so it will work in the
next release.

You could configure and run the continuous ingest and random walk tests.
 We run them for 24 hours, with agitation (random server kill) and without
prior to a release.

These are more likely to stress the underlying filesystem.

-Eric

On Wed, Apr 18, 2012 at 1:17 PM, Keys Botzum <kb...@maprtech.com> wrote:

> Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering,
> all of the Accumulo tests is test/system/auto are now passing. Note that
> the latelastcontact test only passes if I actually install zookeeper on the
> host. This is because of the dependency on zkCli.sh that I mentioned
> earlier.
>
> The final piece of the puzzle was that MapR does aggressive read ahead
> caching of data as well as aggregation of writes to improve performance. As
> with Hbase, we don't think this type of behavior is helpful with something
> like Accumulo. In our specific case, the interaction between Accumulo and
> MapR's behavior results in the large row test failing.
>
> So now I have one more question. To disable the caching and aggregation
> behavior, we need to set these properties:
> <property>
> <name>fs.mapr.readbuffering</name>
> <value>false</value>
> </property>
>
> <property>
> <name>fs.mapr.aggregate.writes</name>
> <value>false</value>
> </property>
>
> If I set them in core-site.xml they of course work but that's a global
> setting. I want to only affect Accumulo. If I set them in
> accumulo-site.xml, I presume they take effect for normal Accumulo usage,
> but I'm nearly certain that settings in accumulo-site.xml do not affect the
> tests as I posted earlier. How can I set those two properties in a way that
> will cause the tests temporary configuration to take them into account? I
> tried editing TestUtils.py TestUtilsMixin settings as did work for the
> Accumulo property table.file.compress.type, but the MapR related properties
> don't seem to take. Ideas?
>
> Also, if all of the auto tests pass successfully do you feel comfortable
> that the testing was sufficient or do you recommend running additional
> tests?
>
> Thanks!
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
Eric,

Clever. I'll add that to the doc as an option.

Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 20, 2012, at 9:44 AM, Eric Newton wrote:

> You should be able to adjust the classpath in conf/accumulo-site.xml, and remove $HADOOP_HOME/conf and just put the updated core-site.xml in the accumulo/conf directory.
> 
> -Eric
> 
> On Fri, Apr 20, 2012 at 9:35 AM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
> 
> I was able to get Accumulo to use a Accumulo specific configuration of Hadoop. It was a bit of a hack. Basically I created a fake Hadoop installation tree that is almost entirely symbolic links to the real tree under /opt/mapr/hadoop. The only real file in the tree is core-site.xml where I set the two properties. The essential steps where:
> 	cd /opt/accumulo
> 	mkdir hadoop
> 	mkdir hadoop/hadoop-0.20.2
> 	cd  hadoop/hadoop-0.20.2
> 	ln -s /opt/mapr/hadoop/hadoop-0.20.2/* .
> 	rm conf
> 	mkdir conf
> 	cd conf
> 	ln -s /opt/mapr/hadoop/hadoop-0.20.2/conf/*
> 	cp core-site.xml t
> 	mv t core-site.xml
> 	edit core-site.xml as needed
> 
> Then I set the HADOOP_HOME in accumulo-env.sh to that directory and everything worked fine.
> 
> By the way, I tried setting HADOOP_CONF_DIR and that had no effect.
> 
> Since I plan to document these steps, I want to make sure I understood your intent and that I haven't missed something. Typically in Hadoop components the ultimate configuration is a combination of each components *-site.xml file. As a result I can set things in, for example, hbase-site.xml that are really Hadoop properties. Assuming I understood what you and Eric were saying, this is not true in Accumulo. That's fine by me, but I just want to make sure I'm not saying things that aren't true.
> 
> Thanks again for all of your help,
> Keys
> 
> p.s. I'm running the random and ingest tests you and Eric suggested as we speak. The random test completed successfully.
> 
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 


Re: Accumulo on MapR Continued

Posted by Eric Newton <er...@gmail.com>.
You should be able to adjust the classpath in conf/accumulo-site.xml, and
remove $HADOOP_HOME/conf and just put the updated core-site.xml in the
accumulo/conf directory.

-Eric

On Fri, Apr 20, 2012 at 9:35 AM, Keys Botzum <kb...@maprtech.com> wrote:

> Keith,
>
> I was able to get Accumulo to use a Accumulo specific configuration of
> Hadoop. It was a bit of a hack. Basically I created a fake Hadoop
> installation tree that is almost entirely symbolic links to the real tree
> under /opt/mapr/hadoop. The only real file in the tree is core-site.xml
> where I set the two properties. The essential steps where:
> cd /opt/accumulo
> mkdir hadoop
> mkdir hadoop/hadoop-0.20.2
> cd  hadoop/hadoop-0.20.2
> ln -s /opt/mapr/hadoop/hadoop-0.20.2/* .
> rm conf
> mkdir conf
> cd conf
> ln -s /opt/mapr/hadoop/hadoop-0.20.2/conf/*
> cp core-site.xml t
> mv t core-site.xml
> edit core-site.xml as needed
>
> Then I set the HADOOP_HOME in accumulo-env.sh to that directory and
> everything worked fine.
>
> By the way, I tried setting HADOOP_CONF_DIR and that had no effect.
>
> Since I plan to document these steps, I want to make sure I understood
> your intent and that I haven't missed something. Typically in Hadoop
> components the ultimate configuration is a combination of each components
> *-site.xml file. As a result I can set things in, for example,
> hbase-site.xml that are really Hadoop properties. Assuming I understood
> what you and Eric were saying, this is not true in Accumulo. That's fine by
> me, but I just want to make sure I'm not saying things that aren't true.
>
> Thanks again for all of your help,
> Keys
>
> p.s. I'm running the random and ingest tests you and Eric suggested as we
> speak. The random test completed successfully.
>
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 18, 2012, at 3:11 PM, Keith Turner wrote:
>
> I suppose accumulo could be pointed to a different hadoop config dir.
>
> On Wed, Apr 18, 2012 at 1:58 PM, Keys Botzum <kb...@maprtech.com> wrote:
>
> Eric and Keith,
>
>
> I will attempt the additional tests you have suggested.
>
>
> Any ideas on what to do regarding those configuration properties? With
> hbase
>
> in hbase-site.xml, we set those properties and they work fine. Is there
> some
>
> incantation I'm missing here? I really don't want those properties to be
>
> global as they will negatively impact performance and are only relevant to
>
> hbase and Accumulo.
>
>
> Thanks,
>
> Keys
>
> ________________________________
>
> Keys Botzum
>
> Senior Principal Technologist
>
> WW Systems Engineering
>
> kbotzum@maprtech.com
>
> 443-718-0098
>
> MapR Technologies
>
> http://www.mapr.com
>
>
>
>
> On Apr 18, 2012, at 1:42 PM, Keith Turner wrote:
>
>
> Settings in accumulo-site.xml do not end up in the hadoop config
>
> object, so setting them will probably have no effect.
>
>
> I would suggest running continuous ingest test and random walk test if
>
> you really want to stress it.  These are the test we use prior to an
>
> accumulo release.  You would need to exclude the random walk security
>
> test, it triggers known bug in 1.4 that are not fixed.
>
>
> Running the test on a cluster overnight would be good.
>
>
> Keith
>
>
> On Wed, Apr 18, 2012 at 1:17 PM, Keys Botzum <kb...@maprtech.com> wrote:
>
>
> Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering,
>
> all of the Accumulo tests is test/system/auto are now passing. Note that
> the
>
> latelastcontact test only passes if I actually install zookeeper on the
>
> host. This is because of the dependency on zkCli.sh that I mentioned
>
> earlier.
>
>
>
> The final piece of the puzzle was that MapR does aggressive read ahead
>
> caching of data as well as aggregation of writes to improve performance. As
>
> with Hbase, we don't think this type of behavior is helpful with something
>
> like Accumulo. In our specific case, the interaction between Accumulo and
>
> MapR's behavior results in the large row test failing.
>
>
>
> So now I have one more question. To disable the caching and aggregation
>
> behavior, we need to set these properties:
>
>
> <property>
>
>
> <name>fs.mapr.readbuffering</name>
>
>
> <value>false</value>
>
>
> </property>
>
>
>
> <property>
>
>
> <name>fs.mapr.aggregate.writes</name>
>
>
> <value>false</value>
>
>
> </property>
>
>
>
> If I set them in core-site.xml they of course work but that's a global
>
> setting. I want to only affect Accumulo. If I set them in
> accumulo-site.xml,
>
> I presume they take effect for normal Accumulo usage, but I'm nearly
> certain
>
> that settings in accumulo-site.xml do not affect the tests as I posted
>
> earlier. How can I set those two properties in a way that will cause the
>
> tests temporary configuration to take them into account? I tried editing
>
> TestUtils.py TestUtilsMixin settings as did work for the Accumulo property
>
> table.file.compress.type, but the MapR related properties don't seem to
>
> take. Ideas?
>
>
>
> Also, if all of the auto tests pass successfully do you feel comfortable
>
> that the testing was sufficient or do you recommend running additional
>
> tests?
>
>
>
> Thanks!
>
>
> Keys
>
>
> ________________________________
>
>
> Keys Botzum
>
>
> Senior Principal Technologist
>
>
> WW Systems Engineering
>
>
> kbotzum@maprtech.com
>
>
> 443-718-0098
>
>
> MapR Technologies
>
>
> http://www.mapr.com
>
>
>
>
>
>

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

I was able to get Accumulo to use a Accumulo specific configuration of Hadoop. It was a bit of a hack. Basically I created a fake Hadoop installation tree that is almost entirely symbolic links to the real tree under /opt/mapr/hadoop. The only real file in the tree is core-site.xml where I set the two properties. The essential steps where:
	cd /opt/accumulo
	mkdir hadoop
	mkdir hadoop/hadoop-0.20.2
	cd  hadoop/hadoop-0.20.2
	ln -s /opt/mapr/hadoop/hadoop-0.20.2/* .
	rm conf
	mkdir conf
	cd conf
	ln -s /opt/mapr/hadoop/hadoop-0.20.2/conf/*
	cp core-site.xml t
	mv t core-site.xml
	edit core-site.xml as needed

Then I set the HADOOP_HOME in accumulo-env.sh to that directory and everything worked fine.

By the way, I tried setting HADOOP_CONF_DIR and that had no effect.

Since I plan to document these steps, I want to make sure I understood your intent and that I haven't missed something. Typically in Hadoop components the ultimate configuration is a combination of each components *-site.xml file. As a result I can set things in, for example, hbase-site.xml that are really Hadoop properties. Assuming I understood what you and Eric were saying, this is not true in Accumulo. That's fine by me, but I just want to make sure I'm not saying things that aren't true.

Thanks again for all of your help,
Keys

p.s. I'm running the random and ingest tests you and Eric suggested as we speak. The random test completed successfully.
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 18, 2012, at 3:11 PM, Keith Turner wrote:

> I suppose accumulo could be pointed to a different hadoop config dir.
> 
> On Wed, Apr 18, 2012 at 1:58 PM, Keys Botzum <kb...@maprtech.com> wrote:
>> Eric and Keith,
>> 
>> I will attempt the additional tests you have suggested.
>> 
>> Any ideas on what to do regarding those configuration properties? With hbase
>> in hbase-site.xml, we set those properties and they work fine. Is there some
>> incantation I'm missing here? I really don't want those properties to be
>> global as they will negatively impact performance and are only relevant to
>> hbase and Accumulo.
>> 
>> Thanks,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 
>> 
>> 
>> On Apr 18, 2012, at 1:42 PM, Keith Turner wrote:
>> 
>> Settings in accumulo-site.xml do not end up in the hadoop config
>> object, so setting them will probably have no effect.
>> 
>> I would suggest running continuous ingest test and random walk test if
>> you really want to stress it.  These are the test we use prior to an
>> accumulo release.  You would need to exclude the random walk security
>> test, it triggers known bug in 1.4 that are not fixed.
>> 
>> Running the test on a cluster overnight would be good.
>> 
>> Keith
>> 
>> On Wed, Apr 18, 2012 at 1:17 PM, Keys Botzum <kb...@maprtech.com> wrote:
>> 
>> Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering,
>> all of the Accumulo tests is test/system/auto are now passing. Note that the
>> latelastcontact test only passes if I actually install zookeeper on the
>> host. This is because of the dependency on zkCli.sh that I mentioned
>> earlier.
>> 
>> 
>> The final piece of the puzzle was that MapR does aggressive read ahead
>> caching of data as well as aggregation of writes to improve performance. As
>> with Hbase, we don't think this type of behavior is helpful with something
>> like Accumulo. In our specific case, the interaction between Accumulo and
>> MapR's behavior results in the large row test failing.
>> 
>> 
>> So now I have one more question. To disable the caching and aggregation
>> behavior, we need to set these properties:
>> 
>> <property>
>> 
>> <name>fs.mapr.readbuffering</name>
>> 
>> <value>false</value>
>> 
>> </property>
>> 
>> 
>> <property>
>> 
>> <name>fs.mapr.aggregate.writes</name>
>> 
>> <value>false</value>
>> 
>> </property>
>> 
>> 
>> If I set them in core-site.xml they of course work but that's a global
>> setting. I want to only affect Accumulo. If I set them in accumulo-site.xml,
>> I presume they take effect for normal Accumulo usage, but I'm nearly certain
>> that settings in accumulo-site.xml do not affect the tests as I posted
>> earlier. How can I set those two properties in a way that will cause the
>> tests temporary configuration to take them into account? I tried editing
>> TestUtils.py TestUtilsMixin settings as did work for the Accumulo property
>> table.file.compress.type, but the MapR related properties don't seem to
>> take. Ideas?
>> 
>> 
>> Also, if all of the auto tests pass successfully do you feel comfortable
>> that the testing was sufficient or do you recommend running additional
>> tests?
>> 
>> 
>> Thanks!
>> 
>> Keys
>> 
>> ________________________________
>> 
>> Keys Botzum
>> 
>> Senior Principal Technologist
>> 
>> WW Systems Engineering
>> 
>> kbotzum@maprtech.com
>> 
>> 443-718-0098
>> 
>> MapR Technologies
>> 
>> http://www.mapr.com
>> 
>> 
>> 


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
I suppose accumulo could be pointed to a different hadoop config dir.

On Wed, Apr 18, 2012 at 1:58 PM, Keys Botzum <kb...@maprtech.com> wrote:
> Eric and Keith,
>
> I will attempt the additional tests you have suggested.
>
> Any ideas on what to do regarding those configuration properties? With hbase
> in hbase-site.xml, we set those properties and they work fine. Is there some
> incantation I'm missing here? I really don't want those properties to be
> global as they will negatively impact performance and are only relevant to
> hbase and Accumulo.
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 18, 2012, at 1:42 PM, Keith Turner wrote:
>
> Settings in accumulo-site.xml do not end up in the hadoop config
> object, so setting them will probably have no effect.
>
> I would suggest running continuous ingest test and random walk test if
> you really want to stress it.  These are the test we use prior to an
> accumulo release.  You would need to exclude the random walk security
> test, it triggers known bug in 1.4 that are not fixed.
>
> Running the test on a cluster overnight would be good.
>
> Keith
>
> On Wed, Apr 18, 2012 at 1:17 PM, Keys Botzum <kb...@maprtech.com> wrote:
>
> Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering,
> all of the Accumulo tests is test/system/auto are now passing. Note that the
> latelastcontact test only passes if I actually install zookeeper on the
> host. This is because of the dependency on zkCli.sh that I mentioned
> earlier.
>
>
> The final piece of the puzzle was that MapR does aggressive read ahead
> caching of data as well as aggregation of writes to improve performance. As
> with Hbase, we don't think this type of behavior is helpful with something
> like Accumulo. In our specific case, the interaction between Accumulo and
> MapR's behavior results in the large row test failing.
>
>
> So now I have one more question. To disable the caching and aggregation
> behavior, we need to set these properties:
>
> <property>
>
> <name>fs.mapr.readbuffering</name>
>
> <value>false</value>
>
> </property>
>
>
> <property>
>
> <name>fs.mapr.aggregate.writes</name>
>
> <value>false</value>
>
> </property>
>
>
> If I set them in core-site.xml they of course work but that's a global
> setting. I want to only affect Accumulo. If I set them in accumulo-site.xml,
> I presume they take effect for normal Accumulo usage, but I'm nearly certain
> that settings in accumulo-site.xml do not affect the tests as I posted
> earlier. How can I set those two properties in a way that will cause the
> tests temporary configuration to take them into account? I tried editing
> TestUtils.py TestUtilsMixin settings as did work for the Accumulo property
> table.file.compress.type, but the MapR related properties don't seem to
> take. Ideas?
>
>
> Also, if all of the auto tests pass successfully do you feel comfortable
> that the testing was sufficient or do you recommend running additional
> tests?
>
>
> Thanks!
>
> Keys
>
> ________________________________
>
> Keys Botzum
>
> Senior Principal Technologist
>
> WW Systems Engineering
>
> kbotzum@maprtech.com
>
> 443-718-0098
>
> MapR Technologies
>
> http://www.mapr.com
>
>
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Eric and Keith,

I will attempt the additional tests you have suggested.

Any ideas on what to do regarding those configuration properties? With hbase in hbase-site.xml, we set those properties and they work fine. Is there some incantation I'm missing here? I really don't want those properties to be global as they will negatively impact performance and are only relevant to hbase and Accumulo.

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 18, 2012, at 1:42 PM, Keith Turner wrote:

> Settings in accumulo-site.xml do not end up in the hadoop config
> object, so setting them will probably have no effect.
> 
> I would suggest running continuous ingest test and random walk test if
> you really want to stress it.  These are the test we use prior to an
> accumulo release.  You would need to exclude the random walk security
> test, it triggers known bug in 1.4 that are not fixed.
> 
> Running the test on a cluster overnight would be good.
> 
> Keith
> 
> On Wed, Apr 18, 2012 at 1:17 PM, Keys Botzum <kb...@maprtech.com> wrote:
>> Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering, all of the Accumulo tests is test/system/auto are now passing. Note that the latelastcontact test only passes if I actually install zookeeper on the host. This is because of the dependency on zkCli.sh that I mentioned earlier.
>> 
>> The final piece of the puzzle was that MapR does aggressive read ahead caching of data as well as aggregation of writes to improve performance. As with Hbase, we don't think this type of behavior is helpful with something like Accumulo. In our specific case, the interaction between Accumulo and MapR's behavior results in the large row test failing.
>> 
>> So now I have one more question. To disable the caching and aggregation behavior, we need to set these properties:
>> <property>
>> <name>fs.mapr.readbuffering</name>
>> <value>false</value>
>> </property>
>> 
>> <property>
>> <name>fs.mapr.aggregate.writes</name>
>> <value>false</value>
>> </property>
>> 
>> If I set them in core-site.xml they of course work but that's a global setting. I want to only affect Accumulo. If I set them in accumulo-site.xml, I presume they take effect for normal Accumulo usage, but I'm nearly certain that settings in accumulo-site.xml do not affect the tests as I posted earlier. How can I set those two properties in a way that will cause the tests temporary configuration to take them into account? I tried editing TestUtils.py TestUtilsMixin settings as did work for the Accumulo property table.file.compress.type, but the MapR related properties don't seem to take. Ideas?
>> 
>> Also, if all of the auto tests pass successfully do you feel comfortable that the testing was sufficient or do you recommend running additional tests?
>> 
>> Thanks!
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
Settings in accumulo-site.xml do not end up in the hadoop config
object, so setting them will probably have no effect.

I would suggest running continuous ingest test and random walk test if
you really want to stress it.  These are the test we use prior to an
accumulo release.  You would need to exclude the random walk security
test, it triggers known bug in 1.4 that are not fixed.

Running the test on a cluster overnight would be good.

Keith

On Wed, Apr 18, 2012 at 1:17 PM, Keys Botzum <kb...@maprtech.com> wrote:
> Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering, all of the Accumulo tests is test/system/auto are now passing. Note that the latelastcontact test only passes if I actually install zookeeper on the host. This is because of the dependency on zkCli.sh that I mentioned earlier.
>
> The final piece of the puzzle was that MapR does aggressive read ahead caching of data as well as aggregation of writes to improve performance. As with Hbase, we don't think this type of behavior is helpful with something like Accumulo. In our specific case, the interaction between Accumulo and MapR's behavior results in the large row test failing.
>
> So now I have one more question. To disable the caching and aggregation behavior, we need to set these properties:
> <property>
> <name>fs.mapr.readbuffering</name>
> <value>false</value>
> </property>
>
> <property>
> <name>fs.mapr.aggregate.writes</name>
> <value>false</value>
> </property>
>
> If I set them in core-site.xml they of course work but that's a global setting. I want to only affect Accumulo. If I set them in accumulo-site.xml, I presume they take effect for normal Accumulo usage, but I'm nearly certain that settings in accumulo-site.xml do not affect the tests as I posted earlier. How can I set those two properties in a way that will cause the tests temporary configuration to take them into account? I tried editing TestUtils.py TestUtilsMixin settings as did work for the Accumulo property table.file.compress.type, but the MapR related properties don't seem to take. Ideas?
>
> Also, if all of the auto tests pass successfully do you feel comfortable that the testing was sufficient or do you recommend running additional tests?
>
> Thanks!
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Thanks to the help of Keith, Todd, and Eric, as well as MapR engineering, all of the Accumulo tests is test/system/auto are now passing. Note that the latelastcontact test only passes if I actually install zookeeper on the host. This is because of the dependency on zkCli.sh that I mentioned earlier.

The final piece of the puzzle was that MapR does aggressive read ahead caching of data as well as aggregation of writes to improve performance. As with Hbase, we don't think this type of behavior is helpful with something like Accumulo. In our specific case, the interaction between Accumulo and MapR's behavior results in the large row test failing. 

So now I have one more question. To disable the caching and aggregation behavior, we need to set these properties:
<property>
<name>fs.mapr.readbuffering</name>
<value>false</value>
</property>

<property>
<name>fs.mapr.aggregate.writes</name>
<value>false</value>
</property>

If I set them in core-site.xml they of course work but that's a global setting. I want to only affect Accumulo. If I set them in accumulo-site.xml, I presume they take effect for normal Accumulo usage, but I'm nearly certain that settings in accumulo-site.xml do not affect the tests as I posted earlier. How can I set those two properties in a way that will cause the tests temporary configuration to take them into account? I tried editing TestUtils.py TestUtilsMixin settings as did work for the Accumulo property table.file.compress.type, but the MapR related properties don't seem to take. Ideas?

Also, if all of the auto tests pass successfully do you feel comfortable that the testing was sufficient or do you recommend running additional tests?

Thanks!
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,


With MapR Engineering's assistance we've been able to determine that this is somehow related to MapR buffering of file reads. We'll continue working to figure out where the issue really lies and how to address it.

I'll post a followup here, hopefully soon.

Thanks again for all of your help. It is really appreciated,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com


On Apr 17, 2012, at 6:42 PM, Keith Turner wrote:

> I think it was failing for 84.  It get some data for row 84 from a
> block, but its not sure if the next block contains more data for that
> row (it does not) so it looks and when it does that fails.
> 
> The fact that you can query all rows in the local fs show that the
> file in DFS is sounds.  Also, the fact that you can query the
> individual row in DFS shows the file is sound.  The file is not
> corrupt, but corruption seems to occur under certain circumstances
> when reading the file.
> 
> The rows that are being looked up in the file are not in sorted order.
> So we are reading randomly from the file.  So it seems that seeking
> the file around alot randomly seems to eventually leave something in
> MapR in a bad state.  At this point I feel the rfile code is ok and
> that it seems like the problem is w/ MapR, because it worked in local
> fs and w/ single row in dfs.  What do you think, does this deduction
> sound reasonable?
> 
> There is one thing that is not being printed out, that is when it
> reads the rfile index.  In this case that is there its failing.  The
> following information comes from the index, but we are not printing
> any debug about reading the index blocks.
> 
>  .... DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 .....
> 
> You can see in the stack trace that failed that its was trying to read
> an index block.
> 
>  at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
> 
> Some of the previous failures you sent me occurred when it was trying
> to read a data block, not an index block.
> 
> At this point I am not sure what else I can do.  Let me know if you
> have any more questions.
> 
> Keith
> 
> 
> On Tue, Apr 17, 2012 at 6:19 PM, Keys Botzum <kb...@maprtech.com> wrote:
>> Keith,
>> 
>> Great idea. Here are the results.
>> 
>> Run with just record 84 against Hadoop file system:
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 84  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>> 17 12:22:22,428 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:22:22,685 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:22:22,690 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:22:22,718 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
>> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: data = 84
>> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 
>> Notice no exception. When I ran it against the entire file, here's the last bit of output:
>> 
>> 17 12:21:39,605 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:21:39,605 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:21:39,612 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
>> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: data = 84
>> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
>> java.lang.reflect.InvocationTargetException
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: invalid distance too far back
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>        at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
>>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
>>        at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
>>        at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:96)
>>        ... 6 more
>> 
>> 
>> That seems to imply that the read error is just after row #84.
>> 
>> I then copied the file out of the file system onto a normal linux file system and ran the test requested:
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ cp /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf /tmp
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery file:///tmp/F000000w.rf
>> 17 12:24:02,893 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,039 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
>> 17 12:24:03,042 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,049 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
>> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: len = 1
>> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: data = 1
>> row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
>> 17 12:24:03,055 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
>> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
>> 17 12:24:03,125 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
>> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 1
>> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 9
>> row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
>> 17 12:24:03,129 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:24:03,160 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
>> 17 12:24:03,160 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,161 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,163 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
>> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: data = 12
>> row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
>> 17 12:24:03,165 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
>> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
>> 17 12:24:03,190 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
>> 17 12:24:03,190 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
>> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,193 [rfile.RelativeKey] DEBUG: data = 15
>> row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
>> 17 12:24:03,193 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
>> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
>> 17 12:24:03,240 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
>> 17 12:24:03,240 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: data = 32
>> row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
>> 17 12:24:03,243 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
>> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
>> 17 12:24:03,267 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
>> 17 12:24:03,267 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data = 35
>> row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
>> 17 12:24:03,271 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
>> 17 12:24:03,271 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
>> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
>> 17 12:24:03,294 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
>> 17 12:24:03,294 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data = 38
>> row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
>> 17 12:24:03,298 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
>> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
>> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
>> 17 12:24:03,316 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093 entries=1 key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
>> 17 12:24:03,316 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,317 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,319 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 46
>> row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
>> 17 12:24:03,321 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
>> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
>> 17 12:24:03,352 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
>> 17 12:24:03,352 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
>> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: data = 58
>> row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
>> 17 12:24:03,355 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:24:03,373 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
>> 17 12:24:03,373 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,375 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 61
>> row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
>> 17 12:24:03,383 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
>> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>> 17 12:24:03,405 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
>> 17 12:24:03,406 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 69
>> row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
>> 17 12:24:03,411 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
>> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
>> 17 12:24:03,426 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
>> 17 12:24:03,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 72
>> row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
>> 17 12:24:03,432 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
>> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,434 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
>> 17 12:24:03,462 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,462 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
>> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 84
>> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:24:03,486 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 92
>> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:24:03,491 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
>> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 17 12:24:03,504 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
>> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 95
>> row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
>> 17 12:24:03,516 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
>> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,519 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
>> 17 12:24:03,535 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
>> 17 12:24:03,535 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data = 98
>> row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
>> 17 12:24:03,545 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
>> 17 12:24:03,545 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
>> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
>> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
>> 
>> No error. Output looks the same until the exception.
>> 
>> Out of curiosity I took advantage of MapR's NFS access and diff'ed the two files:
>> diff /tmp/F000000w.rf /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>> 
>> No difference.
>> 
>> Since we know there is a record 92 after 84, I also tried this just to see what happens:
>> 
>> ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92 file:///tmp/F000000w.rf
>> 17 12:32:36,433 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:32:36,664 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:32:36,669 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 92
>> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:32:36,696 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
>> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
>> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 
>> And using the file within the Hadoop file system:
>> 
>> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>> 17 12:33:38,830 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
>> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>> 17 12:33:39,104 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:33:39,108 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 17 12:33:39,113 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:33:39,114 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: len = 2
>> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data = 92
>> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
>> 17 12:33:39,127 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
>> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: len = 131072
>> 17 12:33:39,134 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
>> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>> 
>> Looks the same to me.
>> 
>> Your standalone test client is very useful. I've run it with MapR client file system tracing enabled to see if that helps to clarify the issue. I'm sharing that with MapR Engineering - the trace isn't secret or anything, I just don't think it would mean much to you.
>> 
>> Also, what sleep in the test can I increase to avoid the race condition that leads to the "table split" error? I ask because I'm trying to run the test w/o compression hoping that might clarify a few things but the false error just confuses things.
>> 
>> If you have any additional ideas or things you want to try, please let me know. I'm more than willing to run additional tests.  I will pursue in parallel with MapR engineering.
>> 
>> Once again, thanks for your help,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 
>> 
>> 
>> On Apr 17, 2012, at 2:57 PM, Keith Turner wrote:
>> 
>>> Ok,
>>> 
>>> I modified LargeRowDirectQuery to support the local fs and optionallly
>>> lookuping an individual row like it used to.  I pushed theses changes
>>> to github.  Can you try running the following two test?
>>> 
>>>  * Lookup just row 84    with   LargeRowDirectQuery 84
>>> .../t-0000007/F000000w.rf
>>>  * Copy file to local fs and run LargeRowDirectQuery file:///tmp/F000000w.rf
>>> 
>>> The "table spit points out of range error" is just a timing issue w/
>>> the test.  I see that w/ HDFS sometimes.  There is a sleep in the test
>>> and sometimes its not long enough.   Thats ok.
>>> 
>>> Keith
>> 


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
I think it was failing for 84.  It get some data for row 84 from a
block, but its not sure if the next block contains more data for that
row (it does not) so it looks and when it does that fails.

The fact that you can query all rows in the local fs show that the
file in DFS is sounds.  Also, the fact that you can query the
individual row in DFS shows the file is sound.  The file is not
corrupt, but corruption seems to occur under certain circumstances
when reading the file.

The rows that are being looked up in the file are not in sorted order.
 So we are reading randomly from the file.  So it seems that seeking
the file around alot randomly seems to eventually leave something in
MapR in a bad state.  At this point I feel the rfile code is ok and
that it seems like the problem is w/ MapR, because it worked in local
fs and w/ single row in dfs.  What do you think, does this deduction
sound reasonable?

There is one thing that is not being printed out, that is when it
reads the rfile index.  In this case that is there its failing.  The
following information comes from the index, but we are not printing
any debug about reading the index blocks.

  .... DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 .....

You can see in the stack trace that failed that its was trying to read
an index block.

  at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)

Some of the previous failures you sent me occurred when it was trying
to read a data block, not an index block.

At this point I am not sure what else I can do.  Let me know if you
have any more questions.

Keith


On Tue, Apr 17, 2012 at 6:19 PM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
>
> Great idea. Here are the results.
>
> Run with just record 84 against Hadoop file system:
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 84  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
> 17 12:22:22,428 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:22:22,685 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:22:22,690 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:22:22,696 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:22:22,718 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: data =
> 17 12:22:22,719 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:22:22,720 [rfile.RelativeKey] DEBUG: data = 84
> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
>
> Notice no exception. When I ran it against the entire file, here's the last bit of output:
>
> 17 12:21:39,605 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:21:39,605 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:21:39,611 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:21:39,612 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: data =
> 17 12:21:39,613 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:21:39,614 [rfile.RelativeKey] DEBUG: data = 84
> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: invalid distance too far back
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>        at java.io.DataInputStream.readInt(DataInputStream.java:370)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
>        at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
>        at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
>        at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:96)
>        ... 6 more
>
>
> That seems to imply that the read error is just after row #84.
>
> I then copied the file out of the file system onto a normal linux file system and ran the test requested:
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ cp /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf /tmp
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery file:///tmp/F000000w.rf
> 17 12:24:02,893 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:24:03,039 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
> 17 12:24:03,042 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,043 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,049 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,050 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: len = 1
> 17 12:24:03,051 [rfile.RelativeKey] DEBUG: data = 1
> row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
> 17 12:24:03,055 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,056 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,061 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
> 17 12:24:03,125 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,126 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,127 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 1
> 17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 9
> row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
> 17 12:24:03,129 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,130 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,131 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:24:03,160 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
> 17 12:24:03,160 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,161 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,163 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,164 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,165 [rfile.RelativeKey] DEBUG: data = 12
> row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
> 17 12:24:03,165 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,166 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,168 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
> 17 12:24:03,190 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
> 17 12:24:03,190 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,191 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
> 17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,193 [rfile.RelativeKey] DEBUG: data = 15
> row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
> 17 12:24:03,193 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,194 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,195 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
> 17 12:24:03,240 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
> 17 12:24:03,240 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,241 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,242 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,243 [rfile.RelativeKey] DEBUG: data = 32
> row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
> 17 12:24:03,243 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,244 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q:
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,245 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
> 17 12:24:03,267 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
> 17 12:24:03,267 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,268 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,270 [rfile.RelativeKey] DEBUG: data = 35
> row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
> 17 12:24:03,271 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
> 17 12:24:03,271 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
> 17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,273 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
> 17 12:24:03,294 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
> 17 12:24:03,294 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,295 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,296 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,297 [rfile.RelativeKey] DEBUG: data = 38
> row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
> 17 12:24:03,298 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,298 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
> 17 12:24:03,299 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,300 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
> 17 12:24:03,316 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093 entries=1 key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
> 17 12:24:03,316 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,317 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,319 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 46
> row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
> 17 12:24:03,321 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,322 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,323 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
> 17 12:24:03,352 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
> 17 12:24:03,352 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,353 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,354 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,355 [rfile.RelativeKey] DEBUG: data = 58
> row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
> 17 12:24:03,355 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,356 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,357 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:24:03,373 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
> 17 12:24:03,373 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,374 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,375 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 61
> row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
> 17 12:24:03,383 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,385 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,386 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
> 17 12:24:03,405 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
> 17 12:24:03,406 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,407 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 69
> row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
> 17 12:24:03,411 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
> 17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,413 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
> 17 12:24:03,426 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
> 17 12:24:03,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,427 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 72
> row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
> 17 12:24:03,432 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,433 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,434 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
> 17 12:24:03,462 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:24:03,462 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,463 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,464 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 84
> row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:24:03,486 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,486 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 92
> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:24:03,491 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,492 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,493 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
> 17 12:24:03,504 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,505 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 95
> row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
> 17 12:24:03,516 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,517 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,518 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,519 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
> 17 12:24:03,535 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
> 17 12:24:03,535 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,536 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,538 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:24:03,539 [rfile.RelativeKey] DEBUG: data = 98
> row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
> 17 12:24:03,545 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
> 17 12:24:03,545 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
> 17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: data =
> 17 12:24:03,547 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
>
> No error. Output looks the same until the exception.
>
> Out of curiosity I took advantage of MapR's NFS access and diff'ed the two files:
> diff /tmp/F000000w.rf /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
>
> No difference.
>
> Since we know there is a record 92 after 84, I also tried this just to see what happens:
>
> ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92 file:///tmp/F000000w.rf
> 17 12:32:36,433 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:32:36,664 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:32:36,669 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:32:36,671 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,690 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 92
> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:32:36,696 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:32:36,698 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: data =
> 17 12:32:36,707 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>
> And using the file within the Hadoop file system:
>
> /opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
> 17 12:33:38,830 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
> last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
> 17 12:33:39,104 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:33:39,108 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 17 12:33:39,113 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:33:39,114 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: len = 2
> 17 12:33:39,123 [rfile.RelativeKey] DEBUG: data = 92
> row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
> 17 12:33:39,127 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 17 12:33:39,133 [rfile.RelativeKey] DEBUG: len = 131072
> 17 12:33:39,134 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: data =
> 17 12:33:39,135 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
>
> Looks the same to me.
>
> Your standalone test client is very useful. I've run it with MapR client file system tracing enabled to see if that helps to clarify the issue. I'm sharing that with MapR Engineering - the trace isn't secret or anything, I just don't think it would mean much to you.
>
> Also, what sleep in the test can I increase to avoid the race condition that leads to the "table split" error? I ask because I'm trying to run the test w/o compression hoping that might clarify a few things but the false error just confuses things.
>
> If you have any additional ideas or things you want to try, please let me know. I'm more than willing to run additional tests.  I will pursue in parallel with MapR engineering.
>
> Once again, thanks for your help,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 17, 2012, at 2:57 PM, Keith Turner wrote:
>
>> Ok,
>>
>> I modified LargeRowDirectQuery to support the local fs and optionallly
>> lookuping an individual row like it used to.  I pushed theses changes
>> to github.  Can you try running the following two test?
>>
>>  * Lookup just row 84    with   LargeRowDirectQuery 84
>> .../t-0000007/F000000w.rf
>>  * Copy file to local fs and run LargeRowDirectQuery file:///tmp/F000000w.rf
>>
>> The "table spit points out of range error" is just a timing issue w/
>> the test.  I see that w/ HDFS sometimes.  There is a sleep in the test
>> and sometimes its not long enough.   Thats ok.
>>
>> Keith
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

Great idea. Here are the results. 

Run with just record 84 against Hadoop file system:

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 84  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
17 12:22:22,428 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:22:22,685 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:22:22,690 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:22:22,696 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:22:22,696 [rfile.RelativeKey] DEBUG: len = 131072
17 12:22:22,718 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
17 12:22:22,719 [rfile.RelativeKey] DEBUG: data = 
17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
17 12:22:22,719 [rfile.RelativeKey] DEBUG: data = 
17 12:22:22,719 [rfile.RelativeKey] DEBUG: len = 0
17 12:22:22,719 [rfile.RelativeKey] DEBUG: data = 
17 12:22:22,719 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 12:22:22,720 [rfile.RelativeKey] DEBUG: len = 2
17 12:22:22,720 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false

Notice no exception. When I ran it against the entire file, here's the last bit of output:

17 12:21:39,605 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:21:39,605 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:21:39,611 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:21:39,611 [rfile.RelativeKey] DEBUG: len = 131072
17 12:21:39,612 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
17 12:21:39,613 [rfile.RelativeKey] DEBUG: data = 
17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
17 12:21:39,613 [rfile.RelativeKey] DEBUG: data = 
17 12:21:39,613 [rfile.RelativeKey] DEBUG: len = 0
17 12:21:39,613 [rfile.RelativeKey] DEBUG: data = 
17 12:21:39,613 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 12:21:39,614 [rfile.RelativeKey] DEBUG: len = 2
17 12:21:39,614 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: invalid distance too far back
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
	at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
	at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:96)
	... 6 more


That seems to imply that the read error is just after row #84.

I then copied the file out of the file system onto a normal linux file system and ran the test requested:

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ cp /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf /tmp

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery file:///tmp/F000000w.rf 
17 12:24:02,893 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:24:03,039 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
17 12:24:03,042 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,043 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,043 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,049 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 
17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,050 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,050 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,050 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,050 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,050 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
17 12:24:03,051 [rfile.RelativeKey] DEBUG: len = 1
17 12:24:03,051 [rfile.RelativeKey] DEBUG: data = 1
row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
17 12:24:03,055 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
17 12:24:03,056 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,056 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,061 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,061 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,061 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
17 12:24:03,125 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
17 12:24:03,126 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,126 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,127 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,127 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,128 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
17 12:24:03,128 [rfile.RelativeKey] DEBUG: len = 1
17 12:24:03,128 [rfile.RelativeKey] DEBUG: data = 9
row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
17 12:24:03,129 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:24:03,130 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,130 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,131 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,131 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,131 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 12:24:03,160 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
17 12:24:03,160 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,161 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,163 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,164 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,164 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,164 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
17 12:24:03,165 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,165 [rfile.RelativeKey] DEBUG: data = 12
row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
17 12:24:03,165 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
17 12:24:03,166 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,166 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,167 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,167 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,167 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,168 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,168 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,168 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,168 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
17 12:24:03,190 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 12:24:03,190 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,191 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,191 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,192 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,192 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 12:24:03,192 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,193 [rfile.RelativeKey] DEBUG: data = 15
row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 12:24:03,193 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
17 12:24:03,194 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,194 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,195 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,195 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,195 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
17 12:24:03,240 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
17 12:24:03,240 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,241 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,241 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,242 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,242 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,242 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
17 12:24:03,243 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,243 [rfile.RelativeKey] DEBUG: data = 32
row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
17 12:24:03,243 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
17 12:24:03,244 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,244 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 
17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,245 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,245 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,245 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
17 12:24:03,267 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 12:24:03,267 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,268 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,268 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,269 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,269 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,270 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,270 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
17 12:24:03,270 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,270 [rfile.RelativeKey] DEBUG: data = 35
row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 12:24:03,271 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
17 12:24:03,271 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,272 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
17 12:24:03,272 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,273 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,273 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,273 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,273 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,273 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
17 12:24:03,294 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
17 12:24:03,294 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,295 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,295 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,296 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,296 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,297 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,297 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
17 12:24:03,297 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,297 [rfile.RelativeKey] DEBUG: data = 38
row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
17 12:24:03,298 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
17 12:24:03,298 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,298 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,299 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
17 12:24:03,299 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,300 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,300 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,300 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,300 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,300 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
17 12:24:03,316 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093 entries=1 key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
17 12:24:03,316 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,317 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,319 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,320 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
17 12:24:03,320 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,320 [rfile.RelativeKey] DEBUG: data = 46
row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
17 12:24:03,321 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 12:24:03,322 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,322 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,323 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,323 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,323 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
17 12:24:03,352 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
17 12:24:03,352 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,353 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,353 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,354 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,354 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,354 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
17 12:24:03,355 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,355 [rfile.RelativeKey] DEBUG: data = 58
row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
17 12:24:03,355 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:24:03,356 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,356 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,357 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,357 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,357 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 12:24:03,373 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
17 12:24:03,373 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,374 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,374 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,375 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,376 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
17 12:24:03,376 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,376 [rfile.RelativeKey] DEBUG: data = 61
row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
17 12:24:03,383 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
17 12:24:03,385 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,385 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,386 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,386 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,386 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
17 12:24:03,405 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
17 12:24:03,406 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,407 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,407 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,407 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,408 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
17 12:24:03,408 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,408 [rfile.RelativeKey] DEBUG: data = 69
row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
17 12:24:03,411 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
17 12:24:03,412 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,412 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
17 12:24:03,412 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,413 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,413 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,413 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,413 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,413 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
17 12:24:03,426 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 12:24:03,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,427 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,427 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,428 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,428 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,428 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,429 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
17 12:24:03,429 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,429 [rfile.RelativeKey] DEBUG: data = 72
row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 12:24:03,432 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 12:24:03,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,433 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,434 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,435 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,435 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,435 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,435 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 12:24:03,462 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:24:03,462 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,463 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,463 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,464 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 12:24:03,464 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,464 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,465 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 12:24:03,465 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,465 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:24:03,486 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:24:03,486 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,486 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,487 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 12:24:03,487 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 12:24:03,488 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,488 [rfile.RelativeKey] DEBUG: data = 92
row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:24:03,491 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 12:24:03,492 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,492 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,492 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,493 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,493 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,493 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,493 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,493 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
17 12:24:03,504 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
17 12:24:03,505 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,505 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,505 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,507 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
17 12:24:03,507 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,507 [rfile.RelativeKey] DEBUG: data = 95
row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
17 12:24:03,516 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
17 12:24:03,517 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,517 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,518 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,518 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,519 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
17 12:24:03,535 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
17 12:24:03,535 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:24:03,536 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,536 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,538 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,538 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,539 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,539 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
17 12:24:03,539 [rfile.RelativeKey] DEBUG: len = 2
17 12:24:03,539 [rfile.RelativeKey] DEBUG: data = 98
row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
17 12:24:03,545 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
17 12:24:03,545 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 131072
17 12:24:03,546 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
17 12:24:03,546 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,547 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,547 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,547 [rfile.RelativeKey] DEBUG: len = 0
17 12:24:03,547 [rfile.RelativeKey] DEBUG: data = 
17 12:24:03,547 [rfile.RelativeKey] DEBUG: Read ts 1334170440112

No error. Output looks the same until the exception.

Out of curiosity I took advantage of MapR's NFS access and diff'ed the two files: 
diff /tmp/F000000w.rf /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf 

No difference.

Since we know there is a record 92 after 84, I also tried this just to see what happens:

./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92 file:///tmp/F000000w.rf 
17 12:32:36,433 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:32:36,664 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:32:36,669 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:32:36,671 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:32:36,671 [rfile.RelativeKey] DEBUG: len = 131072
17 12:32:36,690 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 12:32:36,690 [rfile.RelativeKey] DEBUG: len = 0
17 12:32:36,690 [rfile.RelativeKey] DEBUG: data = 
17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 
17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 0
17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 
17 12:32:36,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 12:32:36,691 [rfile.RelativeKey] DEBUG: len = 2
17 12:32:36,691 [rfile.RelativeKey] DEBUG: data = 92
row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:32:36,696 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 12:32:36,698 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:32:36,698 [rfile.RelativeKey] DEBUG: len = 131072
17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = 
17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = 
17 12:32:36,707 [rfile.RelativeKey] DEBUG: len = 0
17 12:32:36,707 [rfile.RelativeKey] DEBUG: data = 
17 12:32:36,707 [rfile.RelativeKey] DEBUG: Read ts 1334170440135

And using the file within the Hadoop file system:

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 92  /user/mapr/accumulo-SE-test-04-8496/tables/2/t-0000007/F000000w.rf
17 12:33:38,830 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 12:33:39,104 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:33:39,108 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 12:33:39,113 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:33:39,114 [rfile.RelativeKey] DEBUG: len = 131072
17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = 
17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
17 12:33:39,122 [rfile.RelativeKey] DEBUG: data = 
17 12:33:39,122 [rfile.RelativeKey] DEBUG: len = 0
17 12:33:39,123 [rfile.RelativeKey] DEBUG: data = 
17 12:33:39,123 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 12:33:39,123 [rfile.RelativeKey] DEBUG: len = 2
17 12:33:39,123 [rfile.RelativeKey] DEBUG: data = 92
row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 12:33:39,127 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 12:33:39,133 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 12:33:39,133 [rfile.RelativeKey] DEBUG: len = 131072
17 12:33:39,134 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
17 12:33:39,135 [rfile.RelativeKey] DEBUG: data = 
17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
17 12:33:39,135 [rfile.RelativeKey] DEBUG: data = 
17 12:33:39,135 [rfile.RelativeKey] DEBUG: len = 0
17 12:33:39,135 [rfile.RelativeKey] DEBUG: data = 
17 12:33:39,135 [rfile.RelativeKey] DEBUG: Read ts 1334170440135

Looks the same to me.

Your standalone test client is very useful. I've run it with MapR client file system tracing enabled to see if that helps to clarify the issue. I'm sharing that with MapR Engineering - the trace isn't secret or anything, I just don't think it would mean much to you.

Also, what sleep in the test can I increase to avoid the race condition that leads to the "table split" error? I ask because I'm trying to run the test w/o compression hoping that might clarify a few things but the false error just confuses things.

If you have any additional ideas or things you want to try, please let me know. I'm more than willing to run additional tests.  I will pursue in parallel with MapR engineering.

Once again, thanks for your help,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 17, 2012, at 2:57 PM, Keith Turner wrote:

> Ok,
> 
> I modified LargeRowDirectQuery to support the local fs and optionallly
> lookuping an individual row like it used to.  I pushed theses changes
> to github.  Can you try running the following two test?
> 
>  * Lookup just row 84    with   LargeRowDirectQuery 84
> .../t-0000007/F000000w.rf
>  * Copy file to local fs and run LargeRowDirectQuery file:///tmp/F000000w.rf
> 
> The "table spit points out of range error" is just a timing issue w/
> the test.  I see that w/ HDFS sometimes.  There is a sleep in the test
> and sometimes its not long enough.   Thats ok.
> 
> Keith


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
Ok,

I modified LargeRowDirectQuery to support the local fs and optionallly
lookuping an individual row like it used to.  I pushed theses changes
to github.  Can you try running the following two test?

  * Lookup just row 84    with   LargeRowDirectQuery 84
.../t-0000007/F000000w.rf
  * Copy file to local fs and run LargeRowDirectQuery file:///tmp/F000000w.rf

The "table spit points out of range error" is just a timing issue w/
the test.  I see that w/ HDFS sometimes.  There is a sleep in the test
and sometimes its not long enough.   Thats ok.

Keith

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

Once again I thank you for your help. It is much appreciated.  I ran the test you suggested on the existing table and here is the outcome. I ran it three times and the result was always exactly the same - an IOException trying to read block 84.

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery  /user/mapr/accumulo-SE-test-04-28878/tables/2/t-0000007/F000000w.rf 
17 10:10:43,981 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
first key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
last key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 10:10:44,171 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
17 10:10:44,176 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,184 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,185 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,193 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 
17 10:10:44,193 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,193 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,194 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,194 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,194 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,194 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,194 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
17 10:10:44,194 [rfile.RelativeKey] DEBUG: len = 1
17 10:10:44,194 [rfile.RelativeKey] DEBUG: data = 1
row # 1 top key : f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
17 10:10:44,200 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
17 10:10:44,207 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,207 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,208 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
17 10:10:44,208 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,209 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,209 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,209 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,209 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,209 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,209 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
17 10:10:44,425 [rfile.RFile] DEBUG: Getting block offset=2376219 csize=108006 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
17 10:10:44,426 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,433 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,433 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
17 10:10:44,434 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,434 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,434 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,434 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,435 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
17 10:10:44,435 [rfile.RelativeKey] DEBUG: len = 1
17 10:10:44,435 [rfile.RelativeKey] DEBUG: data = 9
row # 9 top key : bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
17 10:10:44,435 [rfile.RFile] DEBUG: Getting block offset=2484225 csize=107991 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 10:10:44,441 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,441 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,442 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 10:10:44,442 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,443 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,443 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,443 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,443 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,443 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,443 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 10:10:44,605 [rfile.RFile] DEBUG: Getting block offset=1080062 csize=107959 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
17 10:10:44,606 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,612 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,612 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,613 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,614 [rfile.RelativeKey] DEBUG: Read ts 1334170440112
17 10:10:44,614 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,614 [rfile.RelativeKey] DEBUG: data = 12
row # 12 top key : _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
17 10:10:44,615 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
17 10:10:44,621 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,621 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,622 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
17 10:10:44,622 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,622 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,622 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,622 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,623 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,623 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,623 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
17 10:10:44,678 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 10:10:44,678 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,685 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,685 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,686 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:10:44,686 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,686 [rfile.RelativeKey] DEBUG: data = 15
row # 15 top key : ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 10:10:44,687 [rfile.RFile] DEBUG: Getting block offset=539977 csize=108001 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
17 10:10:44,688 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,688 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,688 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
17 10:10:44,688 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,689 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,689 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,689 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,689 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,689 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,689 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
17 10:10:44,816 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
17 10:10:44,816 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,820 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,821 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,822 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,822 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
17 10:10:44,822 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,823 [rfile.RelativeKey] DEBUG: data = 32
row # 32 top key : eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
17 10:10:44,823 [rfile.RFile] DEBUG: Getting block offset=3564353 csize=107995 rsize=131092 entries=1 key=f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 7CgMi)$q*Yz[30/HbC<Jb![2PV(<zu%2... TRUNCATED : [] 1334170440101 false
17 10:10:44,828 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,828 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,829 [rfile.RelativeKey] DEBUG: data = f,RZuff_>.36RjOcx05Y1^qA'g'$@Q: 
17 10:10:44,829 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,829 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,829 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,830 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,830 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,830 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,830 [rfile.RelativeKey] DEBUG: Read ts 1334170440101
17 10:10:44,888 [rfile.RFile] DEBUG: Getting block offset=2808288 csize=107987 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 10:10:44,888 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,893 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,893 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,894 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 10:10:44,895 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,895 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,895 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,895 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,895 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,895 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,895 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
17 10:10:44,896 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,896 [rfile.RelativeKey] DEBUG: data = 35
row # 35 top key : cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 10:10:44,897 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
17 10:10:44,901 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,901 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,902 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
17 10:10:44,902 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,902 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,903 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,903 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,903 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,903 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,903 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
17 10:10:44,958 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
17 10:10:44,958 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:44,963 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,964 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,964 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
17 10:10:44,964 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,965 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,965 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,965 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
17 10:10:44,965 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:44,965 [rfile.RelativeKey] DEBUG: data = 38
row # 38 top key : `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
17 10:10:44,966 [rfile.RFile] DEBUG: Getting block offset=1620010 csize=108006 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
17 10:10:44,967 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:44,967 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
17 10:10:44,968 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,968 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,968 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:44,968 [rfile.RelativeKey] DEBUG: data = 
17 10:10:44,968 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
17 10:10:45,004 [rfile.RFile] DEBUG: Getting block offset=16 csize=107979 rsize=131093 entries=1 key=\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
17 10:10:45,004 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,013 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,013 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEc
17 10:10:45,015 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,015 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,015 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,015 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,015 [rfile.RelativeKey] DEBUG: Read ts 1334170440146
17 10:10:45,016 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,016 [rfile.RelativeKey] DEBUG: data = 46
row # 46 top key : \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334170440146 false
17 10:10:45,017 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 10:10:45,021 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,021 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,022 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
17 10:10:45,023 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,023 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,023 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,023 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,023 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,023 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,023 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
17 10:10:45,097 [rfile.RFile] DEBUG: Getting block offset=3888361 csize=107960 rsize=131093 entries=1 key=fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
17 10:10:45,097 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,101 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,101 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,102 [rfile.RelativeKey] DEBUG: data = fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl
17 10:10:45,103 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,103 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,103 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,103 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,103 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,103 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,104 [rfile.RelativeKey] DEBUG: Read ts 1334170440158
17 10:10:45,104 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,104 [rfile.RelativeKey] DEBUG: data = 58
row # 58 top key : fXK1$f#LZUJbYP@0AZ)1d H,LE>WRePl3F#2CD412Gm.-]%[:6CJe+7[a0#X%!u(... TRUNCATED : [] 1334170440158 false
17 10:10:45,105 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 10:10:45,110 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,110 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:10:45,111 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,111 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,111 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,111 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,112 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:10:45,151 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
17 10:10:45,152 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,156 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,157 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,158 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
17 10:10:45,158 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,158 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,158 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,158 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,159 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,159 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,159 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
17 10:10:45,159 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,159 [rfile.RelativeKey] DEBUG: data = 61
row # 61 top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
17 10:10:45,178 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
17 10:10:45,183 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,183 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,184 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
17 10:10:45,184 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,184 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,185 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,185 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,185 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,185 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,185 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
17 10:10:45,223 [rfile.RFile] DEBUG: Getting block offset=1188021 csize=107982 rsize=131093 entries=1 key=`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
17 10:10:45,224 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,228 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,228 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,229 [rfile.RelativeKey] DEBUG: data = `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(
17 10:10:45,229 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,229 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,230 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,230 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,230 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,230 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,230 [rfile.RelativeKey] DEBUG: Read ts 1334170440169
17 10:10:45,230 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,230 [rfile.RelativeKey] DEBUG: data = 69
row # 69 top key : `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334170440169 false
17 10:10:45,240 [rfile.RFile] DEBUG: Getting block offset=1512018 csize=107992 rsize=131093 entries=1 key=`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334170440138 false
17 10:10:45,247 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,247 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,248 [rfile.RelativeKey] DEBUG: data = `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161
17 10:10:45,248 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,248 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,248 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,248 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,248 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,249 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,249 [rfile.RelativeKey] DEBUG: Read ts 1334170440138
17 10:10:45,284 [rfile.RFile] DEBUG: Getting block offset=107995 csize=107979 rsize=131093 entries=1 key=]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 10:10:45,284 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,288 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,288 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,289 [rfile.RelativeKey] DEBUG: data = ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,F
17 10:10:45,290 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,290 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,290 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,290 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,290 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,290 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,290 [rfile.RelativeKey] DEBUG: Read ts 1334170440172
17 10:10:45,291 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,291 [rfile.RelativeKey] DEBUG: data = 72
row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 10:10:45,300 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 10:10:45,308 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,308 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,309 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:10:45,310 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,310 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,310 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,310 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,310 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,310 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,310 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:10:45,359 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 10:10:45,360 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:10:45,364 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:10:45,364 [rfile.RelativeKey] DEBUG: len = 131072
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 0
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 
17 10:10:45,366 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:10:45,366 [rfile.RelativeKey] DEBUG: len = 2
17 10:10:45,366 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: invalid distance too far back
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
	at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
	at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:83)
	... 6 more


Just to check for consistent behavior, I reran the large row test to generate a new set of tables (failed in the usual place) and then ran your new test program to read through the tablet file. It also failed in the exact same place:

/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery  /user/mapr/accumulo-SE-test-04-13876/tables/2/t-0000007/F000000w.rf 

…..
17 10:20:12,078 [rfile.RFile] DEBUG: Getting block offset=431981 csize=107996 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 10:20:12,086 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:20:12,086 [rfile.RelativeKey] DEBUG: len = 131072
17 10:20:12,087 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:20:12,087 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,087 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,087 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,088 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,088 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,088 [rfile.RelativeKey] DEBUG: data = 

17 10:20:12,088 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:20:12,134 [rfile.RFile] DEBUG: Getting block offset=3996321 csize=107990 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 10:20:12,134 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:20:12,139 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:20:12,139 [rfile.RelativeKey] DEBUG: len = 131072
17 10:20:12,140 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 0
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 
17 10:20:12,141 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:20:12,141 [rfile.RelativeKey] DEBUG: len = 2
17 10:20:12,141 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
Thread "org.apache.accumulo.server.test.functional.LargeRowDirectQuery" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: invalid distance too far back
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$IndexBlock.readFields(MultiLevelIndex.java:256)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:657)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:430)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:477)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:436)
	at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:665)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:700)
	at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:616)
	at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1026)
	at org.apache.accumulo.server.test.functional.LargeRowDirectQuery.main(LargeRowDirectQuery.java:83)
	... 6 more



Out of curiosity I ran both tests again with compression in Accumulo disabled (it is already disabled in MapR). My thought here is that perhaps compression obscures some of the issue. I cannot explain this but when run this way, the large row test consistently fails with a "table spit points out of range error" on the client side with nothing obvious on the server. It doesn't look like it even created the tablets. What's odd is that we aren't seeing the EOF error which is what we saw when running this test before without compression in Accumulo 1.4.0 (and I just reran 1.4.0 to validate I'm still getting EOF). 

The only difference I could think of other than debug information is that I did change the java heap settings for the test driver by editing TestUtils.py per Eric's recommendation. So just to be sure, I put those back in. And now I get the usual EOFException in readFully(). Interesting. This seems to imply some kind of very subtle race condition that is impacted by the Java heap settings (likely GC behavior). Now here's the weird part. I then ran your test program over the tablet you suggested and here's the result - notice it appears to succeed:

mapr@SE-test-04:/opt/keith-turner-accumulo-1.4.0-MapR-c9d24ff/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery  /user/mapr/accumulo-SE-test-04-1054/tables/2/t-0000007/F000000w.rf 
….
row # 72 top key : ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334170440172 false
17 10:53:48,483 [rfile.RFile] DEBUG: Getting block offset=524443 csize=131093 rsize=131093 entries=1 key=]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334170440115 false
17 10:53:48,487 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,487 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,487 [rfile.RelativeKey] DEBUG: data = ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1M
17 10:53:48,487 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,488 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,488 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,488 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,488 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,488 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,488 [rfile.RelativeKey] DEBUG: Read ts 1334170440115
17 10:53:48,545 [rfile.RFile] DEBUG: Getting block offset=4851096 csize=131093 rsize=131093 entries=1 key=gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 10:53:48,545 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,551 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,551 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(
17 10:53:48,552 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,552 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,552 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,552 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,552 [rfile.RelativeKey] DEBUG: Read ts 1334170440184
17 10:53:48,553 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,553 [rfile.RelativeKey] DEBUG: data = 84
row # 84 top key : gj<=a%6]#O^TbgU+n%cv1hX*e0V$#<3(0g0NDE8C?8:HVK;\cZG&RK3pvYSYr!;`... TRUNCATED : [] 1334170440184 false
17 10:53:48,588 [rfile.RFile] DEBUG: Getting block offset=3015559 csize=131093 rsize=131093 entries=1 key=c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 10:53:48,588 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,594 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,594 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_Nx
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,595 [rfile.RelativeKey] DEBUG: Read ts 1334170440192
17 10:53:48,595 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,595 [rfile.RelativeKey] DEBUG: data = 92
row # 92 top key : c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334170440192 false
17 10:53:48,605 [rfile.RFile] DEBUG: Getting block offset=3408897 csize=131093 rsize=131093 entries=1 key=cMCOJH <GRW'' .A)R0W2`C5d"97e_/?@.e{:LUzSMRsL[5J3BB"_V=keDKe?/Zk... TRUNCATED : [] 1334170440135 false
17 10:53:48,609 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = cMCOJH <GRW'' .A)R0W2`C5d"97e_/?
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,610 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,610 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,610 [rfile.RelativeKey] DEBUG: Read ts 1334170440135
17 10:53:48,651 [rfile.RFile] DEBUG: Getting block offset=1966639 csize=131093 rsize=131093 entries=1 key=a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
17 10:53:48,651 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,660 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,660 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,661 [rfile.RelativeKey] DEBUG: data = a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4
17 10:53:48,661 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,661 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,661 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,661 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,662 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,662 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,662 [rfile.RelativeKey] DEBUG: Read ts 1334170440195
17 10:53:48,662 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,662 [rfile.RelativeKey] DEBUG: data = 95
row # 95 top key : a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334170440195 false
17 10:53:48,688 [rfile.RFile] DEBUG: Getting block offset=2884467 csize=131092 rsize=131092 entries=1 key=bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334170440109 false
17 10:53:48,690 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,690 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,690 [rfile.RelativeKey] DEBUG: data = bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^
17 10:53:48,691 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,691 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,691 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,691 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,691 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,691 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,691 [rfile.RelativeKey] DEBUG: Read ts 1334170440109
17 10:53:48,728 [rfile.RFile] DEBUG: Getting block offset=655536 csize=131093 rsize=131093 entries=1 key=^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
17 10:53:48,728 [rfile.RelativeKey] DEBUG: entering fastSkip()
17 10:53:48,733 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,733 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
17 10:53:48,734 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,734 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,734 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,734 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,734 [rfile.RelativeKey] DEBUG: Read ts 1334170440198
17 10:53:48,735 [rfile.RelativeKey] DEBUG: len = 2
17 10:53:48,735 [rfile.RelativeKey] DEBUG: data = 98
row # 98 top key : ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440198 false
17 10:53:48,750 [rfile.RFile] DEBUG: Getting block offset=1311115 csize=131093 rsize=131093 entries=1 key=_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334170440112 false
17 10:53:48,754 [rfile.RelativeKey] DEBUG: fieldsSame = 0
17 10:53:48,754 [rfile.RelativeKey] DEBUG: len = 131072
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsU
17 10:53:48,755 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,755 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,755 [rfile.RelativeKey] DEBUG: len = 0
17 10:53:48,755 [rfile.RelativeKey] DEBUG: data = 
17 10:53:48,755 [rfile.RelativeKey] DEBUG: Read ts 1334170440112

If you compare this "success" case to the earlier failure, all the outputs are identical except for the block offset (which I think is related to compression) and size until the failure. Our of curiosity I ran your test program over .../tables/2/t-000000[0-8]/* and all succeeded.

As another test, I left in the Java heap settings, but allowed compression to be used (by editing TestUtils.py to remove the property I added to disable compression).  In this case I get the usual IOException "invalid distance too far back" when I run the large row auto test.  If I run your test client to scan the table it gives the same error as the first test on row #84 - java.io.IOException: invalid distance too far back.



Something very strange is happening here. There is an overwhelming amount of information I can send you, what would be useful? I can run any of the following tests against any build you like:
- compression on, heap for tests large
- compression on, heap default (small) for tests
- compression off, heap for tests large
- compression on, heap default (small) for tests

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 17, 2012, at 12:37 PM, Keith Turner wrote:

> The tablet server failed when it tried to do that read.  One thing of
> interest is that the numbers match up.  In the test program and the
> tablet server, the block offset and size is the same : "offset=3456401
> csize=107952 rsize=131093".  This rules out that the tablet server is
> simply reading data from the wrong location in the file.  However the
> tablet server failed when it tried to read the block at location
> 3456401.  This seems to show that sometimes the data can be read but
> other times it can not.
> 
> I modified the test program to lookup all rows that exist in the file.
> I pushed the changes to github.  This is more similar to what is
> happening on the tablet server.  You no longer need to pass a row # to
> the test program, just a file name.  Just run LargeRowDirectQuery
> <file>.  Can you try running this and see if an exception occurs?
> 
> Keith
> 
> On Mon, Apr 16, 2012 at 9:27 PM, Keys Botzum <kb...@maprtech.com> wrote:
>> Keith,
>> 
>> As requested I have run the new test you provided.
>> 
>> Here is the output from ./run.py -t large row -v 10 -d
>> console:
>> 
>> 
>> 
>> 
>> logs:
>> 
>> 
>> 
>> 
>> Here is the output from the new test program:
>> /opt/keith-turner-accumulo-1.4.0-MapR-630654d/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 61 /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28878/tables/2/
>> 
>> 16 18:10:08,112 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 16 18:10:08,320 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
>> 16 18:10:08,324 [rfile.RelativeKey] DEBUG: entering fastSkip()
>> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: len = 131072
>> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
>> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 2
>> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data = 61
>> top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
>> 16 18:10:08,369 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
>> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: len = 131072
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
>> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>> 
>> Is this what you expected? It seems rather brief. Should I run it differently? Out of curiosity I ran it using 60, 62, and 70 as input just to see what happens:
>> 
>> 60:
>> 
>> 16 18:13:34,076 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 
>> 62:
>> 
>> 16 18:14:07,656 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 
>> 70:
>> 
>> 16 18:15:49,995 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 
>> 
>> Not sure what this means.
>> 
>> 
>> 
>> 
>> Note that I ran the auto test twice. The reason I did so is because the first run produced an error I've never seen before. Here's the error:
>> …. Verify Call Count 2
>> ….
>> Creating Range at row 96 initial bytes are: )L#W+688U29C81-#4okb#-liSsd[!MB7
>> key = )L#W+688U29C81-#4okb#-liSsd[!MB7VO;"*nv/1LN546++1Vu(`deul$`h08,Z... TRUNCATED : [] 1334170440096 false
>> Creating Range at row 97 initial bytes are: N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXL
>> key = N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXLgd%J!jUC"1"b#gMAJH-+.R(Z\JE=j+fK... TRUNCATED : [] 1334170440097 false
>> Creating Range at row 98 initial bytes are: ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> key = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440098 false
>> Creating Range at row 99 initial bytes are: '?[3F /4g=?/#-U3IFa;.n <('vgazD9
>> key = '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M... TRUNCATED : [] 1334170440099 false
>> 16 17:58:24,809 [admin.TableOperations] INFO : Problem with metadata table, it has a hole / != * ... retrying ...
>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>> java.lang.reflect.InvocationTargetException
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.Exception: # of table splits points out of range, #splits=10 table=lr min=1 max=9
>>        at org.apache.accumulo.server.test.functional.FunctionalTest.checkSplits(FunctionalTest.java:216)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test1(LargeRowTest.java:98)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:86)
>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>        ... 6 more
>> 
>> Here's the same sequence from the run that yields the "expected" error:
>> 
>> …. Verify Call Count 2
>> ….
>> DEBUG:test.auto:out: Creating Range at row 98 initial bytes are: ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
>> DEBUG:test.auto:out: key = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440098 false
>> DEBUG:test.auto:out: Creating Range at row 99 initial bytes are: '?[3F /4g=?/#-U3IFa;.n <('vgazD9
>> DEBUG:test.auto:out: key = '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M... TRUNCATED : [] 1334170440099 false
>> DEBUG:test.auto:out: Verify Call Count 3
>> 
>> 
>> There is no corresponding error in the server logs that I can see but I've save them in case you are curious. I have the logs in case you are curious. This might be a spurious error but I include it just in case it gives you an idea.
>> 
>> In both runs there are numerous "incorrect header check" messages in the logs. That might be significant.
>> 
>> If you need anything, do not hesitate to ask.
>> 
>> Thanks again,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 
>> 
>> 
>> On Apr 16, 2012, at 6:32 PM, Keith Turner wrote:
>> 


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
The tablet server failed when it tried to do that read.  One thing of
interest is that the numbers match up.  In the test program and the
tablet server, the block offset and size is the same : "offset=3456401
csize=107952 rsize=131093".  This rules out that the tablet server is
simply reading data from the wrong location in the file.  However the
tablet server failed when it tried to read the block at location
3456401.  This seems to show that sometimes the data can be read but
other times it can not.

I modified the test program to lookup all rows that exist in the file.
 I pushed the changes to github.  This is more similar to what is
happening on the tablet server.  You no longer need to pass a row # to
the test program, just a file name.  Just run LargeRowDirectQuery
<file>.  Can you try running this and see if an exception occurs?

Keith

On Mon, Apr 16, 2012 at 9:27 PM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
>
> As requested I have run the new test you provided.
>
> Here is the output from ./run.py -t large row -v 10 -d
> console:
>
>
>
>
> logs:
>
>
>
>
> Here is the output from the new test program:
> /opt/keith-turner-accumulo-1.4.0-MapR-630654d/bin$ ./accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery 61 /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28878/tables/2/
>
> 16 18:10:08,112 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> 16 18:10:08,320 [rfile.RFile] DEBUG: Getting block offset=2916275 csize=108005 rsize=131093 entries=1 key=d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
> 16 18:10:08,324 [rfile.RelativeKey] DEBUG: entering fastSkip()
> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 16 18:10:08,332 [rfile.RelativeKey] DEBUG: len = 131072
> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: data = d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/
> 16 18:10:08,339 [rfile.RelativeKey] DEBUG: len = 0
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 0
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 0
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data =
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: Read ts 1334170440161
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: len = 2
> 16 18:10:08,340 [rfile.RelativeKey] DEBUG: data = 61
> top key : d_FH?P(Wl`V.D8A4Vyx=[L!u5V6"660/=O vON3r`>I]-J-y\f8dLv[?zmE30/"M... TRUNCATED : [] 1334170440161 false
> 16 18:10:08,369 [rfile.RFile] DEBUG: Getting block offset=3456401 csize=107952 rsize=131093 entries=1 key=eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic69c?.B@KmUDhLn?OmZ&$xgwTLc{x4if:... TRUNCATED : [] 1334170440132 false
> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: fieldsSame = 0
> 16 18:10:08,376 [rfile.RelativeKey] DEBUG: len = 131072
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data = eZ'n/^E<5GmL<8k2\zVe;4P"3nC<%2Ic
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: len = 0
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: data =
> 16 18:10:08,378 [rfile.RelativeKey] DEBUG: Read ts 1334170440132
>
> Is this what you expected? It seems rather brief. Should I run it differently? Out of curiosity I ran it using 60, 62, and 70 as input just to see what happens:
>
> 60:
>
> 16 18:13:34,076 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>
> 62:
>
> 16 18:14:07,656 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>
> 70:
>
> 16 18:15:49,995 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>
>
> Not sure what this means.
>
>
>
>
> Note that I ran the auto test twice. The reason I did so is because the first run produced an error I've never seen before. Here's the error:
> …. Verify Call Count 2
> ….
> Creating Range at row 96 initial bytes are: )L#W+688U29C81-#4okb#-liSsd[!MB7
> key = )L#W+688U29C81-#4okb#-liSsd[!MB7VO;"*nv/1LN546++1Vu(`deul$`h08,Z... TRUNCATED : [] 1334170440096 false
> Creating Range at row 97 initial bytes are: N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXL
> key = N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXLgd%J!jUC"1"b#gMAJH-+.R(Z\JE=j+fK... TRUNCATED : [] 1334170440097 false
> Creating Range at row 98 initial bytes are: ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
> key = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440098 false
> Creating Range at row 99 initial bytes are: '?[3F /4g=?/#-U3IFa;.n <('vgazD9
> key = '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M... TRUNCATED : [] 1334170440099 false
> 16 17:58:24,809 [admin.TableOperations] INFO : Problem with metadata table, it has a hole / != * ... retrying ...
> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.Exception: # of table splits points out of range, #splits=10 table=lr min=1 max=9
>        at org.apache.accumulo.server.test.functional.FunctionalTest.checkSplits(FunctionalTest.java:216)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.test1(LargeRowTest.java:98)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:86)
>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>        ... 6 more
>
> Here's the same sequence from the run that yields the "expected" error:
>
> …. Verify Call Count 2
> ….
> DEBUG:test.auto:out: Creating Range at row 98 initial bytes are: ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_M
> DEBUG:test.auto:out: key = ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334170440098 false
> DEBUG:test.auto:out: Creating Range at row 99 initial bytes are: '?[3F /4g=?/#-U3IFa;.n <('vgazD9
> DEBUG:test.auto:out: key = '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M... TRUNCATED : [] 1334170440099 false
> DEBUG:test.auto:out: Verify Call Count 3
>
>
> There is no corresponding error in the server logs that I can see but I've save them in case you are curious. I have the logs in case you are curious. This might be a spurious error but I include it just in case it gives you an idea.
>
> In both runs there are numerous "incorrect header check" messages in the logs. That might be significant.
>
> If you need anything, do not hesitate to ask.
>
> Thanks again,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 16, 2012, at 6:32 PM, Keith Turner wrote:
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

As requested I have run the new test you provided.

Here is the output from ./run.py -t large row -v 10 -d
console:



Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
I added a little program to the git repo that seek the rfiles directly
for a row.  I want to see if its possible to reproduce the problem
outside of the tablet server.   The program is called
LargeRowDirectQuery.  Looking at what you sent, the scan failed on the
61st generated row.  In the tablet server logs, I could see that it
was trying to read from
/user/mapr/accumulo-SE-test-04-15370/tables/2/t-0000007/F000000w.rf.
So you could run the following command after that test failed.

  accumulo org.apache.accumulo.server.test.functional.LargeRowDirectQuery
61 /user/mapr/accumulo-SE-test-04-15370/tables/2/t-0000007/F000000w.rf

Let me know how running this goes.

Keith

On Fri, Apr 13, 2012 at 8:22 PM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
>
> Once again, thank you for your help. I appreciate your taking the time to create a debug version with more trace.
>
> Attached is everything I think you wanted:
>
> output from running ./run.py -t large row -v 10 -d
>
>
>
>
> contents of temporary log directory:
>
>
>
>
> If needed, I can provide whatever else you might want. As I'm using your build, compression is back on. If for some reason that makes it harder for you to debug this, let me know and I can run it again with compression off.
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 13, 2012, at 7:04 PM, Keith Turner wrote:
>
>> Keys
>>
>> I created a version of Accumulo 1.4.0 with more debugging for this
>> problem on github.  If you have changes, you can send me a pull
>> request.
>>
>>    https://github.com/keith-turner/accumulo-1.4.0-MapR
>>
>>
>> If you pull this down and run it should print info in the tablet
>> server and test.  I would really like to the see the Verify call count
>> that the test prints, because verify is called multiple times in the
>> test.  So far I do not know which one of these verify calls is failing
>> for you.
>>
>>    ./run.py -t largerow -v 10 -d
>>      .
>>      .
>>      .
>>    Verify Call Count 6
>>      .
>>      .
>>      .
>>    Creating Range at row 23 initial bytes are: YlX58$iWq'57eW:[cd@?@?OF.<GHgN )
>>    key = YlX58$iWq'57eW:[cd@?@?OF.<GHgN
>> )vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : [] 1334170440123
>> false
>>
>>
>> The above is the last scan for row YIX58..., with the added debugging
>> I can go to the tserver log and see the following info about this
>> read.  I can find when a scan was started for this range and see
>> everything that rfile did (except for index reads).
>>
>>    13 18:44:55,438 [tabletserver.TabletServer] DEBUG: Starting scan,
>> range= [YlX58$iWq'57eW:[cd@?@?OF.<GHgN
>> )vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : []
>> 9223372036854775807 false,YlX58$iWq'57eW:[cd@?@?OF.<GHgN
>> )vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : []
>> 9223372036854775807 false)
>>    13 18:44:55,465 [rfile.RFile] DEBUG: Getting block offset=6480397
>> csize=107994 rsize=131093 entries=1 key=YlX58$iWq'57eW:[cd@?@?OF.<GHgN
>> )vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : [] 1334170440123
>> false
>>    13 18:44:55,466 [rfile.RelativeKey] DEBUG: entering fastSkip()
>>    13 18:44:55,467 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>>    13 18:44:55,467 [rfile.RelativeKey] DEBUG: len = 131072
>>    13 18:44:55,469 [rfile.RelativeKey] DEBUG: data =
>> YlX58$iWq'57eW:[cd@?@?OF.<GHgN )
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 0
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data =
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 0
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data =
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 0
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data =
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: Read ts 1334170440123
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 2
>>    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data = 23
>>    13 18:44:55,472 [rfile.RFile] DEBUG: Getting block offset=6588391
>> csize=107991 rsize=131093 entries=1
>> key=Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s
>> :OKl:2"cp>]yT(ZePrtEh... TRUNCATED : [] 1334170440149 false
>>    13 18:44:55,476 [rfile.RelativeKey] DEBUG: fieldsSame = 0
>>    13 18:44:55,477 [rfile.RelativeKey] DEBUG: len = 131072
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
>> Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%T
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: len = 0
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: len = 0
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: len = 0
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
>>    13 18:44:55,478 [rfile.RelativeKey] DEBUG: Read ts 1334170440149
>>    13 18:44:55,479 [data.Value] DEBUG: len = 2
>>    13 18:44:55,479 [data.Value] DEBUG: val = 49
>>    13 18:44:55,479 [tabletserver.TabletServer] DEBUG: ScanSess tid
>> 144.51.26.32:63594 2 1 entries in 0.04 secs, nbTimes = [40 40 40.00 1]
>>
>> So maybe you can run this and we can see it what it looks like for the
>> failed scan.
>>
>> Keith
>
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

Once again, thank you for your help. I appreciate your taking the time to create a debug version with more trace.

Attached is everything I think you wanted:

output from running ./run.py -t large row -v 10 -d


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
Keys

I created a version of Accumulo 1.4.0 with more debugging for this
problem on github.  If you have changes, you can send me a pull
request.

    https://github.com/keith-turner/accumulo-1.4.0-MapR


If you pull this down and run it should print info in the tablet
server and test.  I would really like to the see the Verify call count
that the test prints, because verify is called multiple times in the
test.  So far I do not know which one of these verify calls is failing
for you.

    ./run.py -t largerow -v 10 -d
      .
      .
      .
    Verify Call Count 6
      .
      .
      .
    Creating Range at row 23 initial bytes are: YlX58$iWq'57eW:[cd@?@?OF.<GHgN )
    key = YlX58$iWq'57eW:[cd@?@?OF.<GHgN
)vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : [] 1334170440123
false


The above is the last scan for row YIX58..., with the added debugging
I can go to the tserver log and see the following info about this
read.  I can find when a scan was started for this range and see
everything that rfile did (except for index reads).

    13 18:44:55,438 [tabletserver.TabletServer] DEBUG: Starting scan,
range= [YlX58$iWq'57eW:[cd@?@?OF.<GHgN
)vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : []
9223372036854775807 false,YlX58$iWq'57eW:[cd@?@?OF.<GHgN
)vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : []
9223372036854775807 false)
    13 18:44:55,465 [rfile.RFile] DEBUG: Getting block offset=6480397
csize=107994 rsize=131093 entries=1 key=YlX58$iWq'57eW:[cd@?@?OF.<GHgN
)vF2;h$?Ja%aO&]LNeFdTQQP/o1#)%t1W... TRUNCATED : [] 1334170440123
false
    13 18:44:55,466 [rfile.RelativeKey] DEBUG: entering fastSkip()
    13 18:44:55,467 [rfile.RelativeKey] DEBUG: fieldsSame = 0
    13 18:44:55,467 [rfile.RelativeKey] DEBUG: len = 131072
    13 18:44:55,469 [rfile.RelativeKey] DEBUG: data =
YlX58$iWq'57eW:[cd@?@?OF.<GHgN )
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 0
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data =
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 0
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data =
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 0
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data =
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: Read ts 1334170440123
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: len = 2
    13 18:44:55,470 [rfile.RelativeKey] DEBUG: data = 23
    13 18:44:55,472 [rfile.RFile] DEBUG: Getting block offset=6588391
csize=107991 rsize=131093 entries=1
key=Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s
:OKl:2"cp>]yT(ZePrtEh... TRUNCATED : [] 1334170440149 false
    13 18:44:55,476 [rfile.RelativeKey] DEBUG: fieldsSame = 0
    13 18:44:55,477 [rfile.RelativeKey] DEBUG: len = 131072
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%T
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: len = 0
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: len = 0
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: len = 0
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: data =
    13 18:44:55,478 [rfile.RelativeKey] DEBUG: Read ts 1334170440149
    13 18:44:55,479 [data.Value] DEBUG: len = 2
    13 18:44:55,479 [data.Value] DEBUG: val = 49
    13 18:44:55,479 [tabletserver.TabletServer] DEBUG: ScanSess tid
144.51.26.32:63594 2 1 entries in 0.04 secs, nbTimes = [40 40 40.00 1]

So maybe you can run this and we can see it what it looks like for the
failed scan.

Keith

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,


Once again, thanks for you help. Here for reference is the output of this command:
for RF in /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf; do ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF; done |grep 'false ->'|cut -c 1-64 |sort  > /tmp/sorted

As I happen to have a Mac, I used scp to copy that file to my Mac and ran md5 on it. The value was the same ( 91fae26d3b1d0ccc8b7d860a6bdb8385 ) as on the linux box.

I don't know if this is some very subtle difference in the way IO is redirected, sort works, or whatever. but hopefully the file I'm including looks like what you'd expect. Maybe diff will tell us something interesting.



Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
The md5sums of the individual files match.  I put mine inline below.
Odd that the overall md5sum did not match.  Wonder if sort is behaving
different on linux, seems unlikely.

I think a next might be to instrument the test w/ some debug to see
exactly which scan is failing.  Maybe print its range before it does
each scan.  I am curious if it always fails on the same scan.

Keith


On Thu, Apr 12, 2012 at 8:26 AM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
>
> I've run the commands you requested. I hope this is helpful to you. By the way, the reason my path looks a little different is because I'm using MapR's NFS access which makes it a lot easier to get to files.
>
> $ ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28903/tables/1/default_tablet/F0000009.rf |grep 'false ->' |cut -c 1-64 | md5sum
> Setting continuous mode
> 2012-04-12 05:08:44,2798 Program: fileclient on Host: NULL IP: 0.0.0.0, Port: 0, PID: 0
> addcfa8442914899a998d38bbf917d67  -
>
> Looks like that's the result you expect.
>
> Now for the next command:
>
> mapr@SE-test-04:/opt/accumulo-1.4.0/bin$ for RF in /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf; do ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF; done |grep 'false ->'|cut -c 1-64 |sort |md5sum
> 91fae26d3b1d0ccc8b7d860a6bdb8385  -
>
>
> As you can see this result in very different. Just to make sure I also ran this command with the same result:
> $ for RF in `hadoop fs -ls /user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf | awk '{print $NF}'` ; do ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF; done |grep 'false ->'|cut -c 1-64 |sort |md5sum
>
> In case it might be useful I checksummed each file separately:
> $ for RF in `hadoop fs -ls /user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf | awk '{print $NF}'` ; do echo $RF; ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF | grep 'false ->'|cut -c 1-64 | md5sum; done
> /user/mapr/accumulo-SE-test-04-28903/tables/2/default_tablet/F000000y.rf
> b0dda4e93f4fcc04a784dec8f8e9841d  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000000/F000000p.rf
> e0736ed51112529836253e8d0afb3253  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000001/F000000q.rf
> 008d03a0643cfdf83198c60ac9d45807  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000002/F000000s.rf
> 244f3bb7e61a30b4daed3aceb4efa7a1  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000003/F000000r.rf
> 7c8f4ff718b75051c4e6cb684689ca69  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000004/F000000t.rf
> e006faaee8f8a57c3285f7b77779d63b  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000005/F000000u.rf
> e914b210587161e71f398f325472e2bc  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000006/F000000v.rf
> 112f7d8dbe5b4cc9c971f7a7dfb56d9d  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000007/F000000w.rf
> df292cfffd3775a24b91fa56bd1f3d00  -
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000008/F000000x.rf
> a4a89e5cafbed9e89aca773f8f904b8f  -

/user/kturner/accumulo-mac1-5520/tables/2/default_tablet/F000000y.rf
b0dda4e93f4fcc04a784dec8f8e9841d
/user/kturner/accumulo-mac1-5520/tables/2/t-0000000/F000000p.rf
e0736ed51112529836253e8d0afb3253
/user/kturner/accumulo-mac1-5520/tables/2/t-0000001/F000000q.rf
008d03a0643cfdf83198c60ac9d45807
/user/kturner/accumulo-mac1-5520/tables/2/t-0000002/F000000s.rf
244f3bb7e61a30b4daed3aceb4efa7a1
/user/kturner/accumulo-mac1-5520/tables/2/t-0000003/F000000r.rf
7c8f4ff718b75051c4e6cb684689ca69
/user/kturner/accumulo-mac1-5520/tables/2/t-0000004/F000000t.rf
e006faaee8f8a57c3285f7b77779d63b
/user/kturner/accumulo-mac1-5520/tables/2/t-0000005/F000000u.rf
e914b210587161e71f398f325472e2bc
/user/kturner/accumulo-mac1-5520/tables/2/t-0000006/F000000v.rf
112f7d8dbe5b4cc9c971f7a7dfb56d9d
/user/kturner/accumulo-mac1-5520/tables/2/t-0000007/F000000w.rf
df292cfffd3775a24b91fa56bd1f3d00
/user/kturner/accumulo-mac1-5520/tables/2/t-0000008/F000000x.rf
a4a89e5cafbed9e89aca773f8f904b8f

>
> I also collected the first few lines of each file just to make sure the md5sum is on the right stuff. Does this look right to you?
> mapr@SE-test-04:/opt/accumulo-1.4.0/bin$ for RF in `hadoop fs -ls /user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf | awk '{print $NF}'` ; do echo $RF; ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF | grep 'false ->'|head; done
> /user/mapr/accumulo-SE-test-04-28903/tables/2/default_tablet/F000000y.rf
> l0\w{QJV@&S.<B8bf`#^W9aa%1"1J*$8!f2GSrAA$oGVIWqVHO<j$OM&F.54TE77... TRUNCATED : [] 1334232102926 false -> 21
> lp*C*eLap:q6/'Ut/.+t.Wt;Jk%_H=^ yiBw-1q\,]RnC('d B3W'm..WP+8[9r)... TRUNCATED : [] 1334232103163 false -> 78
> n=b\`{B!Bdv:=E:GQAhb`U_d< 6>RX$p_pJ'Gh>%>/,uI":r&g60=`]U-MSr]$i ... TRUNCATED : [] 1334232102935 false -> 18
> o;z=U'`^gr{6Z]7z"hnc-U qiT= #/7.\!"6\jNqK EC*Y#(OCO!*$eWBv+[N$_L... TRUNCATED : [] 1334232103084 false -> 44
> rHwA9Q[MVg^Jnap-hIj(5q5/#Cd{@^^%S,!mQ`.;f(\Ws#K6.[`sB5lI(MuVB^(F... TRUNCATED : [] 1334232103068 false -> 41
> sF!!vX'8{uyE/1p 9pvA^];kP/*?m5gTP9_VfbPD+v7TTnSbC6SH/Uz2=v'^3ryX... TRUNCATED : [] 1334232103090 false -> 67
> uS]C[&m^!kwa04(m$Q=$S0T?gf/F-dEeFDE,ZX[;E"%q@$5c!N{&HfdB8:?r(PIP... TRUNCATED : [] 1334232103084 false -> 64
> uoa=PnI$MC6j=O!H['C- nKGB,Fawd:=JA#>$=%b=H]gFS!+]pSHEHM)'`"-!HW%... TRUNCATED : [] 1334232102935 false -> 7
> vQeKP./>FeKGML)TQxO-4yD:8RkHF:0`CQE8[YGrfp2B!o[aJ* M5*d9Mc'QuP;p... TRUNCATED : [] 1334232103168 false -> 90
> w AY5<o*;8 JRSY"FPI').'KYwPM774v@LU:t34%WPGUFzL];,OuJX3Bj7C'*;h)... TRUNCATED : [] 1334232102940 false -> 4
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000000/F000000p.rf
>  )zUKu6Q,6Xv0>]F.7N '6=&Y)R?>oAJ1qlv#, jX<$ZQj+pXW2DPNn\z`Q;\-:A... TRUNCATED : [] 1334232102998 false -> 53
> !8`u;;$^Q1X.(*NT48T4OesPWA#"-W1q*[W"^ Y,QfPw\Ebci7GMh>6+LV7Y`Si?... TRUNCATED : [] 1334232102883 false -> 24
> "6 =0CNVv+xzEB7+a_2U0f\m(ub:ZBao' $is"=,rXMD=4ps6op.U^BWa#k{Qg0S... TRUNCATED : [] 1334232102994 false -> 50
> #4xO%KdP?9aENYtH2>yKYR\jA`N_+ul4$-<I,$1l7JJez#>w_K%!V#B6.MoNBgfA... TRUNCATED : [] 1334232102998 false -> 76
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000001/F000000q.rf
> %C?ypYJCe e.EY+/K@0x$%cH>dsfc\ENf*arht;/1`.\*FmCq+C0moW=[W_gEECW... TRUNCATED : [] 1334232103096 false -> 47
> %] zG`VVmc:!p0^LT=5IwO{o3]V]54n#f4DxF[VnIdS_$e ce(M XqIp`JGG0QQG... TRUNCATED : [] 1334232102956 false -> 16
> &A]UQ`"1B/@ebqrVxgpUM%6$kPqL4GC_c7#6!v1\RR]Eg5=%>czOZ3Ucp$YK6EcQ... TRUNCATED : [] 1334232103096 false -> 73
> '?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M... TRUNCATED : [] 1334232103172 false -> 99
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000002/F000000s.rf
> (N_S6.V@0$H!wu,OOG9&VAhs&?M#PbNYYB[;qlZ0lZIdSG=/dzd%sDuUWC/Q?8jf... TRUNCATED : [] 1334232103049 false -> 70
> (ju{+.=^\XML(47D>f8l##aX]M5D>baB]?w/;QZ*d$c$m/TAD@.Bp&X6F!%)80zb... TRUNCATED : [] 1334232102883 false -> 13
> )L#W+688U29C81-#4okb#-liSsd[!MB7VO;"*nv/1LN546++1Vu(`deul$`h08,Z... TRUNCATED : [] 1334232103147 false -> 96
> )hc' 6-39g4%1K>2kEQ7LkS>v8[ak9NMZLAePSDF)rfdNfwzmx-z]FRr[J#.)0RM... TRUNCATED : [] 1334232103042 false -> 39
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000003/F000000r.rf
> *c.`lDdN^N-')KrZ)GUU,?W{t<=^Z5ucS6.Y/G+&",_]YAT!"XHg-7`D-@7'-jd'... TRUNCATED : [] 1334232102804 false -> 10
> +u?FaLr_'\blFc;SV&TbU?d?E(a;+hT*PWotDI+0CzLz:0f7K4m'vWXkBi+=z"'f... TRUNCATED : [] 1334232102804 false -> 36
> ,Yd_k`'$D'\w8H{ZfOGd+]Z+ibg%> MJMZF!gc&1Lh&J5]C<ln./xu >RCa+$r%l... TRUNCATED : [] 1334232103121 false -> 93
> ,s/bBSX)LV^0c{DI'M*2"+@@^[XOX?#IMdC(EJ'edl9Gw{LB`k2.cwD1W6_ak"EJ... TRUNCATED : [] 1334232102984 false -> 62
> .&8IEv-*rQ]OZgPK,N;!IZq1[s4\H;{eGNi=9?'Y^: `'B!z*LV!2hW<<A!Un\`(... TRUNCATED : [] 1334232102817 false -> 33
> .@;bx!k<z8$Lq=E1IL&Z@(3'Pl).bnNU3kgZ_%luv>wy!aftzI>\yjG0A4[0YTWz... TRUNCATED : [] 1334232102817 false -> 2
> /$J/&!EE;KK#c"cKY-@2*GSk,K)0amPADoXz:@94#,?Qd1=M?'A {,AIQjK._\h`... TRUNCATED : [] 1334232102984 false -> 59
> 0"<sw)5a`Y^.$::K>UxbSGb*E6;n2D],A JQOB['DfIME A*h_(f M3ff7Y)P\<?... TRUNCATED : [] 1334232103115 false -> 85
> 11&PgKK3=Aa"x:T9DV:j{vxtC:kC!@.::fM?.6t/=41?PWX&y?A^8=798-W^T:L4... TRUNCATED : [] 1334232102978 false -> 56
> 2/:^\SmDbOf=9R!Bq5E#HbDm\%3BNsA>7+FVC8P$^&&`1FJ(FwN,%]'AMVokENSf... TRUNCATED : [] 1334232103106 false -> 82
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000004/F000000t.rf
> BDF+!ccFzK!=:iq;p*?zAA5[#@<'DI,&72Ps93/U.a$e?,r)61I_N>k\(g9VF.F1... TRUNCATED : [] 1334232103090 false -> 57
> C?WNm)i5C2,V2U"Kv+1`!)0F!X5x3EU%0xC$t'H<;08AJOD2G%o{f./ V]#,Jh.;... TRUNCATED : [] 1334232102946 false -> 28
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000005/F000000u.rf
> DQ@oN0?/h@YcO%TcGR[<JqXZ:/H+Lw8L-=$@-)HDH!U]+>>@\H?QgOggk*?#;h`1... TRUNCATED : [] 1334232103068 false -> 54
> EOd"C8I>1NNKX<!e,2J"s]<NSvx\yNg;*Jd>B+Dcio^(h-TB)$@$T[6a$S+;, _r... TRUNCATED : [] 1334232103163 false -> 80
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000006/F000000v.rf
> GJj<2FG[W5^%O(ON13]6>0cIPziuiJy-$44$"{v^c)r9,e $;`&3#`\^Q]gg/F;#... TRUNCATED : [] 1334232103024 false -> 51
> Gx@SyNCp_x7xz[JbN1N\5ZO?E+5P'!:3lQXn\b]N{Ax=n'OsC^TyjbF.VP[[vRu[... TRUNCATED : [] 1334232102846 false -> 20
> H\U)'N'_4DYfl@:o^Z1^g0bw!fOg:!> !U3c7!,#(we=US8`d<aGl$>Df*I4 Z$p... TRUNCATED : [] 1334232103133 false -> 77
> J)EU]dIpNmvV2_s,$m.T=.9<oc#UDP-TcH^\QXI6:I{k[9zR"vI3&s5a<'%N"1fd... TRUNCATED : [] 1334232102846 false -> 17
> JUNohx3g"9yD%DX;I;2tpL<68U'[VP6wsL*V+s=#B7FiUfx6BT_%)5*FMIS6)9N_... TRUNCATED : [] 1334232103036 false -> 74
> K'[F>#BG+ iL;.{\Q8J#fvk-,N1"q'*,`ieiRY;E[;Bj<(.A7Q8_o7[\QPg<[DcF... TRUNCATED : [] 1334232103018 false -> 43
> L"R:B2!WPc&T3vn*k::'FJ3+*R'0`#-vYSFaEN^^TQkiG__ZH1w`+'e#7Gk4s#-t... TRUNCATED : [] 1334232102858 false -> 14
> M4Cf#97"u]fFP2#d<u=8oJ$BW>Gv1V0gV`RRFOf[uCd0(N]nqiD],HAfLpG3d#[M... TRUNCATED : [] 1334232103004 false -> 40
> N2[#tA9K>koomJ,Ti@_S<6,9p)o%J,#hS%c%[Q"(:5]_e=/w>EUXuTX[a=c@U#Ap... TRUNCATED : [] 1334232103036 false -> 66
> N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXLgd%J!jUC"1"b#gMAJH-+.R(Z\JE=j+fK... TRUNCATED : [] 1334232103147 false -> 97
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000007/F000000w.rf
> \wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334232103049 false -> 46
> ]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334232103063 false -> 72
> ]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334232102899 false -> 15
> ^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334232103158 false -> 98
> _B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334232102899 false -> 12
> `&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334232103063 false -> 69
> `T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334232102912 false -> 38
> a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334232103158 false -> 95
> bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334232102912 false -> 9
> c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334232103153 false -> 92
> /user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000008/F000000x.rf
> iennd4L\IJ{GYSC^s&b\X;kDb4oco8!N*QPL8:^0MO"+v'GSuN>4!<TbGc#<u[td... TRUNCATED : [] 1334232103004 false -> 55
> jc$DE;4v&E%IvkvdDa;,9<:h3 =M@kfk'^va9;x0Z@!!W^JyB*!=j\bc\0INf[$L... TRUNCATED : [] 1334232103126 false -> 81
>
>
>
> This doesn't mean much to me, but based on your earlier point I think this is an issue. What else can I gather for you?
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 11, 2012, at 1:04 PM, Keith Turner wrote:
>
>> I generated the hash for the second table, and its the same as the
>> first.  Makes sense, its the same data just split differently.  The
>> reason I did the sort is because there are multiple files.  Depending
>> on where the test fails, the second table may or may not have data.
>>
>> $ for RF in `hadoop fs -ls
>> /user/kturner/accumulo-mac1-5520/tables/2/*/*.rf | awk '{print $NF}'`;
>> do ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
>> -d $RF; done | grep 'false ->' | cut -c 1-64 | sort | md5
>> addcfa8442914899a998d38bbf917d67
>>
>> Keith
>>
>> On Wed, Apr 11, 2012 at 12:35 PM, Keith Turner <ke...@deenlo.com> wrote:
>>> Keys
>>>
>>> This test uses a random number generator w/ a seed, so the test should
>>> always generate the same data.  I ran the test twice in dirty mode and
>>> then generated an md5 hash of the data.  Both times the hash was the
>>> same.  Can you try to do this and see if you get the same hash.
>>>
>>> $ ./run.py -d -t largerow
>>> $ ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
>>> -d /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
>>> | grep 'false ->' | cut -c 1-64 | md5
>>> addcfa8442914899a998d38bbf917d67
>>>
>>> I did the grep to only get key values, the print info command prints
>>> some summary info.  I did the cut inorder to just get row data,
>>> including the timestamp would make the md5sum always change. I ran
>>> this on a mac, I think on linux you will need to run md5sum.
>>>
>>> The test creates two tables.  The md5 is for first table which seems
>>> to have just one file.  I am seeing multiple files in the second
>>> table.  I will put together a command to md5sum the second table and
>>> send that shortly.
>>>
>>> $ hadoop fs -lsr /user/kturner/accumulo-mac1-5520/tables/1
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/default_tablet
>>> -rw-r--r--   3 kturner supergroup   32617935 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000a
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000b
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000c
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000d
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000e
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000f
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000g
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000h
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000i
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000j
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000k
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000l
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000m
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000n
>>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
>>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000o
>>>
>>> Keith
>>>
>>> On Wed, Apr 11, 2012 at 9:48 AM, Keys Botzum <kb...@maprtech.com> wrote:
>>>> Keith,
>>>>
>>>> Thanks for the suggestion.  I made the change to the source as you suggested and rebuilt it using Maven (surprisingly easy).
>>>>
>>>> Here's the log from tserver now. Does this help at all? I can of course provide the complete log or logs if useful to you. I can also provide the temporary tables and such if that's useful.
>>>>
>>>>
>>>> 10 15:44:07,786 [cache.LruBlockCache] DEBUG: Block cache LRU eviction completed. Freed 2494168 bytes.  Priority Sizes: Single=3.2550507MB (3413168), Multi=1
>>>> 3.89547MB (14570456),Memory=0.0MB (0)
>>>> 10 15:44:07,798 [rfile.RelativeKey] DEBUG: len : 131072
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s :OKl:2"cp>]yT(ZeP
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>>>> 10 15:44:07,799 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:52572 2 1 entries in 0.03 secs, nbTimes = [25 25 25.00 1]
>>>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 65
>>>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.tP"RsUOI
>>>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 47
>>>> 10 15:44:07,833 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-SE-test-04-18004/tables/2/t-0000000/F000000p.rf
>>>> 10 15:44:07,834 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... T
>>>> RUNCATED<
>>>> java.io.EOFException
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:381)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:135)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>
>>>>
>>>> It appears to be attempting to read 47 bytes but isn't succeeding. Out of curiosity I changed the code to read what it could and print a warning. Here's the new code version:
>>>>
>>>>
>>>> And this is a snippet of the exception which occurs with that change. Everything else is the same. As you can see my hack gets us past the read of the key, but then the next read fails.
>>>>
>>>> 11 06:42:32,254 [rfile.RelativeKey] DEBUG: data :
>>>> 11 06:42:32,254 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:47993 2 1 entries in 0.02
>>>>  secs, nbTimes = [23 23 23.00 1]
>>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 65
>>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.t
>>>> P"RsUOI
>>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 47
>>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: MISSING BYTES!!: read 45
>>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : )vRS>4 ?c>$Sgn#[QcscA!HAYcF;M_Jg3d&Jzc85$)6Y7^@^@
>>>> 11 06:42:32,288 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-
>>>> SE-test-04-32318/tables/2/t-0000000/F000000q.rf
>>>> 11 06:42:32,289 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\
>>>> ;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>>> java.io.EOFException
>>>>        at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>>        at org.apache.accumulo.core.data.Value.readFields(Value.java:156)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:585)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.j
>>>>       …..
>>>>
>>>> So it looks like we are missing quite a bit of data.
>>>>
>>>> Any help or ideas appreciated.
>>>>
>>>> Thanks,
>>>> Keys
>>>> ________________________________
>>>> Keys Botzum
>>>> Senior Principal Technologist
>>>> WW Systems Engineering
>>>> kbotzum@maprtech.com
>>>> 443-718-0098
>>>> MapR Technologies
>>>> http://www.mapr.com
>>>>
>>>>
>>>>
>>>> On Apr 10, 2012, at 5:23 PM, Keith Turner wrote:
>>>>
>>>>> Keys,
>>>>>
>>>>> Looking at the test, it writes out random rows that 128k in len.  The
>>>>> column family and column qualifier it writes out are 0 bytes long.
>>>>> When the non compression test failed, it was trying to read a column
>>>>> qualifier.  If we assume that it was reading a column qualifier from
>>>>> the test table then it should be calling readFully() with a zero
>>>>> length array.
>>>>>
>>>>> Trying to think how to debug this.  One way may be to change the code
>>>>> in RelativeKey to the following and run the test.  This will show us
>>>>> what its trying to do right before it hits the eof, but it will also
>>>>> generate a lot of noise as things scan the metadata table.
>>>>>
>>>>>  private byte[] read(DataInput in) throws IOException {
>>>>>    int len = WritableUtils.readVInt(in);
>>>>>    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
>>>>>    byte[] data = new byte[len];
>>>>>    in.readFully(data);
>>>>>    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
>>>>> new String(data).substring(0, Math.min(data.length, 60)));
>>>>>    return data;
>>>>>  }
>>>>>
>>>>> Keith
>>>>>
>>>>> On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
>>>>>> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>>>>>>
>>>>>> When I run it, this is the output I see:
>>>>>> ./run.py -t largerowtest -d -v10
>>>>>> ….
>>>>>> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
>>>>>> DEBUG:test.auto:{
>>>>>> 'tserver.compaction.major.delay':'1',
>>>>>> }
>>>>>>
>>>>>> DEBUG:test.auto:
>>>>>> INFO:test.auto:killing accumulo processes everywhere
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
>>>>>> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
>>>>>> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
>>>>>> DEBUG:test.auto:Exit code: 255
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
>>>>>> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
>>>>>> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
>>>>>> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
>>>>>> Instance name : SE-test-04-22187
>>>>>> Enter initial password for root: ******
>>>>>> Confirm initial password for root: ******
>>>>>> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>>>>>> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
>>>>>> DEBUG:test.auto:Exit code: 0
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
>>>>>> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
>>>>>> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
>>>>>> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
>>>>>> DEBUG:test.auto:Exit code: 0
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
>>>>>> DEBUG:test.auto:
>>>>>> DEBUG:test.auto:
>>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
>>>>>> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
>>>>>> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>>>>> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>> DEBUG:test.auto:err:
>>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>>>>        ... 6 more
>>>>>> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>>>>        ... 11 more
>>>>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> DEBUG:test.auto:err:
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>>>>        at $Proxy1.startScan(Unknown Source)
>>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>>>>        ... 13 more
>>>>>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>>>>> java.lang.reflect.InvocationTargetException
>>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>>>>        ... 6 more
>>>>>> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>>>>        ... 11 more
>>>>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>>>>        at $Proxy1.startScan(Unknown Source)
>>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>>>>        ... 13 more
>>>>>>
>>>>>> FAIL
>>>>>> ======================================================================
>>>>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>>>>> ----------------------------------------------------------------------
>>>>>> Traceback (most recent call last):
>>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>>>>    self.waitForStop(handle, self.maxRuntime)
>>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>>>>> AssertionError: False is not true
>>>>>>
>>>>>>
>>>>>> ======================================================================
>>>>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>>>>> ----------------------------------------------------------------------
>>>>>> Traceback (most recent call last):
>>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>>>>    self.waitForStop(handle, self.maxRuntime)
>>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>>>>> AssertionError: False is not true
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>> Ran 1 test in 43.014s
>>>>>>
>>>>>> FAILED (failures=1)
>>>>>>
>>>>>>
>>>>>> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
>>>>>> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>>>>>>
>>>>>>
>>>>>> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>>>>> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
>>>>>> java.io.IOException: invalid distance too far back
>>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>>>> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>>>        at $Proxy0.startScan(Unknown Source)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>>>        ... 15 more
>>>>>> Caused by: java.io.IOException: invalid distance too far back
>>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        ... 1 more
>>>>>>
>>>>>>
>>>>>> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>>>>>>
>>>>>> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>>>>>>
>>>>>> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
>>>>>> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
>>>>>> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>>>>> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>>>>> java.io.EOFException
>>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>>>        at $Proxy0.startScan(Unknown Source)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>>>        ... 15 more
>>>>>> Caused by: java.io.EOFException
>>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>>>        at $Proxy0.startScan(Unknown Source)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>>>        ... 15 more
>>>>>> Caused by: java.io.EOFException
>>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>        ... 1 more
>>>>>>
>>>>>>
>>>>>> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
>>>>>> 1) the file was originally written incorrectly by the writer,
>>>>>> 2) the reader is reading too far
>>>>>>
>>>>>> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>>>>>>
>>>>>> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>>>>>>
>>>>>> Thanks,
>>>>>> Keys
>>>>>> ________________________________
>>>>>> Keys Botzum
>>>>>> Senior Principal Technologist
>>>>>> WW Systems Engineering
>>>>>> kbotzum@maprtech.com
>>>>>> 443-718-0098
>>>>>> MapR Technologies
>>>>>> http://www.mapr.com
>>>>
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

I've run the commands you requested. I hope this is helpful to you. By the way, the reason my path looks a little different is because I'm using MapR's NFS access which makes it a lot easier to get to files.

$ ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28903/tables/1/default_tablet/F0000009.rf |grep 'false ->' |cut -c 1-64 | md5sum
Setting continuous mode
2012-04-12 05:08:44,2798 Program: fileclient on Host: NULL IP: 0.0.0.0, Port: 0, PID: 0
addcfa8442914899a998d38bbf917d67  -

Looks like that's the result you expect.

Now for the next command:

mapr@SE-test-04:/opt/accumulo-1.4.0/bin$ for RF in /mapr/my.cluster.com/user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf; do ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF; done |grep 'false ->'|cut -c 1-64 |sort |md5sum
91fae26d3b1d0ccc8b7d860a6bdb8385  -


As you can see this result in very different. Just to make sure I also ran this command with the same result:
$ for RF in `hadoop fs -ls /user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf | awk '{print $NF}'` ; do ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF; done |grep 'false ->'|cut -c 1-64 |sort |md5sum

In case it might be useful I checksummed each file separately:
$ for RF in `hadoop fs -ls /user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf | awk '{print $NF}'` ; do echo $RF; ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF | grep 'false ->'|cut -c 1-64 | md5sum; done
/user/mapr/accumulo-SE-test-04-28903/tables/2/default_tablet/F000000y.rf
b0dda4e93f4fcc04a784dec8f8e9841d  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000000/F000000p.rf
e0736ed51112529836253e8d0afb3253  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000001/F000000q.rf
008d03a0643cfdf83198c60ac9d45807  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000002/F000000s.rf
244f3bb7e61a30b4daed3aceb4efa7a1  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000003/F000000r.rf
7c8f4ff718b75051c4e6cb684689ca69  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000004/F000000t.rf
e006faaee8f8a57c3285f7b77779d63b  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000005/F000000u.rf
e914b210587161e71f398f325472e2bc  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000006/F000000v.rf
112f7d8dbe5b4cc9c971f7a7dfb56d9d  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000007/F000000w.rf
df292cfffd3775a24b91fa56bd1f3d00  -
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000008/F000000x.rf
a4a89e5cafbed9e89aca773f8f904b8f  -

I also collected the first few lines of each file just to make sure the md5sum is on the right stuff. Does this look right to you?
mapr@SE-test-04:/opt/accumulo-1.4.0/bin$ for RF in `hadoop fs -ls /user/mapr/accumulo-SE-test-04-28903/tables/2/*/*.rf | awk '{print $NF}'` ; do echo $RF; ./accumulo org.apache.accumulo.core.file.rfile.PrintInfo -d $RF | grep 'false ->'|head; done
/user/mapr/accumulo-SE-test-04-28903/tables/2/default_tablet/F000000y.rf
l0\w{QJV@&S.<B8bf`#^W9aa%1"1J*$8!f2GSrAA$oGVIWqVHO<j$OM&F.54TE77... TRUNCATED : [] 1334232102926 false -> 21
lp*C*eLap:q6/'Ut/.+t.Wt;Jk%_H=^ yiBw-1q\,]RnC('d B3W'm..WP+8[9r)... TRUNCATED : [] 1334232103163 false -> 78
n=b\`{B!Bdv:=E:GQAhb`U_d< 6>RX$p_pJ'Gh>%>/,uI":r&g60=`]U-MSr]$i ... TRUNCATED : [] 1334232102935 false -> 18
o;z=U'`^gr{6Z]7z"hnc-U qiT= #/7.\!"6\jNqK EC*Y#(OCO!*$eWBv+[N$_L... TRUNCATED : [] 1334232103084 false -> 44
rHwA9Q[MVg^Jnap-hIj(5q5/#Cd{@^^%S,!mQ`.;f(\Ws#K6.[`sB5lI(MuVB^(F... TRUNCATED : [] 1334232103068 false -> 41
sF!!vX'8{uyE/1p 9pvA^];kP/*?m5gTP9_VfbPD+v7TTnSbC6SH/Uz2=v'^3ryX... TRUNCATED : [] 1334232103090 false -> 67
uS]C[&m^!kwa04(m$Q=$S0T?gf/F-dEeFDE,ZX[;E"%q@$5c!N{&HfdB8:?r(PIP... TRUNCATED : [] 1334232103084 false -> 64
uoa=PnI$MC6j=O!H['C- nKGB,Fawd:=JA#>$=%b=H]gFS!+]pSHEHM)'`"-!HW%... TRUNCATED : [] 1334232102935 false -> 7
vQeKP./>FeKGML)TQxO-4yD:8RkHF:0`CQE8[YGrfp2B!o[aJ* M5*d9Mc'QuP;p... TRUNCATED : [] 1334232103168 false -> 90
w AY5<o*;8 JRSY"FPI').'KYwPM774v@LU:t34%WPGUFzL];,OuJX3Bj7C'*;h)... TRUNCATED : [] 1334232102940 false -> 4
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000000/F000000p.rf
 )zUKu6Q,6Xv0>]F.7N '6=&Y)R?>oAJ1qlv#, jX<$ZQj+pXW2DPNn\z`Q;\-:A... TRUNCATED : [] 1334232102998 false -> 53
!8`u;;$^Q1X.(*NT48T4OesPWA#"-W1q*[W"^ Y,QfPw\Ebci7GMh>6+LV7Y`Si?... TRUNCATED : [] 1334232102883 false -> 24
"6 =0CNVv+xzEB7+a_2U0f\m(ub:ZBao' $is"=,rXMD=4ps6op.U^BWa#k{Qg0S... TRUNCATED : [] 1334232102994 false -> 50
#4xO%KdP?9aENYtH2>yKYR\jA`N_+ul4$-<I,$1l7JJez#>w_K%!V#B6.MoNBgfA... TRUNCATED : [] 1334232102998 false -> 76
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000001/F000000q.rf
%C?ypYJCe e.EY+/K@0x$%cH>dsfc\ENf*arht;/1`.\*FmCq+C0moW=[W_gEECW... TRUNCATED : [] 1334232103096 false -> 47
%] zG`VVmc:!p0^LT=5IwO{o3]V]54n#f4DxF[VnIdS_$e ce(M XqIp`JGG0QQG... TRUNCATED : [] 1334232102956 false -> 16
&A]UQ`"1B/@ebqrVxgpUM%6$kPqL4GC_c7#6!v1\RR]Eg5=%>czOZ3Ucp$YK6EcQ... TRUNCATED : [] 1334232103096 false -> 73
'?[3F /4g=?/#-U3IFa;.n <('vgazD9`DY96xKZsD0:H$)CS>Q%[TG()MKR'Y<M... TRUNCATED : [] 1334232103172 false -> 99
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000002/F000000s.rf
(N_S6.V@0$H!wu,OOG9&VAhs&?M#PbNYYB[;qlZ0lZIdSG=/dzd%sDuUWC/Q?8jf... TRUNCATED : [] 1334232103049 false -> 70
(ju{+.=^\XML(47D>f8l##aX]M5D>baB]?w/;QZ*d$c$m/TAD@.Bp&X6F!%)80zb... TRUNCATED : [] 1334232102883 false -> 13
)L#W+688U29C81-#4okb#-liSsd[!MB7VO;"*nv/1LN546++1Vu(`deul$`h08,Z... TRUNCATED : [] 1334232103147 false -> 96
)hc' 6-39g4%1K>2kEQ7LkS>v8[ak9NMZLAePSDF)rfdNfwzmx-z]FRr[J#.)0RM... TRUNCATED : [] 1334232103042 false -> 39
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000003/F000000r.rf
*c.`lDdN^N-')KrZ)GUU,?W{t<=^Z5ucS6.Y/G+&",_]YAT!"XHg-7`D-@7'-jd'... TRUNCATED : [] 1334232102804 false -> 10
+u?FaLr_'\blFc;SV&TbU?d?E(a;+hT*PWotDI+0CzLz:0f7K4m'vWXkBi+=z"'f... TRUNCATED : [] 1334232102804 false -> 36
,Yd_k`'$D'\w8H{ZfOGd+]Z+ibg%> MJMZF!gc&1Lh&J5]C<ln./xu >RCa+$r%l... TRUNCATED : [] 1334232103121 false -> 93
,s/bBSX)LV^0c{DI'M*2"+@@^[XOX?#IMdC(EJ'edl9Gw{LB`k2.cwD1W6_ak"EJ... TRUNCATED : [] 1334232102984 false -> 62
.&8IEv-*rQ]OZgPK,N;!IZq1[s4\H;{eGNi=9?'Y^: `'B!z*LV!2hW<<A!Un\`(... TRUNCATED : [] 1334232102817 false -> 33
.@;bx!k<z8$Lq=E1IL&Z@(3'Pl).bnNU3kgZ_%luv>wy!aftzI>\yjG0A4[0YTWz... TRUNCATED : [] 1334232102817 false -> 2
/$J/&!EE;KK#c"cKY-@2*GSk,K)0amPADoXz:@94#,?Qd1=M?'A {,AIQjK._\h`... TRUNCATED : [] 1334232102984 false -> 59
0"<sw)5a`Y^.$::K>UxbSGb*E6;n2D],A JQOB['DfIME A*h_(f M3ff7Y)P\<?... TRUNCATED : [] 1334232103115 false -> 85
11&PgKK3=Aa"x:T9DV:j{vxtC:kC!@.::fM?.6t/=41?PWX&y?A^8=798-W^T:L4... TRUNCATED : [] 1334232102978 false -> 56
2/:^\SmDbOf=9R!Bq5E#HbDm\%3BNsA>7+FVC8P$^&&`1FJ(FwN,%]'AMVokENSf... TRUNCATED : [] 1334232103106 false -> 82
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000004/F000000t.rf
BDF+!ccFzK!=:iq;p*?zAA5[#@<'DI,&72Ps93/U.a$e?,r)61I_N>k\(g9VF.F1... TRUNCATED : [] 1334232103090 false -> 57
C?WNm)i5C2,V2U"Kv+1`!)0F!X5x3EU%0xC$t'H<;08AJOD2G%o{f./ V]#,Jh.;... TRUNCATED : [] 1334232102946 false -> 28
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000005/F000000u.rf
DQ@oN0?/h@YcO%TcGR[<JqXZ:/H+Lw8L-=$@-)HDH!U]+>>@\H?QgOggk*?#;h`1... TRUNCATED : [] 1334232103068 false -> 54
EOd"C8I>1NNKX<!e,2J"s]<NSvx\yNg;*Jd>B+Dcio^(h-TB)$@$T[6a$S+;, _r... TRUNCATED : [] 1334232103163 false -> 80
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000006/F000000v.rf
GJj<2FG[W5^%O(ON13]6>0cIPziuiJy-$44$"{v^c)r9,e $;`&3#`\^Q]gg/F;#... TRUNCATED : [] 1334232103024 false -> 51
Gx@SyNCp_x7xz[JbN1N\5ZO?E+5P'!:3lQXn\b]N{Ax=n'OsC^TyjbF.VP[[vRu[... TRUNCATED : [] 1334232102846 false -> 20
H\U)'N'_4DYfl@:o^Z1^g0bw!fOg:!> !U3c7!,#(we=US8`d<aGl$>Df*I4 Z$p... TRUNCATED : [] 1334232103133 false -> 77
J)EU]dIpNmvV2_s,$m.T=.9<oc#UDP-TcH^\QXI6:I{k[9zR"vI3&s5a<'%N"1fd... TRUNCATED : [] 1334232102846 false -> 17
JUNohx3g"9yD%DX;I;2tpL<68U'[VP6wsL*V+s=#B7FiUfx6BT_%)5*FMIS6)9N_... TRUNCATED : [] 1334232103036 false -> 74
K'[F>#BG+ iL;.{\Q8J#fvk-,N1"q'*,`ieiRY;E[;Bj<(.A7Q8_o7[\QPg<[DcF... TRUNCATED : [] 1334232103018 false -> 43
L"R:B2!WPc&T3vn*k::'FJ3+*R'0`#-vYSFaEN^^TQkiG__ZH1w`+'e#7Gk4s#-t... TRUNCATED : [] 1334232102858 false -> 14
M4Cf#97"u]fFP2#d<u=8oJ$BW>Gv1V0gV`RRFOf[uCd0(N]nqiD],HAfLpG3d#[M... TRUNCATED : [] 1334232103004 false -> 40
N2[#tA9K>koomJ,Ti@_S<6,9p)o%J,#hS%c%[Q"(:5]_e=/w>EUXuTX[a=c@U#Ap... TRUNCATED : [] 1334232103036 false -> 66
N`+y-M`C6(hjBsSoLB4xEhrU{x9+0UXLgd%J!jUC"1"b#gMAJH-+.R(Z\JE=j+fK... TRUNCATED : [] 1334232103147 false -> 97
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000007/F000000w.rf
\wH{ZUeR(*LK;s3+{l)1^ZTQr_0pTTEci^D:]c9o@3i`Orq)X91fW&n>kyK8gRs0... TRUNCATED : [] 1334232103049 false -> 46
]-e3O]_IM$p D.(cL7#$?G(X/J4A%+,Ff#5Dre%La%F20aS/%q>PD2-]$Fg6Xf><... TRUNCATED : [] 1334232103063 false -> 72
]I=:DIg/1Yg>QIiL;j!AT([DfXJ o+1MVgBz<^d]Y7me64_Fa7jUA(S[o$_]Q^Hj... TRUNCATED : [] 1334232102899 false -> 15
^+u6DeEB*3quaF)*yr*4h3p?\"?GR^_Mc0)D+g!i&_6YYOe*NMGaES=w9' /Ifd5... TRUNCATED : [] 1334232103158 false -> 98
_B W)s]l{N9OfaV(nJdC]D?(!G,I/ZsULrd?0TWes??%"FQD?O2"Z9A9VC/OF<DJ... TRUNCATED : [] 1334232102899 false -> 12
`&Q93+Z2Pv7fXFzB6t+X3bYZY:GlBZs(]v[Wg[=$ -eaysZ$`-#.\C,gfy18LD_]... TRUNCATED : [] 1334232103063 false -> 69
`T!(fz'7D\"k'yg'?q6$*DH:N3Dd\161I7]_EV#D81`O_5+YT*c%GEXn#mOZ7Pk2... TRUNCATED : [] 1334232102912 false -> 38
a8(b(3^{u(r&u^q&c?+O\b`grmd3[0P4Z;V; ];.A{tOZbnE-eRkIc:$3G'U=D'c... TRUNCATED : [] 1334232103158 false -> 95
bO!XiA85"D4lfdo8Xsc3et5FK7WZL-f^C!gc%JsQ2[[%#mYWffG;rJ(KPc4IN/^x... TRUNCATED : [] 1334232102912 false -> 9
c1S*UH[,cyT,.b-B:{F0e"L8A]VRw_NxP2cIpS64['V*F,1ug!bzbtxLvfcoF7<%... TRUNCATED : [] 1334232103153 false -> 92
/user/mapr/accumulo-SE-test-04-28903/tables/2/t-0000008/F000000x.rf
iennd4L\IJ{GYSC^s&b\X;kDb4oco8!N*QPL8:^0MO"+v'GSuN>4!<TbGc#<u[td... TRUNCATED : [] 1334232103004 false -> 55
jc$DE;4v&E%IvkvdDa;,9<:h3 =M@kfk'^va9;x0Z@!!W^JyB*!=j\bc\0INf[$L... TRUNCATED : [] 1334232103126 false -> 81



This doesn't mean much to me, but based on your earlier point I think this is an issue. What else can I gather for you?

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 11, 2012, at 1:04 PM, Keith Turner wrote:

> I generated the hash for the second table, and its the same as the
> first.  Makes sense, its the same data just split differently.  The
> reason I did the sort is because there are multiple files.  Depending
> on where the test fails, the second table may or may not have data.
> 
> $ for RF in `hadoop fs -ls
> /user/kturner/accumulo-mac1-5520/tables/2/*/*.rf | awk '{print $NF}'`;
> do ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
> -d $RF; done | grep 'false ->' | cut -c 1-64 | sort | md5
> addcfa8442914899a998d38bbf917d67
> 
> Keith
> 
> On Wed, Apr 11, 2012 at 12:35 PM, Keith Turner <ke...@deenlo.com> wrote:
>> Keys
>> 
>> This test uses a random number generator w/ a seed, so the test should
>> always generate the same data.  I ran the test twice in dirty mode and
>> then generated an md5 hash of the data.  Both times the hash was the
>> same.  Can you try to do this and see if you get the same hash.
>> 
>> $ ./run.py -d -t largerow
>> $ ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
>> -d /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
>> | grep 'false ->' | cut -c 1-64 | md5
>> addcfa8442914899a998d38bbf917d67
>> 
>> I did the grep to only get key values, the print info command prints
>> some summary info.  I did the cut inorder to just get row data,
>> including the timestamp would make the md5sum always change. I ran
>> this on a mac, I think on linux you will need to run md5sum.
>> 
>> The test creates two tables.  The md5 is for first table which seems
>> to have just one file.  I am seeing multiple files in the second
>> table.  I will put together a command to md5sum the second table and
>> send that shortly.
>> 
>> $ hadoop fs -lsr /user/kturner/accumulo-mac1-5520/tables/1
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/default_tablet
>> -rw-r--r--   3 kturner supergroup   32617935 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000a
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000b
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000c
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000d
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000e
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000f
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000g
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000h
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000i
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000j
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000k
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000l
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000m
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000n
>> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
>> /user/kturner/accumulo-mac1-5520/tables/1/t-000000o
>> 
>> Keith
>> 
>> On Wed, Apr 11, 2012 at 9:48 AM, Keys Botzum <kb...@maprtech.com> wrote:
>>> Keith,
>>> 
>>> Thanks for the suggestion.  I made the change to the source as you suggested and rebuilt it using Maven (surprisingly easy).
>>> 
>>> Here's the log from tserver now. Does this help at all? I can of course provide the complete log or logs if useful to you. I can also provide the temporary tables and such if that's useful.
>>> 
>>> 
>>> 10 15:44:07,786 [cache.LruBlockCache] DEBUG: Block cache LRU eviction completed. Freed 2494168 bytes.  Priority Sizes: Single=3.2550507MB (3413168), Multi=1
>>> 3.89547MB (14570456),Memory=0.0MB (0)
>>> 10 15:44:07,798 [rfile.RelativeKey] DEBUG: len : 131072
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s :OKl:2"cp>]yT(ZeP
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>>> 10 15:44:07,799 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:52572 2 1 entries in 0.03 secs, nbTimes = [25 25 25.00 1]
>>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 65
>>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.tP"RsUOI
>>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 47
>>> 10 15:44:07,833 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-SE-test-04-18004/tables/2/t-0000000/F000000p.rf
>>> 10 15:44:07,834 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... T
>>> RUNCATED<
>>> java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:381)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:135)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 
>>> 
>>> It appears to be attempting to read 47 bytes but isn't succeeding. Out of curiosity I changed the code to read what it could and print a warning. Here's the new code version:
>>> 
>>> 
>>> And this is a snippet of the exception which occurs with that change. Everything else is the same. As you can see my hack gets us past the read of the key, but then the next read fails.
>>> 
>>> 11 06:42:32,254 [rfile.RelativeKey] DEBUG: data :
>>> 11 06:42:32,254 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:47993 2 1 entries in 0.02
>>>  secs, nbTimes = [23 23 23.00 1]
>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 65
>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.t
>>> P"RsUOI
>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 47
>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: MISSING BYTES!!: read 45
>>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : )vRS>4 ?c>$Sgn#[QcscA!HAYcF;M_Jg3d&Jzc85$)6Y7^@^@
>>> 11 06:42:32,288 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-
>>> SE-test-04-32318/tables/2/t-0000000/F000000q.rf
>>> 11 06:42:32,289 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\
>>> ;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>> java.io.EOFException
>>>        at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>        at org.apache.accumulo.core.data.Value.readFields(Value.java:156)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:585)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.j
>>>       …..
>>> 
>>> So it looks like we are missing quite a bit of data.
>>> 
>>> Any help or ideas appreciated.
>>> 
>>> Thanks,
>>> Keys
>>> ________________________________
>>> Keys Botzum
>>> Senior Principal Technologist
>>> WW Systems Engineering
>>> kbotzum@maprtech.com
>>> 443-718-0098
>>> MapR Technologies
>>> http://www.mapr.com
>>> 
>>> 
>>> 
>>> On Apr 10, 2012, at 5:23 PM, Keith Turner wrote:
>>> 
>>>> Keys,
>>>> 
>>>> Looking at the test, it writes out random rows that 128k in len.  The
>>>> column family and column qualifier it writes out are 0 bytes long.
>>>> When the non compression test failed, it was trying to read a column
>>>> qualifier.  If we assume that it was reading a column qualifier from
>>>> the test table then it should be calling readFully() with a zero
>>>> length array.
>>>> 
>>>> Trying to think how to debug this.  One way may be to change the code
>>>> in RelativeKey to the following and run the test.  This will show us
>>>> what its trying to do right before it hits the eof, but it will also
>>>> generate a lot of noise as things scan the metadata table.
>>>> 
>>>>  private byte[] read(DataInput in) throws IOException {
>>>>    int len = WritableUtils.readVInt(in);
>>>>    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
>>>>    byte[] data = new byte[len];
>>>>    in.readFully(data);
>>>>    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
>>>> new String(data).substring(0, Math.min(data.length, 60)));
>>>>    return data;
>>>>  }
>>>> 
>>>> Keith
>>>> 
>>>> On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
>>>>> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>>>>> 
>>>>> When I run it, this is the output I see:
>>>>> ./run.py -t largerowtest -d -v10
>>>>> ….
>>>>> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
>>>>> DEBUG:test.auto:{
>>>>> 'tserver.compaction.major.delay':'1',
>>>>> }
>>>>> 
>>>>> DEBUG:test.auto:
>>>>> INFO:test.auto:killing accumulo processes everywhere
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
>>>>> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
>>>>> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
>>>>> DEBUG:test.auto:Exit code: 255
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
>>>>> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
>>>>> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
>>>>> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
>>>>> Instance name : SE-test-04-22187
>>>>> Enter initial password for root: ******
>>>>> Confirm initial password for root: ******
>>>>> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>>>>> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
>>>>> DEBUG:test.auto:Exit code: 0
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
>>>>> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
>>>>> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
>>>>> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
>>>>> DEBUG:test.auto:Exit code: 0
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
>>>>> DEBUG:test.auto:
>>>>> DEBUG:test.auto:
>>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
>>>>> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
>>>>> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>>>> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>> DEBUG:test.auto:err:
>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>>>        ... 6 more
>>>>> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>>>        ... 11 more
>>>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> DEBUG:test.auto:err:
>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>>>        at $Proxy1.startScan(Unknown Source)
>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>>>        ... 13 more
>>>>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>>>> java.lang.reflect.InvocationTargetException
>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>>>        ... 6 more
>>>>> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>>>        ... 11 more
>>>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>>>        at $Proxy1.startScan(Unknown Source)
>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>>>        ... 13 more
>>>>> 
>>>>> FAIL
>>>>> ======================================================================
>>>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>>>    self.waitForStop(handle, self.maxRuntime)
>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>>>> AssertionError: False is not true
>>>>> 
>>>>> 
>>>>> ======================================================================
>>>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>>>    self.waitForStop(handle, self.maxRuntime)
>>>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>>>> AssertionError: False is not true
>>>>> 
>>>>> ----------------------------------------------------------------------
>>>>> Ran 1 test in 43.014s
>>>>> 
>>>>> FAILED (failures=1)
>>>>> 
>>>>> 
>>>>> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
>>>>> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>>>>> 
>>>>> 
>>>>> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>>>> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
>>>>> java.io.IOException: invalid distance too far back
>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>>> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>>        at $Proxy0.startScan(Unknown Source)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>>        ... 15 more
>>>>> Caused by: java.io.IOException: invalid distance too far back
>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        ... 1 more
>>>>> 
>>>>> 
>>>>> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>>>>> 
>>>>> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>>>>> 
>>>>> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
>>>>> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
>>>>> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>>>> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>>>> java.io.EOFException
>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>>        at $Proxy0.startScan(Unknown Source)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>>        ... 15 more
>>>>> Caused by: java.io.EOFException
>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>>        at $Proxy0.startScan(Unknown Source)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>>        ... 15 more
>>>>> Caused by: java.io.EOFException
>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>        ... 1 more
>>>>> 
>>>>> 
>>>>> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
>>>>> 1) the file was originally written incorrectly by the writer,
>>>>> 2) the reader is reading too far
>>>>> 
>>>>> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>>>>> 
>>>>> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>>>>> 
>>>>> Thanks,
>>>>> Keys
>>>>> ________________________________
>>>>> Keys Botzum
>>>>> Senior Principal Technologist
>>>>> WW Systems Engineering
>>>>> kbotzum@maprtech.com
>>>>> 443-718-0098
>>>>> MapR Technologies
>>>>> http://www.mapr.com
>>> 


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
I generated the hash for the second table, and its the same as the
first.  Makes sense, its the same data just split differently.  The
reason I did the sort is because there are multiple files.  Depending
on where the test fails, the second table may or may not have data.

$ for RF in `hadoop fs -ls
/user/kturner/accumulo-mac1-5520/tables/2/*/*.rf | awk '{print $NF}'`;
do ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
-d $RF; done | grep 'false ->' | cut -c 1-64 | sort | md5
addcfa8442914899a998d38bbf917d67

Keith

On Wed, Apr 11, 2012 at 12:35 PM, Keith Turner <ke...@deenlo.com> wrote:
> Keys
>
> This test uses a random number generator w/ a seed, so the test should
> always generate the same data.  I ran the test twice in dirty mode and
> then generated an md5 hash of the data.  Both times the hash was the
> same.  Can you try to do this and see if you get the same hash.
>
> $ ./run.py -d -t largerow
> $ ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
> -d /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
> | grep 'false ->' | cut -c 1-64 | md5
> addcfa8442914899a998d38bbf917d67
>
> I did the grep to only get key values, the print info command prints
> some summary info.  I did the cut inorder to just get row data,
> including the timestamp would make the md5sum always change. I ran
> this on a mac, I think on linux you will need to run md5sum.
>
> The test creates two tables.  The md5 is for first table which seems
> to have just one file.  I am seeing multiple files in the second
> table.  I will put together a command to md5sum the second table and
> send that shortly.
>
> $ hadoop fs -lsr /user/kturner/accumulo-mac1-5520/tables/1
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/default_tablet
> -rw-r--r--   3 kturner supergroup   32617935 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000a
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000b
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000c
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000d
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000e
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000f
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000g
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000h
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000i
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000j
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000k
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000l
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000m
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000n
> drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
> /user/kturner/accumulo-mac1-5520/tables/1/t-000000o
>
> Keith
>
> On Wed, Apr 11, 2012 at 9:48 AM, Keys Botzum <kb...@maprtech.com> wrote:
>> Keith,
>>
>> Thanks for the suggestion.  I made the change to the source as you suggested and rebuilt it using Maven (surprisingly easy).
>>
>> Here's the log from tserver now. Does this help at all? I can of course provide the complete log or logs if useful to you. I can also provide the temporary tables and such if that's useful.
>>
>>
>> 10 15:44:07,786 [cache.LruBlockCache] DEBUG: Block cache LRU eviction completed. Freed 2494168 bytes.  Priority Sizes: Single=3.2550507MB (3413168), Multi=1
>> 3.89547MB (14570456),Memory=0.0MB (0)
>> 10 15:44:07,798 [rfile.RelativeKey] DEBUG: len : 131072
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s :OKl:2"cp>]yT(ZeP
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
>> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
>> 10 15:44:07,799 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:52572 2 1 entries in 0.03 secs, nbTimes = [25 25 25.00 1]
>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 65
>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.tP"RsUOI
>> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 47
>> 10 15:44:07,833 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-SE-test-04-18004/tables/2/t-0000000/F000000p.rf
>> 10 15:44:07,834 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... T
>> RUNCATED<
>> java.io.EOFException
>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:381)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:135)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>>
>>
>> It appears to be attempting to read 47 bytes but isn't succeeding. Out of curiosity I changed the code to read what it could and print a warning. Here's the new code version:
>>
>>
>> And this is a snippet of the exception which occurs with that change. Everything else is the same. As you can see my hack gets us past the read of the key, but then the next read fails.
>>
>> 11 06:42:32,254 [rfile.RelativeKey] DEBUG: data :
>> 11 06:42:32,254 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:47993 2 1 entries in 0.02
>>  secs, nbTimes = [23 23 23.00 1]
>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 65
>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.t
>> P"RsUOI
>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 47
>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: MISSING BYTES!!: read 45
>> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : )vRS>4 ?c>$Sgn#[QcscA!HAYcF;M_Jg3d&Jzc85$)6Y7^@^@
>> 11 06:42:32,288 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-
>> SE-test-04-32318/tables/2/t-0000000/F000000q.rf
>> 11 06:42:32,289 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\
>> ;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>> java.io.EOFException
>>        at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>        at org.apache.accumulo.core.data.Value.readFields(Value.java:156)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:585)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.j
>>       …..
>>
>> So it looks like we are missing quite a bit of data.
>>
>> Any help or ideas appreciated.
>>
>> Thanks,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>>
>>
>>
>> On Apr 10, 2012, at 5:23 PM, Keith Turner wrote:
>>
>>> Keys,
>>>
>>> Looking at the test, it writes out random rows that 128k in len.  The
>>> column family and column qualifier it writes out are 0 bytes long.
>>> When the non compression test failed, it was trying to read a column
>>> qualifier.  If we assume that it was reading a column qualifier from
>>> the test table then it should be calling readFully() with a zero
>>> length array.
>>>
>>> Trying to think how to debug this.  One way may be to change the code
>>> in RelativeKey to the following and run the test.  This will show us
>>> what its trying to do right before it hits the eof, but it will also
>>> generate a lot of noise as things scan the metadata table.
>>>
>>>  private byte[] read(DataInput in) throws IOException {
>>>    int len = WritableUtils.readVInt(in);
>>>    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
>>>    byte[] data = new byte[len];
>>>    in.readFully(data);
>>>    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
>>> new String(data).substring(0, Math.min(data.length, 60)));
>>>    return data;
>>>  }
>>>
>>> Keith
>>>
>>> On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
>>>> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>>>>
>>>> When I run it, this is the output I see:
>>>> ./run.py -t largerowtest -d -v10
>>>> ….
>>>> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
>>>> DEBUG:test.auto:{
>>>> 'tserver.compaction.major.delay':'1',
>>>> }
>>>>
>>>> DEBUG:test.auto:
>>>> INFO:test.auto:killing accumulo processes everywhere
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
>>>> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
>>>> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
>>>> DEBUG:test.auto:Exit code: 255
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
>>>> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
>>>> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
>>>> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
>>>> Instance name : SE-test-04-22187
>>>> Enter initial password for root: ******
>>>> Confirm initial password for root: ******
>>>> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>>>> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
>>>> DEBUG:test.auto:Exit code: 0
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
>>>> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
>>>> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
>>>> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
>>>> DEBUG:test.auto:Exit code: 0
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
>>>> DEBUG:test.auto:
>>>> DEBUG:test.auto:
>>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
>>>> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
>>>> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>>> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>> DEBUG:test.auto:err:
>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>>        ... 6 more
>>>> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>>        ... 11 more
>>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>> DEBUG:test.auto:err:
>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>>        at $Proxy1.startScan(Unknown Source)
>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>>        ... 13 more
>>>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>>> java.lang.reflect.InvocationTargetException
>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>>        ... 6 more
>>>> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>>        ... 11 more
>>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>>        at $Proxy1.startScan(Unknown Source)
>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>>        ... 13 more
>>>>
>>>> FAIL
>>>> ======================================================================
>>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>>    self.waitForStop(handle, self.maxRuntime)
>>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>>> AssertionError: False is not true
>>>>
>>>>
>>>> ======================================================================
>>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>>    self.waitForStop(handle, self.maxRuntime)
>>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>>> AssertionError: False is not true
>>>>
>>>> ----------------------------------------------------------------------
>>>> Ran 1 test in 43.014s
>>>>
>>>> FAILED (failures=1)
>>>>
>>>>
>>>> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
>>>> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>>>>
>>>>
>>>> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>>> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
>>>> java.io.IOException: invalid distance too far back
>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>        at $Proxy0.startScan(Unknown Source)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>        ... 15 more
>>>> Caused by: java.io.IOException: invalid distance too far back
>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        ... 1 more
>>>>
>>>>
>>>> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>>>>
>>>> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>>>>
>>>> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
>>>> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
>>>> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>>> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>>> java.io.EOFException
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>        at $Proxy0.startScan(Unknown Source)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>        ... 15 more
>>>> Caused by: java.io.EOFException
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>>        at $Proxy0.startScan(Unknown Source)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>>        at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>>        ... 15 more
>>>> Caused by: java.io.EOFException
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>        ... 1 more
>>>>
>>>>
>>>> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
>>>> 1) the file was originally written incorrectly by the writer,
>>>> 2) the reader is reading too far
>>>>
>>>> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>>>>
>>>> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>>>>
>>>> Thanks,
>>>> Keys
>>>> ________________________________
>>>> Keys Botzum
>>>> Senior Principal Technologist
>>>> WW Systems Engineering
>>>> kbotzum@maprtech.com
>>>> 443-718-0098
>>>> MapR Technologies
>>>> http://www.mapr.com
>>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
Keys

This test uses a random number generator w/ a seed, so the test should
always generate the same data.  I ran the test twice in dirty mode and
then generated an md5 hash of the data.  Both times the hash was the
same.  Can you try to do this and see if you get the same hash.

$ ./run.py -d -t largerow
$ ../../../bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo
-d /user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
| grep 'false ->' | cut -c 1-64 | md5
addcfa8442914899a998d38bbf917d67

I did the grep to only get key values, the print info command prints
some summary info.  I did the cut inorder to just get row data,
including the timestamp would make the md5sum always change. I ran
this on a mac, I think on linux you will need to run md5sum.

The test creates two tables.  The md5 is for first table which seems
to have just one file.  I am seeing multiple files in the second
table.  I will put together a command to md5sum the second table and
send that shortly.

$ hadoop fs -lsr /user/kturner/accumulo-mac1-5520/tables/1
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/default_tablet
-rw-r--r--   3 kturner supergroup   32617935 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/default_tablet/F0000009.rf
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000a
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000b
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000c
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000d
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000e
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000f
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000g
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000h
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000i
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000j
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000k
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000l
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:11
/user/kturner/accumulo-mac1-5520/tables/1/t-000000m
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
/user/kturner/accumulo-mac1-5520/tables/1/t-000000n
drwxr-xr-x   - kturner supergroup          0 2012-04-11 12:12
/user/kturner/accumulo-mac1-5520/tables/1/t-000000o

Keith

On Wed, Apr 11, 2012 at 9:48 AM, Keys Botzum <kb...@maprtech.com> wrote:
> Keith,
>
> Thanks for the suggestion.  I made the change to the source as you suggested and rebuilt it using Maven (surprisingly easy).
>
> Here's the log from tserver now. Does this help at all? I can of course provide the complete log or logs if useful to you. I can also provide the temporary tables and such if that's useful.
>
>
> 10 15:44:07,786 [cache.LruBlockCache] DEBUG: Block cache LRU eviction completed. Freed 2494168 bytes.  Priority Sizes: Single=3.2550507MB (3413168), Multi=1
> 3.89547MB (14570456),Memory=0.0MB (0)
> 10 15:44:07,798 [rfile.RelativeKey] DEBUG: len : 131072
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s :OKl:2"cp>]yT(ZeP
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
> 10 15:44:07,799 [rfile.RelativeKey] DEBUG: data :
> 10 15:44:07,799 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:52572 2 1 entries in 0.03 secs, nbTimes = [25 25 25.00 1]
> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 65
> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.tP"RsUOI
> 10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 47
> 10 15:44:07,833 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-SE-test-04-18004/tables/2/t-0000000/F000000p.rf
> 10 15:44:07,834 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... T
> RUNCATED<
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:381)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:135)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>
>
> It appears to be attempting to read 47 bytes but isn't succeeding. Out of curiosity I changed the code to read what it could and print a warning. Here's the new code version:
>
>
> And this is a snippet of the exception which occurs with that change. Everything else is the same. As you can see my hack gets us past the read of the key, but then the next read fails.
>
> 11 06:42:32,254 [rfile.RelativeKey] DEBUG: data :
> 11 06:42:32,254 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:47993 2 1 entries in 0.02
>  secs, nbTimes = [23 23 23.00 1]
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 65
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.t
> P"RsUOI
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 47
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: MISSING BYTES!!: read 45
> 11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : )vRS>4 ?c>$Sgn#[QcscA!HAYcF;M_Jg3d&Jzc85$)6Y7^@^@
> 11 06:42:32,288 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-
> SE-test-04-32318/tables/2/t-0000000/F000000q.rf
> 11 06:42:32,289 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\
> ;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
> java.io.EOFException
>        at java.io.DataInputStream.readInt(DataInputStream.java:375)
>        at org.apache.accumulo.core.data.Value.readFields(Value.java:156)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:585)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.j
>       …..
>
> So it looks like we are missing quite a bit of data.
>
> Any help or ideas appreciated.
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 10, 2012, at 5:23 PM, Keith Turner wrote:
>
>> Keys,
>>
>> Looking at the test, it writes out random rows that 128k in len.  The
>> column family and column qualifier it writes out are 0 bytes long.
>> When the non compression test failed, it was trying to read a column
>> qualifier.  If we assume that it was reading a column qualifier from
>> the test table then it should be calling readFully() with a zero
>> length array.
>>
>> Trying to think how to debug this.  One way may be to change the code
>> in RelativeKey to the following and run the test.  This will show us
>> what its trying to do right before it hits the eof, but it will also
>> generate a lot of noise as things scan the metadata table.
>>
>>  private byte[] read(DataInput in) throws IOException {
>>    int len = WritableUtils.readVInt(in);
>>    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
>>    byte[] data = new byte[len];
>>    in.readFully(data);
>>    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
>> new String(data).substring(0, Math.min(data.length, 60)));
>>    return data;
>>  }
>>
>> Keith
>>
>> On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
>>> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>>>
>>> When I run it, this is the output I see:
>>> ./run.py -t largerowtest -d -v10
>>> ….
>>> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
>>> DEBUG:test.auto:{
>>> 'tserver.compaction.major.delay':'1',
>>> }
>>>
>>> DEBUG:test.auto:
>>> INFO:test.auto:killing accumulo processes everywhere
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
>>> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
>>> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
>>> DEBUG:test.auto:Exit code: 255
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
>>> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
>>> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
>>> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
>>> Instance name : SE-test-04-22187
>>> Enter initial password for root: ******
>>> Confirm initial password for root: ******
>>> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>>> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
>>> DEBUG:test.auto:Exit code: 0
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
>>> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
>>> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
>>> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
>>> DEBUG:test.auto:Exit code: 0
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
>>> DEBUG:test.auto:
>>> DEBUG:test.auto:
>>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
>>> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
>>> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>> DEBUG:test.auto:err:
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>        ... 6 more
>>> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>        ... 11 more
>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>> DEBUG:test.auto:err:
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>        at $Proxy1.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>        ... 13 more
>>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>>> java.lang.reflect.InvocationTargetException
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>>        ... 6 more
>>> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>>        ... 11 more
>>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>>        at $Proxy1.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>>        ... 13 more
>>>
>>> FAIL
>>> ======================================================================
>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>    self.waitForStop(handle, self.maxRuntime)
>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>> AssertionError: False is not true
>>>
>>>
>>> ======================================================================
>>> FAIL: runTest (simple.largeRow.LargeRowTest)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>>    self.waitForStop(handle, self.maxRuntime)
>>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>>    self.assert_(self.processResult(out, err, handle.returncode))
>>> AssertionError: False is not true
>>>
>>> ----------------------------------------------------------------------
>>> Ran 1 test in 43.014s
>>>
>>> FAILED (failures=1)
>>>
>>>
>>> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
>>> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>>>
>>>
>>> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
>>> java.io.IOException: invalid distance too far back
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>        at $Proxy0.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>        ... 15 more
>>> Caused by: java.io.IOException: invalid distance too far back
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        ... 1 more
>>>
>>>
>>> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>>>
>>> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>>>
>>> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
>>> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
>>> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>>> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>>> java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>        at $Proxy0.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>        ... 15 more
>>> Caused by: java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>>        at $Proxy0.startScan(Unknown Source)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>>        ... 15 more
>>> Caused by: java.io.EOFException
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>        ... 1 more
>>>
>>>
>>> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
>>> 1) the file was originally written incorrectly by the writer,
>>> 2) the reader is reading too far
>>>
>>> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>>>
>>> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>>>
>>> Thanks,
>>> Keys
>>> ________________________________
>>> Keys Botzum
>>> Senior Principal Technologist
>>> WW Systems Engineering
>>> kbotzum@maprtech.com
>>> 443-718-0098
>>> MapR Technologies
>>> http://www.mapr.com
>

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
Keith,

Thanks for the suggestion.  I made the change to the source as you suggested and rebuilt it using Maven (surprisingly easy).

Here's the log from tserver now. Does this help at all? I can of course provide the complete log or logs if useful to you. I can also provide the temporary tables and such if that's useful.


10 15:44:07,786 [cache.LruBlockCache] DEBUG: Block cache LRU eviction completed. Freed 2494168 bytes.  Priority Sizes: Single=3.2550507MB (3413168), Multi=1
3.89547MB (14570456),Memory=0.0MB (0)
10 15:44:07,798 [rfile.RelativeKey] DEBUG: len : 131072
10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : Z"?7-,mE:5Di&ou.4/4.i+9zGo0K8%%TsSt#!&a!&s :OKl:2"cp>]yT(ZeP
10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : 
10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : 
10 15:44:07,799 [rfile.RelativeKey] DEBUG: len : 0
10 15:44:07,799 [rfile.RelativeKey] DEBUG: data : 
10 15:44:07,799 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:52572 2 1 entries in 0.03 secs, nbTimes = [25 25 25.00 1] 
10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 65
10 15:44:07,828 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.tP"RsUOI
10 15:44:07,828 [rfile.RelativeKey] DEBUG: len : 47
10 15:44:07,833 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-SE-test-04-18004/tables/2/t-0000000/F000000p.rf
10 15:44:07,834 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... T
RUNCATED<
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:381)
        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:135)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


It appears to be attempting to read 47 bytes but isn't succeeding. Out of curiosity I changed the code to read what it could and print a warning. Here's the new code version:


And this is a snippet of the exception which occurs with that change. Everything else is the same. As you can see my hack gets us past the read of the key, but then the next read fails.

11 06:42:32,254 [rfile.RelativeKey] DEBUG: data : 
11 06:42:32,254 [tabletserver.TabletServer] DEBUG: ScanSess tid 10.250.99.204:47993 2 1 entries in 0.02
 secs, nbTimes = [23 23 23.00 1] 
11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 65
11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : SjC)$OQ7U!9Ng:i1#2Sxl.a"(d=Js>d!1u`)WAFAs{>n=7H<tM?.t
P"RsUOI
11 06:42:32,283 [rfile.RelativeKey] DEBUG: len : 47
11 06:42:32,283 [rfile.RelativeKey] DEBUG: MISSING BYTES!!: read 45
11 06:42:32,283 [rfile.RelativeKey] DEBUG: data : )vRS>4 ?c>$Sgn#[QcscA!HAYcF;M_Jg3d&Jzc85$)6Y7^@^@
11 06:42:32,288 [problems.ProblemReports] DEBUG: Filing problem report 2 FILE_READ /user/mapr/accumulo-
SE-test-04-32318/tables/2/t-0000000/F000000q.rf
11 06:42:32,289 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\
;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:375)
        at org.apache.accumulo.core.data.Value.readFields(Value.java:156)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:585)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.j
       …..

So it looks like we are missing quite a bit of data.

Any help or ideas appreciated.

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 10, 2012, at 5:23 PM, Keith Turner wrote:

> Keys,
> 
> Looking at the test, it writes out random rows that 128k in len.  The
> column family and column qualifier it writes out are 0 bytes long.
> When the non compression test failed, it was trying to read a column
> qualifier.  If we assume that it was reading a column qualifier from
> the test table then it should be calling readFully() with a zero
> length array.
> 
> Trying to think how to debug this.  One way may be to change the code
> in RelativeKey to the following and run the test.  This will show us
> what its trying to do right before it hits the eof, but it will also
> generate a lot of noise as things scan the metadata table.
> 
>  private byte[] read(DataInput in) throws IOException {
>    int len = WritableUtils.readVInt(in);
>    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
>    byte[] data = new byte[len];
>    in.readFully(data);
>    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
> new String(data).substring(0, Math.min(data.length, 60)));
>    return data;
>  }
> 
> Keith
> 
> On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
>> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>> 
>> When I run it, this is the output I see:
>> ./run.py -t largerowtest -d -v10
>> ….
>> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
>> DEBUG:test.auto:{
>> 'tserver.compaction.major.delay':'1',
>> }
>> 
>> DEBUG:test.auto:
>> INFO:test.auto:killing accumulo processes everywhere
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
>> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
>> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
>> DEBUG:test.auto:Exit code: 255
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
>> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
>> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
>> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
>> Instance name : SE-test-04-22187
>> Enter initial password for root: ******
>> Confirm initial password for root: ******
>> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
>> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
>> DEBUG:test.auto:Exit code: 0
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
>> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
>> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
>> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
>> DEBUG:test.auto:Exit code: 0
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
>> DEBUG:test.auto:
>> DEBUG:test.auto:
>> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
>> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
>> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>> DEBUG:test.auto:err:
>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>        ... 6 more
>> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>        ... 11 more
>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>> DEBUG:test.auto:err:
>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>        at $Proxy1.startScan(Unknown Source)
>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>        ... 13 more
>> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
>> java.lang.reflect.InvocationTargetException
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>>        ... 6 more
>> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>>        ... 11 more
>> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>>        at $Proxy1.startScan(Unknown Source)
>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>>        ... 13 more
>> 
>> FAIL
>> ======================================================================
>> FAIL: runTest (simple.largeRow.LargeRowTest)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>    self.waitForStop(handle, self.maxRuntime)
>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>    self.assert_(self.processResult(out, err, handle.returncode))
>> AssertionError: False is not true
>> 
>> 
>> ======================================================================
>> FAIL: runTest (simple.largeRow.LargeRowTest)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>>    self.waitForStop(handle, self.maxRuntime)
>>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>>    self.assert_(self.processResult(out, err, handle.returncode))
>> AssertionError: False is not true
>> 
>> ----------------------------------------------------------------------
>> Ran 1 test in 43.014s
>> 
>> FAILED (failures=1)
>> 
>> 
>> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
>> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>> 
>> 
>> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
>> java.io.IOException: invalid distance too far back
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>        at $Proxy0.startScan(Unknown Source)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>        ... 15 more
>> Caused by: java.io.IOException: invalid distance too far back
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        ... 1 more
>> 
>> 
>> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>> 
>> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>> 
>> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
>> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
>> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
>> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
>> java.io.EOFException
>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>        at $Proxy0.startScan(Unknown Source)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>        ... 15 more
>> Caused by: java.io.EOFException
>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
>> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>>        at $Proxy0.startScan(Unknown Source)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>>        ... 15 more
>> Caused by: java.io.EOFException
>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        ... 1 more
>> 
>> 
>> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
>> 1) the file was originally written incorrectly by the writer,
>> 2) the reader is reading too far
>> 
>> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>> 
>> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>> 
>> Thanks,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com


Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keith Turner <ke...@deenlo.com>.
Keys,

Looking at the test, it writes out random rows that 128k in len.  The
column family and column qualifier it writes out are 0 bytes long.
When the non compression test failed, it was trying to read a column
qualifier.  If we assume that it was reading a column qualifier from
the test table then it should be calling readFully() with a zero
length array.

Trying to think how to debug this.  One way may be to change the code
in RelativeKey to the following and run the test.  This will show us
what its trying to do right before it hits the eof, but it will also
generate a lot of noise as things scan the metadata table.

  private byte[] read(DataInput in) throws IOException {
    int len = WritableUtils.readVInt(in);
    Logger.getLogger(RelativeKey.class.getName()).debug("len : " + len);
    byte[] data = new byte[len];
    in.readFully(data);
    Logger.getLogger(RelativeKey.class.getName()).debug("data : " +
new String(data).substring(0, Math.min(data.length, 60)));
    return data;
  }

Keith

On Tue, Apr 10, 2012 at 2:08 PM, Keys Botzum <kb...@maprtech.com> wrote:
> At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest
>
> When I run it, this is the output I see:
> ./run.py -t largerowtest -d -v10
> ….
> 09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
> DEBUG:test.auto:{
> 'tserver.compaction.major.delay':'1',
> }
>
> DEBUG:test.auto:
> INFO:test.auto:killing accumulo processes everywhere
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
> DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
> INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
> DEBUG:test.auto:Exit code: 255
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
> DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
> 10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
> 10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
> Instance name : SE-test-04-22187
> Enter initial password for root: ******
> Confirm initial password for root: ******
> 10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
> 10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
> DEBUG:test.auto:Exit code: 0
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
> DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
> 10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
> 10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
> DEBUG:test.auto:Exit code: 0
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
> DEBUG:test.auto:
> DEBUG:test.auto:
> DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
> DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
> DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> DEBUG:test.auto:err:    at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
> DEBUG:test.auto:err:
>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>        ... 6 more
> DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>        ... 11 more
> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
> DEBUG:test.auto:err:
>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>        at $Proxy1.startScan(Unknown Source)
>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>        ... 13 more
> ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.start.Main$1.run(Main.java:89)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
>        at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
>        at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
>        ... 6 more
> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
>        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
>        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
>        ... 11 more
> Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
>        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>        at $Proxy1.startScan(Unknown Source)
>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>        ... 13 more
>
> FAIL
> ======================================================================
> FAIL: runTest (simple.largeRow.LargeRowTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>    self.waitForStop(handle, self.maxRuntime)
>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>    self.assert_(self.processResult(out, err, handle.returncode))
> AssertionError: False is not true
>
>
> ======================================================================
> FAIL: runTest (simple.largeRow.LargeRowTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
>    self.waitForStop(handle, self.maxRuntime)
>  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
>    self.assert_(self.processResult(out, err, handle.returncode))
> AssertionError: False is not true
>
> ----------------------------------------------------------------------
> Ran 1 test in 43.014s
>
> FAILED (failures=1)
>
>
> The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
> Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run.
>
>
> 09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
> 09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
> java.io.IOException: invalid distance too far back
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> 09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
> 09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>        at $Proxy0.startScan(Unknown Source)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>        ... 15 more
> Caused by: java.io.IOException: invalid distance too far back
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
>        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>        at java.io.FilterInputStream.read(FilterInputStream.java:66)
>        at java.io.DataInputStream.readByte(DataInputStream.java:248)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        ... 1 more
>
>
> After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.
>
> When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:
>
> 10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
> 10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
> 10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
> 10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>        at $Proxy0.startScan(Unknown Source)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>        ... 15 more
> Caused by: java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> 10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
> 10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>        at $Proxy0.startScan(Unknown Source)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
>        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
>        ... 15 more
> Caused by: java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
>        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
>        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
>        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
>        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
>        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
>        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
>        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
>        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
>        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
>        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
>        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
>        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        ... 1 more
>
>
> So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas:
> 1) the file was originally written incorrectly by the writer,
> 2) the reader is reading too far
>
> This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.
>
> If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.
>
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com

Re: Accumulo on MapR Continued - LargeRowTest

Posted by Keys Botzum <kb...@maprtech.com>.
At this point all but two of the Accumulo test/system/auto tests have completed successfully. This test is failing and I'm not quite sure why: org.apache.accumulo.server.test.functional.LargeRowTest

When I run it, this is the output I see:
./run.py -t largerowtest -d -v10
….
09:45:18 runTest (simple.largeRow.LargeRowTest) ............................. DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest getConfig
DEBUG:test.auto:{
'tserver.compaction.major.delay':'1',
}

DEBUG:test.auto:
INFO:test.auto:killing accumulo processes everywhere
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000 SE-test-04-22187.*org.apache.accumulo.start
DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
INFO:test.auto:Error output from command: rmr: cannot remove /user/mapr/accumulo-SE-test-04-22187: No such file or directory.
DEBUG:test.auto:Exit code: 255
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init --clear-instance-name
DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO : Hadoop Filesystem is maprfs:///
10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is /user/mapr/accumulo-SE-test-04-22187
10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
Instance name : SE-test-04-22187
Enter initial password for root: ******
Confirm initial password for root: ******
10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with username: root at the request of user !SYSTEM
DEBUG:test.auto:Exit code: 0
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.master.state.SetGoalState NORMAL
DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO : Attempting to talk to zookeeper
10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
DEBUG:test.auto:Exit code: 0
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest setup
DEBUG:test.auto:
DEBUG:test.auto:
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run
DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root -p secret -i SE-test-04-22187 org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
DEBUG:test.auto:err: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
DEBUG:test.auto:err: 	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
DEBUG:test.auto:err: 
	at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
	at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
	at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
	at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
	at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
	at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
	... 6 more
DEBUG:test.auto:err: Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
	at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
	at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
	... 11 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
	at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
DEBUG:test.auto:err: 
	at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
	at $Proxy1.startScan(Unknown Source)
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
	... 13 more
ERROR:test.auto:This looks like a stack trace: Thread "org.apache.accumulo.server.test.functional.FunctionalTest" died null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.start.Main$1.run(Main.java:89)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
	at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
	at org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
	at org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
	at org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
	at org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
	at org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
	... 6 more
Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.250.99.204:39253
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
	at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
	at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
	... 11 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
	at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
	at $Proxy1.startScan(Unknown Source)
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
	... 13 more

FAIL
======================================================================
FAIL: runTest (simple.largeRow.LargeRowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
    self.waitForStop(handle, self.maxRuntime)
  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
    self.assert_(self.processResult(out, err, handle.returncode))
AssertionError: False is not true


======================================================================
FAIL: runTest (simple.largeRow.LargeRowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
    self.waitForStop(handle, self.maxRuntime)
  File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in waitForStop
    self.assert_(self.processResult(out, err, handle.returncode))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 43.014s

FAILED (failures=1)


The only log that seems to have any relevant output is the tserver_xxxx.log file. In it I found this error:
Note that the timestamps here do not match the previous timestamps. This is just because I forgot to capture the data from the run that corresponds exactly to this run. 


09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF _^<q\\M`X\\Er... TRUNCATED
java.io.IOException: invalid distance too far back
        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy0.startScan(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: invalid distance too far back
        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
        ... 15 more
Caused by: java.io.IOException: invalid distance too far back
        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
        at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        ... 1 more


After guessing that the zlib error might be hiding the "real" error, I decided to disable compression in Accumulo (compression in MapR is transparent and the results are not affected by whether it is on or off). Normally I'd set table.file.compress.type to none in the accumulo-site.xml file but that doesn't work for the tests as they generate they own site files automatically. I hand edited TestUtil.py to generate a site file with that property set.

When I rerun the test, I get the same output from run.py, but the server error in tserver_xxx.log is very different:

10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga... TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED as alias 8
10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG... TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb... TRUNCATED as alias 22
10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent 2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ... TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR... TRUNCATED as alias 16
10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@... TRUNCATED<
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy0.startScan(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
        ... 15 more
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent !0<;~ as alias 2
10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error processing startScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy0.startScan(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
        at org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
        ... 15 more
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
        at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
        at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
        at org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
        at org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
        at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
        at org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
        at org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        ... 1 more


So the error would seem to be related to reading past the end of the file. What I can't determine is what is the reason. From examining the Accumulo source, it's clear that Accumulo has read the length of a key (I think) and is now trying to read the key value. That second read is what is failing. The question is why? Some ideas: 
1) the file was originally written incorrectly by the writer, 
2) the reader is reading too far

This could be caused by a issue in Accumulo or in MapR. It might be that MapR more strongly enforces end of file reads than stock Hadoop.

If anyone has suggestions on how to look into this further from the Accumulo side, I'd really appreciate it.

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com

Re: Accumulo on MapR Continued

Posted by Eric Newton <er...@gmail.com>.
Thanks Keys,

That was a lot of digging around in the tests!

Tickets made (ACCUMULO-521 - ACCUMULO-527} and fixed in the 1.4 branch and
trunk.  Let me know if you have issues.

-Eric

On Tue, Apr 10, 2012 at 4:24 PM, Eric Newton <er...@gmail.com> wrote:

> This is great!  I'll make tickets.
>
> -Eric
>
>
> On Tue, Apr 10, 2012 at 1:24 PM, Keys Botzum <kb...@maprtech.com> wrote:
>
>> First, I want to thank Billie, Todd, and Eric for their help so far. It
>> is much appreciated.
>>
>> Now, on to the results. As I stated before I'm attempting to run the
>> Accumulo auto tests from test/system/auto. When run standalone, most tests
>> complete successfully on MapR but there have been some issues. That said,
>> I'm really glad that Accumulo has all of these tests. Very useful for
>> verification.
>>
>> First, we found one item that required a change to MapR:
>>
>> simple.bulkSplitOptimization.BulkSplitOptimizationTest failed with a seek
>> error which has now been addressed thanks to Billie's help. This is MapR
>> change 6539 and we'll put that into a release soon.
>>
>> Secondly, the Accumulo tests seem to have various minor glitches. I've
>> documented them here for reference. I hope this is useful to the Accumulo
>> community. I could open JIRA defects if you want but I don't have an
>> account on the tracking system. Just let me know.
>>
>> 1) simple.examples.Examples was failing with a class loader issue. By
>> examining simple/examples.py I was able to determine that the package names
>> were not correct when referring to certain subtests, such as
>> RandomBatchScanner. Specifically it referred to org.apache.accumulo.examples.client.RandomBatchScanner
>> rather than the appropriate org.apache.accumulo.examples.*simple*.client.RandomBatchScanner.
>> The same is true for RandomBatchWriter.
>>
>> 2) simple.mapreduce.MapReduceTest fails because it assumes
>> ZOOKEEPER_HOME is set as an environment variable. Most of the other scripts
>> appear to get this information from TestUtils.py. The workaround is simply
>> to set that variable before using run.py.
>>
>> 3) simple.dynamic.DynamicClassloader fails with class loader issues
>> referencing both Accumulo and Hadoop classes. I examined simple/dynamic.py
>> and determined that the class path references weren't sufficient where
>> javac was invoked. I added the following to the class path command line
>> options for javac and now it works:
>>  /opt/accumulo-1.4.0/lib/*:/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-0.20.2/lib/*
>>
>> Obviously that's a bit of a hack. I'm sure someone more familiar with the
>> code can do this better.
>>
>> 4) stress.weird.LateLastContact fails with a ZOOKEEPER_HOME reference
>> just like MapReduceTest.
>>
>> 5) simple.deleterows.DeleteRowsSplitTest fails with a timeout. The test
>> is coded to wait 120 seconds which wasn't' quite long enough on my system.
>> I changed it to 180 seconds and it finishes cleanly after 138 seconds. Is
>> this typical or expected?
>>
>> 6) simple.zooCacheTest.ZooCacheTest has one minor issue. The issue is
>> that the test times out on my system after 60 seconds. I changed the
>> timeout to 120 seconds and the test completes successfully in 95 seconds.
>> Is this typical or expected?
>>
>> 7) MapR does not normally install a full zookeeper install on each node.
>> Instead we install a zookeeper "client" on each node which is basically the
>> zookeeper JAR file - /opt/mapr/lib/zookeeper-3.3.2.jar. If I set
>> ZOOKEEPER_HOME to /opt/mapr/lib almost everything works fine since Accumulo
>> seems to only need the JAR file at that location. The one exception is
>> the stress.weird.LateLastContact which directly references zkClient.sh (in
>> stress/weird.py) which isn't part of the MapR install. I wanted your
>> opinion on this. Is the test's behavior crucial? Does Accumulo need more of
>> zookeeper than the JAR? I want Accumulo to work,  so if there really is a
>> strong need for more of Zookeeper I'll recommend we make adjustments to the
>> product once we understand the requirement.
>>
>> Third, there are still two tests that are failing when run in standalone
>> mode and I haven't determined the issue: simple.largeRow.LargeRowTest
>> and simple.batchScanSplit.BatchScanSplitTest. I will post details on
>> those in a moment as separate messages as this email is long enough.
>>
>> Thanks again for all of your help so far,
>> Keys
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>>
>>
>>
>

Re: Accumulo on MapR Continued

Posted by Eric Newton <er...@gmail.com>.
This is great!  I'll make tickets.

-Eric

On Tue, Apr 10, 2012 at 1:24 PM, Keys Botzum <kb...@maprtech.com> wrote:

> First, I want to thank Billie, Todd, and Eric for their help so far. It is
> much appreciated.
>
> Now, on to the results. As I stated before I'm attempting to run the
> Accumulo auto tests from test/system/auto. When run standalone, most tests
> complete successfully on MapR but there have been some issues. That said,
> I'm really glad that Accumulo has all of these tests. Very useful for
> verification.
>
> First, we found one item that required a change to MapR:
>
> simple.bulkSplitOptimization.BulkSplitOptimizationTest failed with a seek
> error which has now been addressed thanks to Billie's help. This is MapR
> change 6539 and we'll put that into a release soon.
>
> Secondly, the Accumulo tests seem to have various minor glitches. I've
> documented them here for reference. I hope this is useful to the Accumulo
> community. I could open JIRA defects if you want but I don't have an
> account on the tracking system. Just let me know.
>
> 1) simple.examples.Examples was failing with a class loader issue. By
> examining simple/examples.py I was able to determine that the package names
> were not correct when referring to certain subtests, such as
> RandomBatchScanner. Specifically it referred to org.apache.accumulo.examples.client.RandomBatchScanner
> rather than the appropriate org.apache.accumulo.examples.*simple*.client.RandomBatchScanner.
> The same is true for RandomBatchWriter.
>
> 2) simple.mapreduce.MapReduceTest fails because it assumes ZOOKEEPER_HOME
> is set as an environment variable. Most of the other scripts appear to get
> this information from TestUtils.py. The workaround is simply to set that
> variable before using run.py.
>
> 3) simple.dynamic.DynamicClassloader fails with class loader issues
> referencing both Accumulo and Hadoop classes. I examined simple/dynamic.py
> and determined that the class path references weren't sufficient where
> javac was invoked. I added the following to the class path command line
> options for javac and now it works:
>  /opt/accumulo-1.4.0/lib/*:/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-0.20.2/lib/*
>
> Obviously that's a bit of a hack. I'm sure someone more familiar with the
> code can do this better.
>
> 4) stress.weird.LateLastContact fails with a ZOOKEEPER_HOME reference
> just like MapReduceTest.
>
> 5) simple.deleterows.DeleteRowsSplitTest fails with a timeout. The test
> is coded to wait 120 seconds which wasn't' quite long enough on my system.
> I changed it to 180 seconds and it finishes cleanly after 138 seconds. Is
> this typical or expected?
>
> 6) simple.zooCacheTest.ZooCacheTest has one minor issue. The issue is
> that the test times out on my system after 60 seconds. I changed the
> timeout to 120 seconds and the test completes successfully in 95 seconds.
> Is this typical or expected?
>
> 7) MapR does not normally install a full zookeeper install on each node.
> Instead we install a zookeeper "client" on each node which is basically the
> zookeeper JAR file - /opt/mapr/lib/zookeeper-3.3.2.jar. If I set
> ZOOKEEPER_HOME to /opt/mapr/lib almost everything works fine since Accumulo
> seems to only need the JAR file at that location. The one exception is
> the stress.weird.LateLastContact which directly references zkClient.sh (in
> stress/weird.py) which isn't part of the MapR install. I wanted your
> opinion on this. Is the test's behavior crucial? Does Accumulo need more of
> zookeeper than the JAR? I want Accumulo to work,  so if there really is a
> strong need for more of Zookeeper I'll recommend we make adjustments to the
> product once we understand the requirement.
>
> Third, there are still two tests that are failing when run in standalone
> mode and I haven't determined the issue: simple.largeRow.LargeRowTest and simple.batchScanSplit.BatchScanSplitTest.
> I will post details on those in a moment as separate messages as this email
> is long enough.
>
> Thanks again for all of your help so far,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
A minor correction on #3 below.  I was talking with a colleague and he pointed out that /opt/mapr/lib does not need to be on the class path as Hadoop programs only need Hadoop classes on their paths. So #3 below becomes:


> 3) simple.dynamic.DynamicClassloader fails with class loader issues referencing both Accumulo and Hadoop classes. I examined simple/dynamic.py and determined that the class path references weren't sufficient where javac was invoked. I added the following to the class path command line options for javac and now it works:  /opt/accumulo-1.4.0/lib/*:/opt/mapr/hadoop/hadoop-0.20.2/lib/*


Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 10, 2012, at 1:24 PM, Keys Botzum wrote:

> First, I want to thank Billie, Todd, and Eric for their help so far. It is much appreciated.
> 
> Now, on to the results. As I stated before I'm attempting to run the Accumulo auto tests from test/system/auto. When run standalone, most tests complete successfully on MapR but there have been some issues. That said, I'm really glad that Accumulo has all of these tests. Very useful for verification.
> 
> First, we found one item that required a change to MapR:
> 
> simple.bulkSplitOptimization.BulkSplitOptimizationTest failed with a seek error which has now been addressed thanks to Billie's help. This is MapR change 6539 and we'll put that into a release soon.
> 
> Secondly, the Accumulo tests seem to have various minor glitches. I've documented them here for reference. I hope this is useful to the Accumulo community. I could open JIRA defects if you want but I don't have an account on the tracking system. Just let me know.
> 
> 1) simple.examples.Examples was failing with a class loader issue. By examining simple/examples.py I was able to determine that the package names were not correct when referring to certain subtests, such as RandomBatchScanner. Specifically it referred to org.apache.accumulo.examples.client.RandomBatchScanner rather than the appropriate org.apache.accumulo.examples.simple.client.RandomBatchScanner. The same is true for RandomBatchWriter.
> 
> 2) simple.mapreduce.MapReduceTest fails because it assumes ZOOKEEPER_HOME is set as an environment variable. Most of the other scripts appear to get this information from TestUtils.py. The workaround is simply to set that variable before using run.py.
> 
> 3) simple.dynamic.DynamicClassloader fails with class loader issues referencing both Accumulo and Hadoop classes. I examined simple/dynamic.py and determined that the class path references weren't sufficient where javac was invoked. I added the following to the class path command line options for javac and now it works:  /opt/accumulo-1.4.0/lib/*:/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-0.20.2/lib/*
> 
> Obviously that's a bit of a hack. I'm sure someone more familiar with the code can do this better.
> 
> 4) stress.weird.LateLastContact fails with a ZOOKEEPER_HOME reference just like MapReduceTest.
> 
> 5) simple.deleterows.DeleteRowsSplitTest fails with a timeout. The test is coded to wait 120 seconds which wasn't' quite long enough on my system. I changed it to 180 seconds and it finishes cleanly after 138 seconds. Is this typical or expected? 
> 
> 6) simple.zooCacheTest.ZooCacheTest has one minor issue. The issue is that the test times out on my system after 60 seconds. I changed the timeout to 120 seconds and the test completes successfully in 95 seconds. Is this typical or expected?
> 
> 7) MapR does not normally install a full zookeeper install on each node. Instead we install a zookeeper "client" on each node which is basically the zookeeper JAR file - /opt/mapr/lib/zookeeper-3.3.2.jar. If I set ZOOKEEPER_HOME to /opt/mapr/lib almost everything works fine since Accumulo seems to only need the JAR file at that location. The one exception is the stress.weird.LateLastContact which directly references zkClient.sh (in stress/weird.py) which isn't part of the MapR install. I wanted your opinion on this. Is the test's behavior crucial? Does Accumulo need more of zookeeper than the JAR? I want Accumulo to work,  so if there really is a strong need for more of Zookeeper I'll recommend we make adjustments to the product once we understand the requirement.
> 
> Third, there are still two tests that are failing when run in standalone mode and I haven't determined the issue: simple.largeRow.LargeRowTest and simple.batchScanSplit.BatchScanSplitTest. I will post details on those in a moment as separate messages as this email is long enough.
> 
> Thanks again for all of your help so far,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 


Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
First, I want to thank Billie, Todd, and Eric for their help so far. It is much appreciated.

Now, on to the results. As I stated before I'm attempting to run the Accumulo auto tests from test/system/auto. When run standalone, most tests complete successfully on MapR but there have been some issues. That said, I'm really glad that Accumulo has all of these tests. Very useful for verification.

First, we found one item that required a change to MapR:

simple.bulkSplitOptimization.BulkSplitOptimizationTest failed with a seek error which has now been addressed thanks to Billie's help. This is MapR change 6539 and we'll put that into a release soon.

Secondly, the Accumulo tests seem to have various minor glitches. I've documented them here for reference. I hope this is useful to the Accumulo community. I could open JIRA defects if you want but I don't have an account on the tracking system. Just let me know.

1) simple.examples.Examples was failing with a class loader issue. By examining simple/examples.py I was able to determine that the package names were not correct when referring to certain subtests, such as RandomBatchScanner. Specifically it referred to org.apache.accumulo.examples.client.RandomBatchScanner rather than the appropriate org.apache.accumulo.examples.simple.client.RandomBatchScanner. The same is true for RandomBatchWriter.

2) simple.mapreduce.MapReduceTest fails because it assumes ZOOKEEPER_HOME is set as an environment variable. Most of the other scripts appear to get this information from TestUtils.py. The workaround is simply to set that variable before using run.py.

3) simple.dynamic.DynamicClassloader fails with class loader issues referencing both Accumulo and Hadoop classes. I examined simple/dynamic.py and determined that the class path references weren't sufficient where javac was invoked. I added the following to the class path command line options for javac and now it works:  /opt/accumulo-1.4.0/lib/*:/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-0.20.2/lib/*

Obviously that's a bit of a hack. I'm sure someone more familiar with the code can do this better.

4) stress.weird.LateLastContact fails with a ZOOKEEPER_HOME reference just like MapReduceTest.

5) simple.deleterows.DeleteRowsSplitTest fails with a timeout. The test is coded to wait 120 seconds which wasn't' quite long enough on my system. I changed it to 180 seconds and it finishes cleanly after 138 seconds. Is this typical or expected? 

6) simple.zooCacheTest.ZooCacheTest has one minor issue. The issue is that the test times out on my system after 60 seconds. I changed the timeout to 120 seconds and the test completes successfully in 95 seconds. Is this typical or expected?

7) MapR does not normally install a full zookeeper install on each node. Instead we install a zookeeper "client" on each node which is basically the zookeeper JAR file - /opt/mapr/lib/zookeeper-3.3.2.jar. If I set ZOOKEEPER_HOME to /opt/mapr/lib almost everything works fine since Accumulo seems to only need the JAR file at that location. The one exception is the stress.weird.LateLastContact which directly references zkClient.sh (in stress/weird.py) which isn't part of the MapR install. I wanted your opinion on this. Is the test's behavior crucial? Does Accumulo need more of zookeeper than the JAR? I want Accumulo to work,  so if there really is a strong need for more of Zookeeper I'll recommend we make adjustments to the product once we understand the requirement.

Third, there are still two tests that are failing when run in standalone mode and I haven't determined the issue: simple.largeRow.LargeRowTest and simple.batchScanSplit.BatchScanSplitTest. I will post details on those in a moment as separate messages as this email is long enough.

Thanks again for all of your help so far,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



Re: Accumulo on MapR Continued

Posted by Eric Newton <er...@gmail.com>.
Things definitely get sketchy towards the end of the run.  The tests are
generally ordered from "if this doesn't work, don't bother checking in the
code."  Towards the end we get to "try fake i/o freeze-up using a custom
LD_PRELOAD hack" which is platform and timing sensitive.  If you get
through the first half of the tests, it's a very good sign.  If you re-run
a test and it magically works, that's a sign that its a test problem.

Here are some helpful ways of running the test suite that you may have
discovered already.

To start the tests at some point:

./run.py -s someTest


To run a particular test:

./run.py -t pattern


To see the test run with lots of details:

./run.py -t SomeTest -v 10


There is also a mechanism to run the tests under map/reduce, so you can
invoke them concurrently.  It is not very stable.

-Eric

On Sun, Apr 8, 2012 at 8:43 PM, Keys Botzum <kb...@maprtech.com> wrote:

> Eric,
>
> Thanks for that hint. Do you have a sense (or is this written somewhere)
> as to what tests may not work? I am running the tests as a user with very
> high permissions so I think they can write anywhere in HDFS, but I can
> double check that easily enough.
>
> If there are tests that are known not to work, I can avoid those tests or
> at least not post so much to the mailing list about such tests.
>
> If you don't know, I'll just carry on and post what I learn. Obvious
> script bugs I can easily fix. Other more subtle issues require more
> investigation.
>
>
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Apr 8, 2012, at 10:23 AM, Eric Newton wrote:
>
> The auto tests are a bit flakey, especially when they rely on local
> conditions, like the speed at which my workstation will run something.  If
> you have found a bug in one of the scripts, please make a Jira ticket.
>
> Right now several of the tests assume they have superuser rights to
> scribble all over the distributed file system, see ACCUMULO-355.
>
> -Eric
>
> On Sun, Apr 8, 2012 at 10:01 AM, Keys Botzum <kb...@maprtech.com> wrote:
>
>> Billie,
>>
>> I apologize for taking so long to reply to your message. As soon as I
>> read it I thought to myself "this is a very good argument." I did some more
>> research myself and talked with folks at MapR and we agreed that the right
>> thing to do here is to change MapR to be tolerant of a seek to position
>> just after the last byte.
>>
>> I've delayed posting this because I wanted to report the results of the
>> change and retest. This was delayed for reasons that have nothing to do
>> with Accumulo or really MapR, I just was too busy.
>>
>> In any case, we've tried the change in MapR (being tracked as
>> change 6539) and this does indeed address the issue below. The test case in
>> question now passes. Thank you VERY much for your help. It is much
>> appreciated.
>>
>> I am continuing testing and have encountered additional issues but I
>> haven't yet had time to analyze them. I will post something when I have a
>> more coherent question. I would just like to confirm that all of the auto
>> tests do run successfully on stock Apache Hadoop? I ask because at least
>> one of the tests appears to have a bug in the script that runs it. I just
>> want to make sure I'm testing the right stuff.
>>
>> Incidentally, I have upgrade to the official Accumulo 1.4.0 release.
>>
>>
>> Keys
>>
>>
>> On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <
>> billie.j.rinaldi@ugov.gov> wrote:
>>
>>> It is my understanding that the file length is a valid file position.
>>>  If you are at position 0 and read one byte, you get the first byte of the
>>> file.  If you are at position length-1 and read one byte, you get the last
>>> byte of the file.  At that point you are at position equal to the file
>>> length, and attempting to read an additional byte will give an EOF
>>> indicator (for an InputStream read(), this would mean returning -1).
>>>  Seeking to the file length should just cause a read to give an EOF.
>>>
>>> Billie
>>>
>>>
>>> ----- Original Message -----
>>> > From: "Keys Botzum" <kb...@maprtech.com>
>>> > To: user@accumulo.apache.org
>>> > Sent: Friday, March 30, 2012 10:12:00 PM
>>> > Subject: Re: Accumulo on MapR Continued
>>> > Billie,
>>> >
>>> >
>>> > Thanks for the quick reply. I did some checking and the Hadoop docs
>>> > indicate that seeking past the end of the file is illegal.
>>> >
>>> >
>>> >
>>> >
>>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
>>> > seek
>>> > public void seek (long desired)
>>> > throws IOException Description copied from interface: Seekable Seek to
>>> > the given offset from the start of the file. The next read() will be
>>> > from that location. Can't seek past the end of the file.
>>> >
>>> > Specified by: seek in interface Seekable Throws: IOException
>>> >
>>> >
>>> > The code below that is seeking to the file length is actually seeking
>>> > one byte past the end of the file since the first byte in a file is
>>> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
>>> >
>>> >
>>> > Thanks,
>>> > Keys
>>> > ________________________________
>>> > Keys Botzum
>>> > Senior Principal Technologist
>>> > WW Systems Engineering
>>> > kbotzum@maprtech.com
>>> > 443-718-0098
>>> > MapR Technologies
>>> > http://www.mapr.com
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
>>> >
>>> >
>>> >
>>> > I ran the following JUnit test and verified that the value printed out
>>> > was the file length.
>>> >
>>> > @Test
>>> > public void testSeekToEOF() throws IOException {
>>> > FileSystem fs = FileSystem.get(new Configuration());
>>> > FSDataInputStream fdi = fs.open(new Path("README"));
>>> > System.out.println("seeking to end: " + fdi.available());
>>> > fdi.seek(fdi.available());
>>> > fdi.close();
>>> > }
>>> >
>>> >
>>> > ----- Original Message -----
>>> >
>>> >
>>> > From: "Keys Botzum" < kbotzum@maprtech.com >
>>> >
>>> >
>>> > To: user@accumulo.apache.org
>>> >
>>> >
>>> > Sent: Friday, March 30, 2012 4:56:09 PM
>>> >
>>> >
>>> > Subject: Re: Accumulo on MapR Continued
>>> >
>>> >
>>> > Can you share the program? I can try running it and answer your
>>> >
>>> >
>>> > question. I can also of course check with MapR engineering to see if
>>> >
>>> >
>>> > that's the issue.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Much appreciated,
>>> >
>>> >
>>> > Keys
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > ________________________________
>>> >
>>> >
>>> > Keys Botzum
>>> >
>>> >
>>> > Senior Principal Technologist
>>> >
>>> >
>>> > WW Systems Engineering
>>> >
>>> >
>>> > kbotzum@maprtech.com
>>> >
>>> >
>>> > 443-718-0098
>>> >
>>> >
>>> > MapR Technologies
>>> >
>>> >
>>> > http://www.mapr.com
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
>>> >
>>> >
>>> > kbotzum@maprtech.com > wrote:
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Seeking beyond EOF, filelen 733, wantpos 733
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > It looks like Accumulo is trying to seek to the last position in a
>>> >
>>> >
>>> > file. I just wrote a little program that opens a file in HDFS and
>>> >
>>> >
>>> > seeks to its file length, and it didn't throw an error. Does MapR not
>>> >
>>> >
>>> > allow this? If that is the desired MapR behavior, we could try to
>>> >
>>> >
>>> > determine whether Accumulo could avoid performing that seek.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Billie
>>>
>>
>>
>>
>> --
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com <http://mapr.com/>
>>
>>
>
>

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
Eric,

Thanks for that hint. Do you have a sense (or is this written somewhere) as to what tests may not work? I am running the tests as a user with very high permissions so I think they can write anywhere in HDFS, but I can double check that easily enough.

If there are tests that are known not to work, I can avoid those tests or at least not post so much to the mailing list about such tests.

If you don't know, I'll just carry on and post what I learn. Obvious script bugs I can easily fix. Other more subtle issues require more investigation.


Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Apr 8, 2012, at 10:23 AM, Eric Newton wrote:

> The auto tests are a bit flakey, especially when they rely on local conditions, like the speed at which my workstation will run something.  If you have found a bug in one of the scripts, please make a Jira ticket.
> 
> Right now several of the tests assume they have superuser rights to scribble all over the distributed file system, see ACCUMULO-355.
> 
> -Eric
> 
> On Sun, Apr 8, 2012 at 10:01 AM, Keys Botzum <kb...@maprtech.com> wrote:
> Billie,
> 
> I apologize for taking so long to reply to your message. As soon as I read it I thought to myself "this is a very good argument." I did some more research myself and talked with folks at MapR and we agreed that the right thing to do here is to change MapR to be tolerant of a seek to position just after the last byte. 
> 
> I've delayed posting this because I wanted to report the results of the change and retest. This was delayed for reasons that have nothing to do with Accumulo or really MapR, I just was too busy.
> 
> In any case, we've tried the change in MapR (being tracked as change 6539) and this does indeed address the issue below. The test case in question now passes. Thank you VERY much for your help. It is much appreciated.
> 
> I am continuing testing and have encountered additional issues but I haven't yet had time to analyze them. I will post something when I have a more coherent question. I would just like to confirm that all of the auto tests do run successfully on stock Apache Hadoop? I ask because at least one of the tests appears to have a bug in the script that runs it. I just want to make sure I'm testing the right stuff.
> 
> Incidentally, I have upgrade to the official Accumulo 1.4.0 release.
> 
> 
> Keys
> 
> 
> On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <bi...@ugov.gov> wrote:
> It is my understanding that the file length is a valid file position.  If you are at position 0 and read one byte, you get the first byte of the file.  If you are at position length-1 and read one byte, you get the last byte of the file.  At that point you are at position equal to the file length, and attempting to read an additional byte will give an EOF indicator (for an InputStream read(), this would mean returning -1).  Seeking to the file length should just cause a read to give an EOF.
> 
> Billie
> 
> 
> ----- Original Message -----
> > From: "Keys Botzum" <kb...@maprtech.com>
> > To: user@accumulo.apache.org
> > Sent: Friday, March 30, 2012 10:12:00 PM
> > Subject: Re: Accumulo on MapR Continued
> > Billie,
> >
> >
> > Thanks for the quick reply. I did some checking and the Hadoop docs
> > indicate that seeking past the end of the file is illegal.
> >
> >
> >
> > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
> > seek
> > public void seek (long desired)
> > throws IOException Description copied from interface: Seekable Seek to
> > the given offset from the start of the file. The next read() will be
> > from that location. Can't seek past the end of the file.
> >
> > Specified by: seek in interface Seekable Throws: IOException
> >
> >
> > The code below that is seeking to the file length is actually seeking
> > one byte past the end of the file since the first byte in a file is
> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
> >
> >
> > Thanks,
> > Keys
> > ________________________________
> > Keys Botzum
> > Senior Principal Technologist
> > WW Systems Engineering
> > kbotzum@maprtech.com
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
> >
> >
> >
> > I ran the following JUnit test and verified that the value printed out
> > was the file length.
> >
> > @Test
> > public void testSeekToEOF() throws IOException {
> > FileSystem fs = FileSystem.get(new Configuration());
> > FSDataInputStream fdi = fs.open(new Path("README"));
> > System.out.println("seeking to end: " + fdi.available());
> > fdi.seek(fdi.available());
> > fdi.close();
> > }
> >
> >
> > ----- Original Message -----
> >
> >
> > From: "Keys Botzum" < kbotzum@maprtech.com >
> >
> >
> > To: user@accumulo.apache.org
> >
> >
> > Sent: Friday, March 30, 2012 4:56:09 PM
> >
> >
> > Subject: Re: Accumulo on MapR Continued
> >
> >
> > Can you share the program? I can try running it and answer your
> >
> >
> > question. I can also of course check with MapR engineering to see if
> >
> >
> > that's the issue.
> >
> >
> >
> >
> >
> >
> >
> >
> > Much appreciated,
> >
> >
> > Keys
> >
> >
> >
> >
> >
> > ________________________________
> >
> >
> > Keys Botzum
> >
> >
> > Senior Principal Technologist
> >
> >
> > WW Systems Engineering
> >
> >
> > kbotzum@maprtech.com
> >
> >
> > 443-718-0098
> >
> >
> > MapR Technologies
> >
> >
> > http://www.mapr.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
> >
> >
> > kbotzum@maprtech.com > wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> >
> >
> >
> >
> >
> >
> >
> >
> > Seeking beyond EOF, filelen 733, wantpos 733
> >
> >
> >
> >
> >
> > It looks like Accumulo is trying to seek to the last position in a
> >
> >
> > file. I just wrote a little program that opens a file in HDFS and
> >
> >
> > seeks to its file length, and it didn't throw an error. Does MapR not
> >
> >
> > allow this? If that is the desired MapR behavior, we could try to
> >
> >
> > determine whether Accumulo could avoid performing that seek.
> >
> >
> >
> >
> >
> > Billie
> 
> 
> 
> -- 
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 


Re: Accumulo on MapR Continued

Posted by Eric Newton <er...@gmail.com>.
The auto tests are a bit flakey, especially when they rely on local
conditions, like the speed at which my workstation will run something.  If
you have found a bug in one of the scripts, please make a Jira ticket.

Right now several of the tests assume they have superuser rights to
scribble all over the distributed file system, see ACCUMULO-355.

-Eric

On Sun, Apr 8, 2012 at 10:01 AM, Keys Botzum <kb...@maprtech.com> wrote:

> Billie,
>
> I apologize for taking so long to reply to your message. As soon as I read
> it I thought to myself "this is a very good argument." I did some more
> research myself and talked with folks at MapR and we agreed that the right
> thing to do here is to change MapR to be tolerant of a seek to position
> just after the last byte.
>
> I've delayed posting this because I wanted to report the results of the
> change and retest. This was delayed for reasons that have nothing to do
> with Accumulo or really MapR, I just was too busy.
>
> In any case, we've tried the change in MapR (being tracked as change 6539)
> and this does indeed address the issue below. The test case in question now
> passes. Thank you VERY much for your help. It is much appreciated.
>
> I am continuing testing and have encountered additional issues but I
> haven't yet had time to analyze them. I will post something when I have a
> more coherent question. I would just like to confirm that all of the auto
> tests do run successfully on stock Apache Hadoop? I ask because at least
> one of the tests appears to have a bug in the script that runs it. I just
> want to make sure I'm testing the right stuff.
>
> Incidentally, I have upgrade to the official Accumulo 1.4.0 release.
>
>
> Keys
>
>
> On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <
> billie.j.rinaldi@ugov.gov> wrote:
>
>> It is my understanding that the file length is a valid file position.  If
>> you are at position 0 and read one byte, you get the first byte of the
>> file.  If you are at position length-1 and read one byte, you get the last
>> byte of the file.  At that point you are at position equal to the file
>> length, and attempting to read an additional byte will give an EOF
>> indicator (for an InputStream read(), this would mean returning -1).
>>  Seeking to the file length should just cause a read to give an EOF.
>>
>> Billie
>>
>>
>> ----- Original Message -----
>> > From: "Keys Botzum" <kb...@maprtech.com>
>> > To: user@accumulo.apache.org
>> > Sent: Friday, March 30, 2012 10:12:00 PM
>> > Subject: Re: Accumulo on MapR Continued
>> > Billie,
>> >
>> >
>> > Thanks for the quick reply. I did some checking and the Hadoop docs
>> > indicate that seeking past the end of the file is illegal.
>> >
>> >
>> >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
>> > seek
>> > public void seek (long desired)
>> > throws IOException Description copied from interface: Seekable Seek to
>> > the given offset from the start of the file. The next read() will be
>> > from that location. Can't seek past the end of the file.
>> >
>> > Specified by: seek in interface Seekable Throws: IOException
>> >
>> >
>> > The code below that is seeking to the file length is actually seeking
>> > one byte past the end of the file since the first byte in a file is
>> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
>> >
>> >
>> > Thanks,
>> > Keys
>> > ________________________________
>> > Keys Botzum
>> > Senior Principal Technologist
>> > WW Systems Engineering
>> > kbotzum@maprtech.com
>> > 443-718-0098
>> > MapR Technologies
>> > http://www.mapr.com
>> >
>> >
>> >
>> >
>> >
>> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
>> >
>> >
>> >
>> > I ran the following JUnit test and verified that the value printed out
>> > was the file length.
>> >
>> > @Test
>> > public void testSeekToEOF() throws IOException {
>> > FileSystem fs = FileSystem.get(new Configuration());
>> > FSDataInputStream fdi = fs.open(new Path("README"));
>> > System.out.println("seeking to end: " + fdi.available());
>> > fdi.seek(fdi.available());
>> > fdi.close();
>> > }
>> >
>> >
>> > ----- Original Message -----
>> >
>> >
>> > From: "Keys Botzum" < kbotzum@maprtech.com >
>> >
>> >
>> > To: user@accumulo.apache.org
>> >
>> >
>> > Sent: Friday, March 30, 2012 4:56:09 PM
>> >
>> >
>> > Subject: Re: Accumulo on MapR Continued
>> >
>> >
>> > Can you share the program? I can try running it and answer your
>> >
>> >
>> > question. I can also of course check with MapR engineering to see if
>> >
>> >
>> > that's the issue.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Much appreciated,
>> >
>> >
>> > Keys
>> >
>> >
>> >
>> >
>> >
>> > ________________________________
>> >
>> >
>> > Keys Botzum
>> >
>> >
>> > Senior Principal Technologist
>> >
>> >
>> > WW Systems Engineering
>> >
>> >
>> > kbotzum@maprtech.com
>> >
>> >
>> > 443-718-0098
>> >
>> >
>> > MapR Technologies
>> >
>> >
>> > http://www.mapr.com
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
>> >
>> >
>> > kbotzum@maprtech.com > wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Seeking beyond EOF, filelen 733, wantpos 733
>> >
>> >
>> >
>> >
>> >
>> > It looks like Accumulo is trying to seek to the last position in a
>> >
>> >
>> > file. I just wrote a little program that opens a file in HDFS and
>> >
>> >
>> > seeks to its file length, and it didn't throw an error. Does MapR not
>> >
>> >
>> > allow this? If that is the desired MapR behavior, we could try to
>> >
>> >
>> > determine whether Accumulo could avoid performing that seek.
>> >
>> >
>> >
>> >
>> >
>> > Billie
>>
>
>
>
> --
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com <http://mapr.com>
>
>

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
Billie,

I apologize for taking so long to reply to your message. As soon as I read
it I thought to myself "this is a very good argument." I did some more
research myself and talked with folks at MapR and we agreed that the right
thing to do here is to change MapR to be tolerant of a seek to position
just after the last byte.

I've delayed posting this because I wanted to report the results of the
change and retest. This was delayed for reasons that have nothing to do
with Accumulo or really MapR, I just was too busy.

In any case, we've tried the change in MapR (being tracked as change 6539)
and this does indeed address the issue below. The test case in question now
passes. Thank you VERY much for your help. It is much appreciated.

I am continuing testing and have encountered additional issues but I
haven't yet had time to analyze them. I will post something when I have a
more coherent question. I would just like to confirm that all of the auto
tests do run successfully on stock Apache Hadoop? I ask because at least
one of the tests appears to have a bug in the script that runs it. I just
want to make sure I'm testing the right stuff.

Incidentally, I have upgrade to the official Accumulo 1.4.0 release.


Keys


On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi <
billie.j.rinaldi@ugov.gov> wrote:

> It is my understanding that the file length is a valid file position.  If
> you are at position 0 and read one byte, you get the first byte of the
> file.  If you are at position length-1 and read one byte, you get the last
> byte of the file.  At that point you are at position equal to the file
> length, and attempting to read an additional byte will give an EOF
> indicator (for an InputStream read(), this would mean returning -1).
>  Seeking to the file length should just cause a read to give an EOF.
>
> Billie
>
>
> ----- Original Message -----
> > From: "Keys Botzum" <kb...@maprtech.com>
> > To: user@accumulo.apache.org
> > Sent: Friday, March 30, 2012 10:12:00 PM
> > Subject: Re: Accumulo on MapR Continued
> > Billie,
> >
> >
> > Thanks for the quick reply. I did some checking and the Hadoop docs
> > indicate that seeking past the end of the file is illegal.
> >
> >
> >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
> > seek
> > public void seek (long desired)
> > throws IOException Description copied from interface: Seekable Seek to
> > the given offset from the start of the file. The next read() will be
> > from that location. Can't seek past the end of the file.
> >
> > Specified by: seek in interface Seekable Throws: IOException
> >
> >
> > The code below that is seeking to the file length is actually seeking
> > one byte past the end of the file since the first byte in a file is
> > byte zero. To me this seems like an Accumulo bug. Can this be fixed?
> >
> >
> > Thanks,
> > Keys
> > ________________________________
> > Keys Botzum
> > Senior Principal Technologist
> > WW Systems Engineering
> > kbotzum@maprtech.com
> > 443-718-0098
> > MapR Technologies
> > http://www.mapr.com
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
> >
> >
> >
> > I ran the following JUnit test and verified that the value printed out
> > was the file length.
> >
> > @Test
> > public void testSeekToEOF() throws IOException {
> > FileSystem fs = FileSystem.get(new Configuration());
> > FSDataInputStream fdi = fs.open(new Path("README"));
> > System.out.println("seeking to end: " + fdi.available());
> > fdi.seek(fdi.available());
> > fdi.close();
> > }
> >
> >
> > ----- Original Message -----
> >
> >
> > From: "Keys Botzum" < kbotzum@maprtech.com >
> >
> >
> > To: user@accumulo.apache.org
> >
> >
> > Sent: Friday, March 30, 2012 4:56:09 PM
> >
> >
> > Subject: Re: Accumulo on MapR Continued
> >
> >
> > Can you share the program? I can try running it and answer your
> >
> >
> > question. I can also of course check with MapR engineering to see if
> >
> >
> > that's the issue.
> >
> >
> >
> >
> >
> >
> >
> >
> > Much appreciated,
> >
> >
> > Keys
> >
> >
> >
> >
> >
> > ________________________________
> >
> >
> > Keys Botzum
> >
> >
> > Senior Principal Technologist
> >
> >
> > WW Systems Engineering
> >
> >
> > kbotzum@maprtech.com
> >
> >
> > 443-718-0098
> >
> >
> > MapR Technologies
> >
> >
> > http://www.mapr.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
> >
> >
> > kbotzum@maprtech.com > wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> >
> >
> >
> >
> >
> >
> >
> >
> > Seeking beyond EOF, filelen 733, wantpos 733
> >
> >
> >
> >
> >
> > It looks like Accumulo is trying to seek to the last position in a
> >
> >
> > file. I just wrote a little program that opens a file in HDFS and
> >
> >
> > seeks to its file length, and it didn't throw an error. Does MapR not
> >
> >
> > allow this? If that is the desired MapR behavior, we could try to
> >
> >
> > determine whether Accumulo could avoid performing that seek.
> >
> >
> >
> >
> >
> > Billie
>



-- 
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com <http://mapr.com>

Re: Accumulo on MapR Continued

Posted by Billie J Rinaldi <bi...@ugov.gov>.
It is my understanding that the file length is a valid file position.  If you are at position 0 and read one byte, you get the first byte of the file.  If you are at position length-1 and read one byte, you get the last byte of the file.  At that point you are at position equal to the file length, and attempting to read an additional byte will give an EOF indicator (for an InputStream read(), this would mean returning -1).  Seeking to the file length should just cause a read to give an EOF.

Billie


----- Original Message -----
> From: "Keys Botzum" <kb...@maprtech.com>
> To: user@accumulo.apache.org
> Sent: Friday, March 30, 2012 10:12:00 PM
> Subject: Re: Accumulo on MapR Continued
> Billie,
> 
> 
> Thanks for the quick reply. I did some checking and the Hadoop docs
> indicate that seeking past the end of the file is illegal.
> 
> 
> 
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
> seek
> public void seek (long desired)
> throws IOException Description copied from interface: Seekable Seek to
> the given offset from the start of the file. The next read() will be
> from that location. Can't seek past the end of the file.
> 
> Specified by: seek in interface Seekable Throws: IOException
> 
> 
> The code below that is seeking to the file length is actually seeking
> one byte past the end of the file since the first byte in a file is
> byte zero. To me this seems like an Accumulo bug. Can this be fixed?
> 
> 
> Thanks,
> Keys
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 
> 
> 
> 
> On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:
> 
> 
> 
> I ran the following JUnit test and verified that the value printed out
> was the file length.
> 
> @Test
> public void testSeekToEOF() throws IOException {
> FileSystem fs = FileSystem.get(new Configuration());
> FSDataInputStream fdi = fs.open(new Path("README"));
> System.out.println("seeking to end: " + fdi.available());
> fdi.seek(fdi.available());
> fdi.close();
> }
> 
> 
> ----- Original Message -----
> 
> 
> From: "Keys Botzum" < kbotzum@maprtech.com >
> 
> 
> To: user@accumulo.apache.org
> 
> 
> Sent: Friday, March 30, 2012 4:56:09 PM
> 
> 
> Subject: Re: Accumulo on MapR Continued
> 
> 
> Can you share the program? I can try running it and answer your
> 
> 
> question. I can also of course check with MapR engineering to see if
> 
> 
> that's the issue.
> 
> 
> 
> 
> 
> 
> 
> 
> Much appreciated,
> 
> 
> Keys
> 
> 
> 
> 
> 
> ________________________________
> 
> 
> Keys Botzum
> 
> 
> Senior Principal Technologist
> 
> 
> WW Systems Engineering
> 
> 
> kbotzum@maprtech.com
> 
> 
> 443-718-0098
> 
> 
> MapR Technologies
> 
> 
> http://www.mapr.com
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
> 
> 
> kbotzum@maprtech.com > wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> 
> 
> 
> 
> 
> 
> 
> 
> Seeking beyond EOF, filelen 733, wantpos 733
> 
> 
> 
> 
> 
> It looks like Accumulo is trying to seek to the last position in a
> 
> 
> file. I just wrote a little program that opens a file in HDFS and
> 
> 
> seeks to its file length, and it didn't throw an error. Does MapR not
> 
> 
> allow this? If that is the desired MapR behavior, we could try to
> 
> 
> determine whether Accumulo could avoid performing that seek.
> 
> 
> 
> 
> 
> Billie

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
Billie,

Thanks for the quick reply.  I did some checking and the Hadoop docs indicate that seeking past the end of the file is illegal. 

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29
 seek

public void seek(long desired)
          throws IOException
Description copied from interface: Seekable
Seek to the given offset from the start of the file. The next read() will be from that location. Can't seek past the end of the file.

Specified by:
seek in interface Seekable
Throws:
IOException

The code below that is seeking to the file length is actually seeking one byte past the end of the file since the first byte in a file is byte zero. To me this seems like an Accumulo bug. Can this be fixed?

Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote:

> I ran the following JUnit test and verified that the value printed out was the file length.
> 
>  @Test
>  public void testSeekToEOF() throws IOException {
>    FileSystem fs = FileSystem.get(new Configuration());
>    FSDataInputStream fdi = fs.open(new Path("README"));
>    System.out.println("seeking to end: " + fdi.available());
>    fdi.seek(fdi.available());
>    fdi.close();
>  }
> 
> 
> ----- Original Message -----
>> From: "Keys Botzum" <kb...@maprtech.com>
>> To: user@accumulo.apache.org
>> Sent: Friday, March 30, 2012 4:56:09 PM
>> Subject: Re: Accumulo on MapR Continued
>> Can you share the program? I can try running it and answer your
>> question. I can also of course check with MapR engineering to see if
>> that's the issue.
>> 
>> 
>> Much appreciated,
>> Keys
>> 
>> ________________________________
>> Keys Botzum
>> Senior Principal Technologist
>> WW Systems Engineering
>> kbotzum@maprtech.com
>> 443-718-0098
>> MapR Technologies
>> http://www.mapr.com
>> 
>> 
>> 
>> 
>> 
>> On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
>> 
>> 
>> 
>> On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
>> kbotzum@maprtech.com > wrote:
>> 
>> 
>> 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
>> 
>> 
>> Seeking beyond EOF, filelen 733, wantpos 733
>> 
>> It looks like Accumulo is trying to seek to the last position in a
>> file. I just wrote a little program that opens a file in HDFS and
>> seeks to its file length, and it didn't throw an error. Does MapR not
>> allow this? If that is the desired MapR behavior, we could try to
>> determine whether Accumulo could avoid performing that seek.
>> 
>> Billie


Re: Accumulo on MapR Continued

Posted by Billie J Rinaldi <bi...@ugov.gov>.
I ran the following JUnit test and verified that the value printed out was the file length.

  @Test
  public void testSeekToEOF() throws IOException {
    FileSystem fs = FileSystem.get(new Configuration());
    FSDataInputStream fdi = fs.open(new Path("README"));
    System.out.println("seeking to end: " + fdi.available());
    fdi.seek(fdi.available());
    fdi.close();
  }


----- Original Message -----
> From: "Keys Botzum" <kb...@maprtech.com>
> To: user@accumulo.apache.org
> Sent: Friday, March 30, 2012 4:56:09 PM
> Subject: Re: Accumulo on MapR Continued
> Can you share the program? I can try running it and answer your
> question. I can also of course check with MapR engineering to see if
> that's the issue.
> 
> 
> Much appreciated,
> Keys
> 
> ________________________________
> Keys Botzum
> Senior Principal Technologist
> WW Systems Engineering
> kbotzum@maprtech.com
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
> 
> 
> 
> 
> 
> On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:
> 
> 
> 
> On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <
> kbotzum@maprtech.com > wrote:
> 
> 
> 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> 
> 
> Seeking beyond EOF, filelen 733, wantpos 733
> 
> It looks like Accumulo is trying to seek to the last position in a
> file. I just wrote a little program that opens a file in HDFS and
> seeks to its file length, and it didn't throw an error. Does MapR not
> allow this? If that is the desired MapR behavior, we could try to
> determine whether Accumulo could avoid performing that seek.
> 
> Billie

Re: Accumulo on MapR Continued

Posted by Keys Botzum <kb...@maprtech.com>.
Can you share the program? I can try running it and answer your question. I can also of course check with MapR engineering to see if that's the issue.

Much appreciated,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
kbotzum@maprtech.com
443-718-0098
MapR Technologies
http://www.mapr.com



On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote:

> On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <kb...@maprtech.com> wrote:
>> 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
>> Seeking beyond EOF, filelen 733, wantpos 733
> 
> It looks like Accumulo is trying to seek to the last position in a file.  I just wrote a little program that opens a file in HDFS and seeks to its file length, and it didn't throw an error.  Does MapR not allow this?  If that is the desired MapR behavior, we could try to determine whether Accumulo could avoid performing that seek.
> 
> Billie


Re: Accumulo on MapR Continued

Posted by Billie J Rinaldi <bi...@ugov.gov>.
On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" <kb...@maprtech.com> wrote:
> 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey
> Seeking beyond EOF, filelen 733, wantpos 733

It looks like Accumulo is trying to seek to the last position in a file.  I just wrote a little program that opens a file in HDFS and seeks to its file length, and it didn't throw an error.  Does MapR not allow this?  If that is the desired MapR behavior, we could try to determine whether Accumulo could avoid performing that seek.

Billie