Space needed to user SequenceFile.Sorter

[Mark question <ma...@gmail.com>]

I don't know why I can't see my emails immediately sent to the group ...
anyways,

I'm sorting a sequenceFile using it's sorter on my local filesystem. The
inputFile size is 1937690478 bytes.

but after 14 minutes of sorting.. I get :

TEST SORTING ..
java.io.FileNotFoundException: File does not exist:
/usr/mark/tmp/mapred/local/SortedOutput.0
    at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
    at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:676)
    at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
    at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1353)
    at
org.apache.hadoop.io.SequenceFile$Sorter.cloneFileAttributes(SequenceFile.java:2663)
    at
org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:2712)
    at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2285)
    at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2324)
    at
CrossPartitionSimilarity.TestSorter(CrossPartitionSimilarity.java:164)
    at CrossPartitionSimilarity.main(CrossPartitionSimilarity.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


Yet, the file is still there:  wc -c SortedOutput.0   --->  1918661230
../tmp/mapred/local/SortedOutput.0
and  if it is because of space, I checked and it can hold up to 209 GB. So,
my question are there restrictions on some JVM configurations that I should
take care of ?

Thank you,
Maha

Re: cannot run more than 6 jvms on one machine

[Brian Bockelman <bb...@cse.unl.edu>]

Check the overcommit VM settings on your kernel.  These prevent swap from being used on older JVMs, and cause out-of-memory errors to be given by Java even when there is free memory.

Brian

On May 2, 2011, at 11:51 AM, Steve Loughran wrote:

> On 29/04/2011 03:37, stanley.shi@emc.com wrote:
>> Hi guys, I have a very strange problem here, hope someone can help.
>> 
>> I have a virtual machine with 8 GB memory, the os is CentOS 5.5 x64 version; I got this VM from someone else (means they may have done some configuration on this VM)
>> I cannot start even 6 easiest jvm at the same time (for example, start the following java code simultaneously 6 times:
>> 
>> import java.io.IOException;
>> public class InputTest {
>>         public static void main(String[] args) throws IOException, InterruptedException {
>>                 int length = 0;
>>                 long sum=0;
>>                 do {
>>                         sum+=length++;
>>                         System.out.println("Current sum: "+ sum);
>>                         Thread.sleep(1000);
>> 
>>                 } while (length<1000000);
>> 
>>         }
>> }
>> 
>> 
>> Any one  have any comments?
> 
> 
> 1. check swap space
> 2. you work for a storage company, so get some more...

Re: cannot run more than 6 jvms on one machine

[Steve Loughran <st...@apache.org>]

On 29/04/2011 03:37, stanley.shi@emc.com wrote:
> Hi guys, I have a very strange problem here, hope someone can help.
>
> I have a virtual machine with 8 GB memory, the os is CentOS 5.5 x64 version; I got this VM from someone else (means they may have done some configuration on this VM)
> I cannot start even 6 easiest jvm at the same time (for example, start the following java code simultaneously 6 times:
>
> import java.io.IOException;
> public class InputTest {
>          public static void main(String[] args) throws IOException, InterruptedException {
>                  int length = 0;
>                  long sum=0;
>                  do {
>                          sum+=length++;
>                          System.out.println("Current sum: "+ sum);
>                          Thread.sleep(1000);
>
>                  } while (length<1000000);
>
>          }
> }
>
>
> Any one  have any comments?


1. check swap space
2. you work for a storage company, so get some more...

cannot run more than 6 jvms on one machine

[st...@emc.com]

Hi guys, I have a very strange problem here, hope someone can help.

I have a virtual machine with 8 GB memory, the os is CentOS 5.5 x64 version; I got this VM from someone else (means they may have done some configuration on this VM)
I cannot start even 6 easiest jvm at the same time (for example, start the following java code simultaneously 6 times:

import java.io.IOException;
public class InputTest {
        public static void main(String[] args) throws IOException, InterruptedException {
                int length = 0;
                long sum=0;
                do {
                        sum+=length++;
                        System.out.println("Current sum: "+ sum);
                        Thread.sleep(1000);

                } while (length <1000000);

        }
}


Any one  have any comments?

Thanks,
Stanley