You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Pratyush Das <re...@gmail.com> on 2022/05/23 22:26:22 UTC

Configuring Hadoop heap size

In hadoop-env.sh, I have set
export HADOOP_HEAPSIZE_MAX=30000
export HADOOP_HEAPSIZE_MIN=30000

And then restarted my Hadoop cluster with stop-all.sh and start-all.sh

When I print my Hadoop environment variables with hadoop envvar, the above
two variables aren't printed out.

I have a small program -
int main() {
  hdfsFS x2 = hdfsConnect("127.0.0.1", 9000);
  hdfsFile x3 = hdfsOpenFile(x2, "hdfs://127.0.0.1:9000/10G.txt", 0, 0, 0,
0); // A 10GB file
  int64_t size = hdfsAvailable(x2, x3);
  char *x5 = malloc(sizeof(char) * size);
  hdfsPread(x2, x3, 0, x5, size);
  free(x5);
  hdfsCloseFile(x2, x3);
  return 1;
}

that errors on hdfsPread with the error log -
hdfsPread: NewByteArray error:
OutOfMemoryError: Requestedarray size exceeds VM
limitjava.lang.OutOfMemoryError: Requested array size exceeds VM limit

It looks like the HEAPSIZE I set is not being respected. Could someone
please help me with this?

Do I need to format namenode for the new environment variables to be used?

-- 
Pratyush Das