You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tajo.apache.org by Christian Schwabe <Ch...@gmx.com> on 2014/09/17 13:45:13 UTC
Worker Resources
Hello guys,
I promised I do not pretend to solve this problem together with you. I would like to redeem and more questions that bring us closer to a solution, hopefully this promise. My Macbook Pro has 8 GB of RAM and 4 cores available. My configuration in tajo-site.xml looks like this:
<configuration>
<property>
<name>tajo.resource-tracker.heartbeat.timeout-secs</name>
<value>240000</value> // or, your own longer value. default is 120*1000 (2 minutes)
</property>
<property>
<name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
<value>4</value>
</property>
<property>
<name>tajo.worker.resource.memory-mb</name>
<value>2048</value>
</property>
</configuration>
Actually, I would now expect a fourfold parallel processing. However, I can only see a maximum of three tasks at once with the status "running". With this configuration, I can still establish the following: Tajo processed, judging by the progress indicator is now more data than before. Up to now, only about 75% possible I land now at about 90% until Tajo crashed again.
I attached a log for this run. Also the following session variables were applied for this run:
tajo> \set JOIN_TASK_INPUT_SIZE 64
tajo> \set JOIN_PER_SHUFFLE_SIZE 64
Do you have any ideas I can try?
Best regards and very thankful,
Chris
Re: Worker Resources
Posted by Jinho Kim <jh...@apache.org>.
Hello Chris,
Looks like killed by kernel.
please update latest revision and increase the heap in tajo-env.sh.
export TAJO_WORKER_HEAPSIZE= parallel processing * 2GB or more
export TAJO_WORKER_OPTS="-XX:+UseParallelOldGC"
Worker resource is the following:
parallel worker(512 * 3) + query master (512)
Thanks
-Jinho
Best regards
2014-09-17 21:40 GMT+09:00 Christian Schwabe <Ch...@gmx.com>:
> Hello guys,
>
> The log has been strangely cut off, I noticed straight. Attached again the
> correct and complete log.
>
> Best regards,
>
> Chris
>
>
> Am 17.09.2014 um 13:45 schrieb Christian Schwabe
> <Ch...@gmx.com>:
>
> Hello guys,
>
> I promised I do not pretend to solve this problem together with you. I would
> like to redeem and more questions that bring us closer to a solution,
> hopefully this promise. My Macbook Pro has 8 GB of RAM and 4 cores
> available. My configuration in tajo-site.xml looks like this:
>
> <configuration>
> <property>
> <name>tajo.resource-tracker.heartbeat.timeout-secs</name>
> <value>240000</value> // or, your own longer value. default is 120*1000
> (2 minutes)
> </property>
> <property>
> <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
> <value>4</value>
> </property>
> <property>
> <name>tajo.worker.resource.memory-mb</name>
> <value>2048</value>
> </property>
> </configuration>
>
>
> Actually, I would now expect a fourfold parallel processing. However, I can
> only see a maximum of three tasks at once with the status"running". With
> this configuration, I can still establish the following: Tajo processed,
> judging by the progress indicator is now more data than before. Up to now,
> only about 75% possible I land now at about 90% until Tajo crashed again.
> I attached a log for this run. Also the following session variables were
> applied for this run:
> tajo> \set JOIN_TASK_INPUT_SIZE 64
> tajo> \set JOIN_PER_SHUFFLE_SIZE 64
>
> Do you have any ideas I can try?
>
>
>
> Best regards and very thankful,
> Chris
>
>
>
> <tajo-chris-master-christians-mbp.fritz.box.log><tajo-chris-worker-christians-mbp.fritz.box.log>
>
>
>
Re: Worker Resources
Posted by Christian Schwabe <Ch...@gmx.com>.
Hello guys,
The log has been strangely cut off, I noticed straight. Attached again the correct and complete log.
Best regards,
Chris
Am 17.09.2014 um 13:45 schrieb Christian Schwabe <Ch...@gmx.com>:
> Hello guys,
>
> I promised I do not pretend to solve this problem together with you. I would like to redeem and more questions that bring us closer to a solution, hopefully this promise. My Macbook Pro has 8 GB of RAM and 4 cores available. My configuration in tajo-site.xml looks like this:
>
> <configuration>
> <property>
> <name>tajo.resource-tracker.heartbeat.timeout-secs</name>
> <value>240000</value> // or, your own longer value. default is 120*1000 (2 minutes)
> </property>
> <property>
> <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
> <value>4</value>
> </property>
> <property>
> <name>tajo.worker.resource.memory-mb</name>
> <value>2048</value>
> </property>
> </configuration>
>
>
> Actually, I would now expect a fourfold parallel processing. However, I can only see a maximum of three tasks at once with the status"running". With this configuration, I can still establish the following: Tajo processed, judging by the progress indicator is now more data than before. Up to now, only about 75% possible I land now at about 90% until Tajo crashed again.
> I attached a log for this run. Also the following session variables were applied for this run:
> tajo> \set JOIN_TASK_INPUT_SIZE 64
> tajo> \set JOIN_PER_SHUFFLE_SIZE 64
>
> Do you have any ideas I can try?
>
>
>
> Best regards and very thankful,
> Chris
>
>
>
> <tajo-chris-master-christians-mbp.fritz.box.log><tajo-chris-worker-christians-mbp.fritz.box.log>