You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@apex.apache.org by Max Bridgewater <ma...@gmail.com> on 2016/11/18 17:41:42 UTC

Apex with SSH Tunel

Hi Folks,

I am playing with Apex (DataTorrent RTS Enterprise). Local deployment in
Ubuntu 16 box works fine.

However, when I deploy on a remote host, I am not apple to launch demo
applications. My suspicion is that this is due to having to open an SSH
tunnel to access the gateway. All activities other than launching the apps
seem to work fine.

My question: is there another port I need to open? Anybody is aware of
issues running/accessing Apex behind a proxy or firewall?

Unfortunately the UI does not provide much information. I am attaching some
screenshots.

Thanks,
Max.

Re: Apex with SSH Tunel

Posted by Munagala Ramanath <ra...@datatorrent.com>.

Since the diagnostic is about exceeding virtual memory limits, please see:
    http://docs.datatorrent.com/configuration/#yarn-vmem-pmem-ratio-tuning
for an alternative solution.

Ram

On Mon, Nov 21, 2016 at 4:20 AM, Max Bridgewater <ma...@gmail.com>
wrote:

> The issue turned out to be memory allocation. Here is the relevant YARN
> error message:
>
> 2016-11-21 11:44:30,020 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.container.ContainerImpl: Container
> container_1479728463466_0001_02_000001 transitioned from LOCALIZED to
> RUNNING
> 2016-11-21 11:44:31,858 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting
> resource-monitoring for container_1479728463466_0001_02_000001
> 2016-11-21 11:44:31,858 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping
> resource-monitoring for container_1479728463466_0001_01_000001
> 2016-11-21 11:44:31,867 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 26632 for container-id container_1479728463466_0001_02_000001:
> 194.5 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
> 2016-11-21 11:44:34,875 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage
> of ProcessTree 26632 for container-id container_1479728463466_0001_02_000001:
> 532.4 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used
> 2016-11-21 11:44:34,876 WARN org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree
> for container: container_1479728463466_0001_02_000001 has processes older
> than 1 iteration running over the configured limit. Limit=2254857728,
> current usage = 2822131712
> 2016-11-21 11:44:34,876 WARN org.apache.hadoop.yarn.server.
> nodemanager.containermanager.monitor.ContainersMonitorImpl: Container
> [pid=26632,containerID=container_1479728463466_0001_02_000001] is running
> beyond virtual memory limits. Current usage: 532.4 MB of 1 GB physical
> memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
>
>
> I solved it by adding this to yarn-site.xml:
>
> <property>
>         <name>yarn.scheduler.minimum-allocation-mb</name>
>         <value>1000</value>
> </property>
>
>
> Thanks,
> Max.
>
>
>
> On Sat, Nov 19, 2016 at 10:30 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
>> Max,
>>
>> The app failure does not depend on the gateway. The gateway is a daemon
>> to launch Apex apps on YARN and to get metrics for the Apex apps from YARN
>> and AM for each app, so it won't affect app execution once YARN accepts the
>> application. For some reason the AM itself is failing. I cannot figure out
>> the cause from the logs. It is possible that the app packages for these
>> apps have hadoop dependencies packaged, it is one of the most common causes
>> for AM failure.
>>
>> Regards,
>> Ashwin.
>>
>> On Sat, Nov 19, 2016 at 3:08 PM, Max Bridgewater <
>> max.bridgewater@gmail.com> wrote:
>>
>>> Please find the AppMaster.stderr attached as well as dt.log.
>>> AppMaster.stdout is empty. I am still wondering if there is another port
>>> that is needed or if the UI is using websocket.
>>>
>>> On Sat, Nov 19, 2016 at 5:40 PM, Ashwin Chandra Putta <
>>> ashwinchandrap@gmail.com> wrote:
>>>
>>>> Max,
>>>>
>>>> Can you share the app master logs of the failed application?
>>>>
>>>> Regards,
>>>> Ashwin.
>>>>
>>>> On Sat, Nov 19, 2016 at 4:45 AM, Max Bridgewater <
>>>> max.bridgewater@gmail.com> wrote:
>>>>
>>>>> Hi Ahswin,
>>>>>
>>>>> Thanks for the feedback. I created a completely new instance, trying
>>>>> the follow the instructions more precisely. I attached the logs again. As
>>>>> you can see they are very clean. Despite this, PIDemo is still failing
>>>>> without any meaningful error message. Same things happens with
>>>>> WorldCountDemo. After launching, it stays in ACCEPTED status for 10 to 15
>>>>> seconds and switch to FAILED.
>>>>>
>>>>> Max.
>>>>>
>>>>> On Fri, Nov 18, 2016 at 2:30 PM, Ashwin Chandra Putta <
>>>>> ashwinchandrap@gmail.com> wrote:
>>>>>
>>>>>> Also, there are write permission errors on /user/dtadmin/datatorrent
>>>>>> in hdfs. Please make dtadmin user own /user/dtadmin/
>>>>>>
>>>>>> Permission denied: user=dtadmin, access=WRITE,
>>>>>> inode="/user/dtadmin/datatorrent":hduser:supergroup:drwxr-xr-x
>>>>>>
>>>>>> Regards,
>>>>>> Ashwin.
>>>>>>
>>>>>> On Fri, Nov 18, 2016 at 11:27 AM, Ashwin Chandra Putta <
>>>>>> ashwinchandrap@gmail.com> wrote:
>>>>>>
>>>>>>> The end </property> tag is missing between line 30 and 31. It is for
>>>>>>> the property dt.attr.DEBUG.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ashwin.
>>>>>>>
>>>>>>> On Fri, Nov 18, 2016 at 10:16 AM, Max Bridgewater <
>>>>>>> max.bridgewater@gmail.com> wrote:
>>>>>>>
>>>>>>>> Here is the log folder. Note that it refers to a malformed
>>>>>>>> properties.xml. I am attaching that properties file as well.
>>>>>>>>
>>>>>>>> On Fri, Nov 18, 2016 at 1:08 PM, Ashwin Chandra Putta <
>>>>>>>> ashwinchandrap@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Max,
>>>>>>>>>
>>>>>>>>> Can you share the gateway logs?
>>>>>>>>>
>>>>>>>>> You will find them under /var/log/datatorrent for global install,
>>>>>>>>> or under ~/.dt/logs for local install.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Ashwin.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Ashwin.
>>>>>>>>>
>>>>>>>>> On Nov 18, 2016 9:41 AM, "Max Bridgewater" <
>>>>>>>>> max.bridgewater@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Folks,
>>>>>>>>>>
>>>>>>>>>> I am playing with Apex (DataTorrent RTS Enterprise). Local
>>>>>>>>>> deployment in Ubuntu 16 box works fine.
>>>>>>>>>>
>>>>>>>>>> However, when I deploy on a remote host, I am not apple to launch
>>>>>>>>>> demo applications. My suspicion is that this is due to having to open an
>>>>>>>>>> SSH tunnel to access the gateway. All activities other than launching the
>>>>>>>>>> apps seem to work fine.
>>>>>>>>>>
>>>>>>>>>> My question: is there another port I need to open? Anybody is
>>>>>>>>>> aware of issues running/accessing Apex behind a proxy or firewall?
>>>>>>>>>>
>>>>>>>>>> Unfortunately the UI does not provide much information. I am
>>>>>>>>>> attaching some screenshots.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Max.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ashwin.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Regards,
>>>>>> Ashwin.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards,
>>>> Ashwin.
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> Ashwin.
>>
>
>