You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Roman Shaposhnik <rv...@apache.org> on 2015/08/18 01:20:33 UTC

Weird networking behavior on Amazon EC via OSv deployment

Hi!

I'm working on creating a Geode appliance for OSv
along the same lines as what Cassandra has done
(slides are at the bottom of the page: http://osv.io/)

In my local environment everything works reasonably
OK, but when I deploy on Amazon EC I run into:
     https://groups.google.com/forum/#!topic/osv-dev/BpF0uSrMR-M

I'd appreciate if somebody can let me know:
    #1 if there are well-known gotchas with Geode
         and EC2 in general, especially when it comes
         to networking

    #2 how can I turn on full tracing/debug output of
         networking layer in order to potentially get additional
         clues to what's going on

Thanks,
Roman.

Re: Weird networking behavior on Amazon EC via OSv deployment

Posted by Xiaojian Zhou <gz...@pivotal.io>.
No. "ALL" does not count. I remembered that I have to manually specify the
range.

On Mon, Aug 17, 2015 at 6:06 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> On Mon, Aug 17, 2015 at 5:40 PM, Xiaojian Zhou <gz...@pivotal.io> wrote:
> > This error msg actually is not the root cause of the issue. To avoid the
> > error msg, you can modify sysctl.conf of your instance to add 2 lines:
> > # unicast buffer size
> > net.core.rmem_max = 2096304
> > net.core.wmem_max = 2096304
>
> Thanks for the suggestion, but I am actually not running it on Linux
> (or UNIX for that matter).
>
> > 2) You might not open enough ports for tcp and udp. That could be the
> real
> > root cause. To do that, you can either turn the ports (1024-65535) from
> AWS
> > GUI interface or via following commands:
> > aws ec2 authorize-security-group-ingress --group-id
> your_security_group_id
> > --protocol tcp --port 22 --cidr 0.0.0.0/00
> > aws ec2 authorize-security-group-ingress --group-id
> your_security_group_id
> > --protocol tcp --port 1024-65535 --cidr 0.0.0.0/00
> > aws ec2 authorize-security-group-ingress --group-id
> your_security_group_id
> > --protocol udp --port 1024-65535 --cidr 0.0.0.0/00
>
> Hm. That's a good point, but the security group I'm running it
> from has ALL the ports open so I guess it shouldn't be a problem
> then, right?
>
> Thanks,
> Roman.
>

Re: Weird networking behavior on Amazon EC via OSv deployment

Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Aug 17, 2015 at 5:40 PM, Xiaojian Zhou <gz...@pivotal.io> wrote:
> This error msg actually is not the root cause of the issue. To avoid the
> error msg, you can modify sysctl.conf of your instance to add 2 lines:
> # unicast buffer size
> net.core.rmem_max = 2096304
> net.core.wmem_max = 2096304

Thanks for the suggestion, but I am actually not running it on Linux
(or UNIX for that matter).

> 2) You might not open enough ports for tcp and udp. That could be the real
> root cause. To do that, you can either turn the ports (1024-65535) from AWS
> GUI interface or via following commands:
> aws ec2 authorize-security-group-ingress --group-id your_security_group_id
> --protocol tcp --port 22 --cidr 0.0.0.0/00
> aws ec2 authorize-security-group-ingress --group-id your_security_group_id
> --protocol tcp --port 1024-65535 --cidr 0.0.0.0/00
> aws ec2 authorize-security-group-ingress --group-id your_security_group_id
> --protocol udp --port 1024-65535 --cidr 0.0.0.0/00

Hm. That's a good point, but the security group I'm running it
from has ALL the ports open so I guess it shouldn't be a problem
then, right?

Thanks,
Roman.

Re: Weird networking behavior on Amazon EC via OSv deployment

Posted by Xiaojian Zhou <gz...@pivotal.io>.
Hi,

I did not use your image to reproduce, but for the 2 issues I think you can
try following 2 steps:
1) >> [warning 2015/08/16 04:36:15.260 GMT server3 <ReconnectThread>
>> tid=0x4a] requested unicast receive buffer size of 65,535 but got
>> 32,767: This is an operating system limit. If you would like to change
>> it, please see your system administrator.

This error msg actually is not the root cause of the issue. To avoid the
error msg, you can modify sysctl.conf of your instance to add 2 lines:
# unicast buffer size
net.core.rmem_max = 2096304
net.core.wmem_max = 2096304

2) You might not open enough ports for tcp and udp. That could be the real
root cause. To do that, you can either turn the ports (1024-65535) from AWS
GUI interface or via following commands:
aws ec2 authorize-security-group-ingress --group-id your_security_group_id
--protocol tcp --port 22 --cidr 0.0.0.0/00
aws ec2 authorize-security-group-ingress --group-id your_security_group_id
--protocol tcp --port 1024-65535 --cidr 0.0.0.0/00
aws ec2 authorize-security-group-ingress --group-id your_security_group_id
--protocol udp --port 1024-65535 --cidr 0.0.0.0/00

Regards
Xiaojian Zhou


On Mon, Aug 17, 2015 at 4:20 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> Hi!
>
> I'm working on creating a Geode appliance for OSv
> along the same lines as what Cassandra has done
> (slides are at the bottom of the page: http://osv.io/)
>
> In my local environment everything works reasonably
> OK, but when I deploy on Amazon EC I run into:
>      https://groups.google.com/forum/#!topic/osv-dev/BpF0uSrMR-M
>
> I'd appreciate if somebody can let me know:
>     #1 if there are well-known gotchas with Geode
>          and EC2 in general, especially when it comes
>          to networking
>
>     #2 how can I turn on full tracing/debug output of
>          networking layer in order to potentially get additional
>          clues to what's going on
>
> Thanks,
> Roman.
>