You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by sri hari kali charan Tummala <ka...@gmail.com> on 2019/09/26 12:49:35 UTC
Apache Ignite Cloud Formation Template
Hi All,
did anyone have an cloud formation example for deploying Apache Ignite on
AWS or steps to deploy or some successful deployment of Ignite onto AWS.
this below blog has missing artifacts so not working.
https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
--
Thanks & Regards
Sri Tummala
Re: Apache Ignite Cloud Formation Template
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
If that is really Apache Ignite 1.5, please don't use it. Please only use
the latest 2.7.6 release.
Regards,
--
Ilya Kasnacheev
пт, 27 сент. 2019 г. в 18:56, sri hari kali charan Tummala <
kali.tummala@gmail.com>:
> I found one example of cloud formation after launching the template and
> started to execute ignite.sh it is failing with class not found exception
> AWSCredentialsProviderWrapper.
>
> I sued:-
> Ignitefabric.1.5.0
> aws java sdk 1.3.21.1
>
>
>
> https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json
>
> ConfireIgnite.sh:-
>
> #!/bin/bash
>
> #
> # This is a modified version of the file stored at s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
> # which changes the config to use the instance provided credentials rather than requiring access/secret to be passed in
> #
> # Parameters are
> # 1 - Cache Name
> # 2 - Number of replicas
> # 3 - S3 Bucket Name
> #
>
> echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the \"License\"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an \"AS IS\" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
>
> <beans xmlns=\"http://www.springframework.org/schema/beans\"
> xmlns:util=\"http://www.springframework.org/schema/util\"
> xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"
> xsi:schemaLocation=\"
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://www.springframework.org/schema/util
> http://www.springframework.org/schema/util/spring-util.xsd\">
> <!--
> Alter configuration below as needed.
> -->
> <bean id=\"grid.cfg\" class=\"org.apache.ignite.configuration.IgniteConfiguration\">
> <property name=\"cacheConfiguration\">
> <list>
> <bean class=\"org.apache.ignite.configuration.CacheConfiguration\">
> <property name=\"name\" value=\"$1\"/>" > /tmp/igniteconfig.xml
>
> echo " <property name=\"cacheMode\" value=\"PARTITIONED\"/>
> <property name=\"atomicWriteOrderMode\" value=\"PRIMARY\"/>
> <property name=\"writeSynchronizationMode\" value=\"PRIMARY_SYNC\"/>" >> /tmp/igniteconfig.xml
>
> availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
> memoryOverhead=$((availfreeMemory/1024/1024/10))
> availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
> if [[ $availfreeMemoryinGB -gt 8 ]]; then
> offheapmemoryinGB=$((availfreeMemoryinGB-8))
> echo " <property name=\"memoryMode\" value=\"ONHEAP_TIERED\" />
> <property name=\"offHeapMaxMemory\" value=\"#{$offheapmemoryinGB * 1024L * 1024L * 1024L}\" />" >> /tmp/igniteconfig.xml
> echo "8g" > /tmp/heapsize.log
> else
> echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
> fi
> echo " <property name=\"evictionPolicy\">
> <bean class=\"org.apache.ignite.cache.eviction.lru.LruEvictionPolicy\">
> <property name=\"maxSize\" value=\"100000000\"/>
> </bean>
> </property>" >> /tmp/igniteconfig.xml
> echo " <property name=\"swapEnabled\" value=\"false\"/>" >> /tmp/igniteconfig.xml
> echo " <property name=\"atomicityMode\" value=\"ATOMIC\" />" >> /tmp/igniteconfig.xml
> echo " <property name=\"backups\" value=\"$2\" />" >> /tmp/igniteconfig.xml
> echo " </bean>" >> /tmp/igniteconfig.xml
> echo " </list>" >> /tmp/igniteconfig.xml
> echo "</property>" >> /tmp/igniteconfig.xml
> echo "<property name=\"discoverySpi\">
> <bean class=\"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi\">
> <property name=\"ipFinder\">
> <bean class=\"org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder\">
> <property name=\"awsCredentials\" ref=\"aws.creds\"/>
> <property name=\"bucketName\" value=\"$3\"/>
> </bean>
> </property>
> </bean>
> </property>
> <property name=\"communicationSpi\">
> <bean class=\"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi\">
> <property name=\"slowClientQueueLimit\" value=\"1000\"/>
> </bean>
> </property>
> </bean>
> <!-- AWS credentials. Use the default chain from the environment/instance. -->
> <bean id=\"aws.creds.factory\" class=\"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\"/>
> <bean id=\"aws.creds\" class=\"com.amazon.connections.data.auth.AWSCredentialsProviderWrapper\">
> <constructor-arg ref=\"aws.creds.factory\"/>
> </bean>
>
> </beans>" >> /tmp/igniteconfig.xml
>
>
> On Thu, Sep 26, 2019 at 5:34 PM sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> looks complex and hard to get it right, it would be nice if Ignite
>> community builds one working cloud formation template to launch ignite
>> cluster and in the cloud formation template output a JDBC connection for
>> connecting ignite cluster which would be awesome.
>>
>> Many would leverage this stack to build Ignite cluster on AWS or how
>> about ignite on EMR , while launching emr add ingite bootstrap script which
>> launches ignite in emr which would be lot easier also JDBC connection for
>> connecting ignite, this would change things for Ignite world.
>>
>> Thanks
>> Sri
>>
>>
>>
>> On Thu, Sep 26, 2019 at 3:51 PM Denis Magda <dm...@apache.org> wrote:
>>
>>> Hello Sri,
>>>
>>> You’re right, you need to start several aws instances to create a
>>> distributed cluster if you follow the guide below. Ignite configuration for
>>> all the nodes/instances has to have a properly configured IPFinder. You can
>>> use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
>>> are assigned dynamically.
>>> https://apacheignite.readme.io/docs/tcpip-discovery
>>>
>>> Let us know if it helps and we’ll update the docs.
>>>
>>> Denis
>>>
>>> On Thursday, September 26, 2019, sri hari kali charan Tummala <
>>> kali.tummala@gmail.com> wrote:
>>>
>>>> Hi Community,
>>>>
>>>> this doesn't launch a multinode ignite cluster am I right? it just
>>>> launches a single node ignite cluster on one ec2 instance.
>>>>
>>>> https://apacheignite.readme.io/docs/aws-deployment
>>>>
>>>> Thanks
>>>> Sri
>>>>
>>>> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
>>>> kali.tummala@gmail.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> did anyone have an cloud formation example for deploying Apache Ignite
>>>>> on AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>>>>>
>>>>> this below blog has missing artifacts so not working.
>>>>>
>>>>>
>>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>>
>>>>> --
>>>>> Thanks & Regards
>>>>> Sri Tummala
>>>>>
>>>>>
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>>>
>>> --
>>> -
>>> Denis
>>>
>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
Re: Apache Ignite Cloud Formation Template
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
I found one example of cloud formation after launching the template and
started to execute ignite.sh it is failing with class not found exception
AWSCredentialsProviderWrapper.
I sued:-
Ignitefabric.1.5.0
aws java sdk 1.3.21.1
https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json
ConfireIgnite.sh:-
#!/bin/bash
#
# This is a modified version of the file stored at
s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
# which changes the config to use the instance provided credentials
rather than requiring access/secret to be passed in
#
# Parameters are
# 1 - Cache Name
# 2 - Number of replicas
# 3 - S3 Bucket Name
#
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the \"License\"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an \"AS IS\" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans xmlns=\"http://www.springframework.org/schema/beans\"
xmlns:util=\"http://www.springframework.org/schema/util\"
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"
xsi:schemaLocation=\"
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd\">
<!--
Alter configuration below as needed.
-->
<bean id=\"grid.cfg\"
class=\"org.apache.ignite.configuration.IgniteConfiguration\">
<property name=\"cacheConfiguration\">
<list>
<bean class=\"org.apache.ignite.configuration.CacheConfiguration\">
<property name=\"name\" value=\"$1\"/>" > /tmp/igniteconfig.xml
echo " <property name=\"cacheMode\" value=\"PARTITIONED\"/>
<property name=\"atomicWriteOrderMode\" value=\"PRIMARY\"/>
<property name=\"writeSynchronizationMode\"
value=\"PRIMARY_SYNC\"/>" >> /tmp/igniteconfig.xml
availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
memoryOverhead=$((availfreeMemory/1024/1024/10))
availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
if [[ $availfreeMemoryinGB -gt 8 ]]; then
offheapmemoryinGB=$((availfreeMemoryinGB-8))
echo " <property name=\"memoryMode\" value=\"ONHEAP_TIERED\" />
<property name=\"offHeapMaxMemory\" value=\"#{$offheapmemoryinGB *
1024L * 1024L * 1024L}\" />" >> /tmp/igniteconfig.xml
echo "8g" > /tmp/heapsize.log
else
echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
fi
echo " <property name=\"evictionPolicy\">
<bean class=\"org.apache.ignite.cache.eviction.lru.LruEvictionPolicy\">
<property name=\"maxSize\" value=\"100000000\"/>
</bean>
</property>" >> /tmp/igniteconfig.xml
echo " <property name=\"swapEnabled\" value=\"false\"/>" >>
/tmp/igniteconfig.xml
echo " <property name=\"atomicityMode\" value=\"ATOMIC\" />" >>
/tmp/igniteconfig.xml
echo " <property name=\"backups\" value=\"$2\" />" >> /tmp/igniteconfig.xml
echo " </bean>" >> /tmp/igniteconfig.xml
echo " </list>" >> /tmp/igniteconfig.xml
echo "</property>" >> /tmp/igniteconfig.xml
echo "<property name=\"discoverySpi\">
<bean class=\"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi\">
<property name=\"ipFinder\">
<bean
class=\"org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder\">
<property name=\"awsCredentials\" ref=\"aws.creds\"/>
<property name=\"bucketName\" value=\"$3\"/>
</bean>
</property>
</bean>
</property>
<property name=\"communicationSpi\">
<bean class=\"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi\">
<property name=\"slowClientQueueLimit\" value=\"1000\"/>
</bean>
</property>
</bean>
<!-- AWS credentials. Use the default chain from the environment/instance. -->
<bean id=\"aws.creds.factory\"
class=\"com.amazonaws.auth.DefaultAWSCredentialsProviderChain\"/>
<bean id=\"aws.creds\"
class=\"com.amazon.connections.data.auth.AWSCredentialsProviderWrapper\">
<constructor-arg ref=\"aws.creds.factory\"/>
</bean>
</beans>" >> /tmp/igniteconfig.xml
On Thu, Sep 26, 2019 at 5:34 PM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> looks complex and hard to get it right, it would be nice if Ignite
> community builds one working cloud formation template to launch ignite
> cluster and in the cloud formation template output a JDBC connection for
> connecting ignite cluster which would be awesome.
>
> Many would leverage this stack to build Ignite cluster on AWS or how about
> ignite on EMR , while launching emr add ingite bootstrap script which
> launches ignite in emr which would be lot easier also JDBC connection for
> connecting ignite, this would change things for Ignite world.
>
> Thanks
> Sri
>
>
>
> On Thu, Sep 26, 2019 at 3:51 PM Denis Magda <dm...@apache.org> wrote:
>
>> Hello Sri,
>>
>> You’re right, you need to start several aws instances to create a
>> distributed cluster if you follow the guide below. Ignite configuration for
>> all the nodes/instances has to have a properly configured IPFinder. You can
>> use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
>> are assigned dynamically.
>> https://apacheignite.readme.io/docs/tcpip-discovery
>>
>> Let us know if it helps and we’ll update the docs.
>>
>> Denis
>>
>> On Thursday, September 26, 2019, sri hari kali charan Tummala <
>> kali.tummala@gmail.com> wrote:
>>
>>> Hi Community,
>>>
>>> this doesn't launch a multinode ignite cluster am I right? it just
>>> launches a single node ignite cluster on one ec2 instance.
>>>
>>> https://apacheignite.readme.io/docs/aws-deployment
>>>
>>> Thanks
>>> Sri
>>>
>>> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
>>> kali.tummala@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> did anyone have an cloud formation example for deploying Apache Ignite
>>>> on AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>>>>
>>>> this below blog has missing artifacts so not working.
>>>>
>>>>
>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>>
>> --
>> -
>> Denis
>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
--
Thanks & Regards
Sri Tummala
Re: Apache Ignite Cloud Formation Template
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
looks complex and hard to get it right, it would be nice if Ignite
community builds one working cloud formation template to launch ignite
cluster and in the cloud formation template output a JDBC connection for
connecting ignite cluster which would be awesome.
Many would leverage this stack to build Ignite cluster on AWS or how about
ignite on EMR , while launching emr add ingite bootstrap script which
launches ignite in emr which would be lot easier also JDBC connection for
connecting ignite, this would change things for Ignite world.
Thanks
Sri
On Thu, Sep 26, 2019 at 3:51 PM Denis Magda <dm...@apache.org> wrote:
> Hello Sri,
>
> You’re right, you need to start several aws instances to create a
> distributed cluster if you follow the guide below. Ignite configuration for
> all the nodes/instances has to have a properly configured IPFinder. You can
> use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
> are assigned dynamically.
> https://apacheignite.readme.io/docs/tcpip-discovery
>
> Let us know if it helps and we’ll update the docs.
>
> Denis
>
> On Thursday, September 26, 2019, sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> Hi Community,
>>
>> this doesn't launch a multinode ignite cluster am I right? it just
>> launches a single node ignite cluster on one ec2 instance.
>>
>> https://apacheignite.readme.io/docs/aws-deployment
>>
>> Thanks
>> Sri
>>
>> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
>> kali.tummala@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> did anyone have an cloud formation example for deploying Apache Ignite
>>> on AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>>>
>>> this below blog has missing artifacts so not working.
>>>
>>>
>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>
>
> --
> -
> Denis
>
>
--
Thanks & Regards
Sri Tummala
Re: Apache Ignite Cloud Formation Template
Posted by Denis Magda <dm...@apache.org>.
Hello Sri,
You’re right, you need to start several aws instances to create a
distributed cluster if you follow the guide below. Ignite configuration for
all the nodes/instances has to have a properly configured IPFinder. You can
use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
are assigned dynamically.
https://apacheignite.readme.io/docs/tcpip-discovery
Let us know if it helps and we’ll update the docs.
Denis
On Thursday, September 26, 2019, sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> Hi Community,
>
> this doesn't launch a multinode ignite cluster am I right? it just
> launches a single node ignite cluster on one ec2 instance.
>
> https://apacheignite.readme.io/docs/aws-deployment
>
> Thanks
> Sri
>
> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> Hi All,
>>
>> did anyone have an cloud formation example for deploying Apache Ignite on
>> AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>>
>> this below blog has missing artifacts so not working.
>>
>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-
>> oltp-and-analytics-with-apache-ignite-on-aws/
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
--
-
Denis
Re: Apache Ignite Cloud Formation Template
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
Hi Community,
this doesn't launch a multinode ignite cluster am I right? it just launches
a single node ignite cluster on one ec2 instance.
https://apacheignite.readme.io/docs/aws-deployment
Thanks
Sri
On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> Hi All,
>
> did anyone have an cloud formation example for deploying Apache Ignite on
> AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>
> this below blog has missing artifacts so not working.
>
>
> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>
> --
> Thanks & Regards
> Sri Tummala
>
>
--
Thanks & Regards
Sri Tummala