You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Devin Anderson <da...@akamai.com> on 2020/02/20 13:17:38 UTC
REST API requests hang with no response
Hi all,
I'm seeing issues wherein the Apache Ignite REST API appears to accept
requests, but doesn't ever reply. This doesn't always happen; for example, if
I make a request that I expect the API to reject, I get back a response:
----------
# curl -v -X GET 'http://127.0.0.1:8080/ignite'
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /ignite HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Date: Thu, 20 Feb 2020 11:21:22 GMT
< Content-Type: application/json;charset=utf-8
< Content-Length: 0
* Server Jetty(9.4.11.v20180605) is not blacklisted
< Server: Jetty(9.4.11.v20180605)
<
* Connection #0 to host 127.0.0.1 left intact
----------
I expect the (above) request above to fail because I don't supply the required
`cmd` parameter.
However, if I make a request that I believe should succeed, I never receive a
response:
----------
# curl -v 'http://127.0.0.1:8080/ignite?cmd=version'
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /ignite?cmd=version HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
>
----------
`curl` will wait forever for a response that never arrives.
I tried looking for logs output by the REST API so that I could debug the
issue, but failed to find any log entries emitted from the REST API.
Given that I'm new to Apache Ignite, I suspect that I'm almost certainly doing
something wrong, but I have absolutely no idea what that could be.
I have a few questions:
1.) Has anyone encountered an issue similar to this with the REST API? If so,
how did you solve it?
2.) Can anyone tell me where I might find REST API log entries?
3.) I have persistence enabled in the XML configuration. Would that conflict
with the REST API?
4.) I have additional nodes (3 nodes in total) in the XML configuration.
AFAICT from log entries and some liberal use of `ss`, the nodes are
communicating with each other. Would having multiple nodes conflict with the
REST API?
5.) Is there anything else I'm missing that might not be obvious to me because
I'm new to Apache Ignite?
Other than (3) and (4), I think the configuration file I'm using is rather bland:
----------
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<!--
Default memory region that grows endlessly. A cache is bound to this
memory region unless it sets another one in its CacheConfiguration/
-->
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="name" value="Default_Region"/>
<!-- 100 MB memory region with disabled eviction -->
<property name="initialSize" value="#{100 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>[other node address]</value><value>[other node
address]</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
----------
(Obviously, each instance of '[other node address]' is substituted with a real
IP address in the actual files I'm using.)
Thanks in advance for any help.
--
Devin
Re: REST API requests hang with no response
Posted by Devin Anderson <da...@akamai.com>.
Hi Ilya,
I have a working cluster now. Thanks again for your help. :)
Devin
On 2/21/20 4:31 AM, Devin Anderson wrote:
> Hi Ilya,
>
> Thank you very much. I suspect (1) is the missing piece of information I was
> looking for. I'll add it when I get a chance and will report back.
>
> Thanks again,
>
> Devin
>
> On 2/21/20 4:24 AM, Ilya Kasnacheev wrote:
>> Hello!
>>
>> I'll start in the reverse order:
>>
>> 3. If everything is OK, the process should take around a second (for
>> non-persistent cluster).
>> 2. As soon you see "Topology snapshot" in the log, the process is complete.
>> 1. From your configuration I see suspicious that it only lists two other
>> nodes and not itself. It is recommended that any node can discover itself
>> and start a cluster as 1st node. Try adding <value>[own ip address]</value>
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 21 февр. 2020 г. в 14:52, Devin Anderson <danderso@akamai.com
>> <ma...@akamai.com>>:
>>
>> Hi Ilya,
>>
>> That certainly makes sense, but I'm not totally sure how to act on that
>> information yet. A couple questions:
>>
>> 1.) Given the configuration file I posted (which is basically the same on
>> each
>> node in the cluster, save that the IP addresses of the nodes to discover
>> are
>> different), should each of the nodes eventually join the cluster? Am I
>> missing
>> anything?
>> 2.) Is there a message that I should see in the logs that identifies that
>> the
>> node has successfully joined the cluster?
>> 3.) How long should I expect the process to take with three nodes (just a
>> loose
>> estimate)?
>>
>> Thanks for your ongoing help. It's much appreciated. :)
>>
>> Devin
>>
>> On 2/21/20 3:23 AM, Ilya Kasnacheev wrote:
>> > Hello!
>> >
>> > Your node has never finished joining to cluster nor was able to
>> self-discover
>> > to form a cluster of its own, as evident by:
>> >
>> >
>> > "main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
>> > condition [0x00007f7e8875f000]
>> > java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
>> > at
>> >
>> org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*(ServerImpl.java:1131)
>> > at
>> >
>> org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*(ServerImpl.java:910)
>> > at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
>> > at
>> >
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
>> > at
>> >
>> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>> > at
>> >
>> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
>> > at
>> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
>> > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
>> > at
>> >
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
>> > at
>> >
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
>> > - locked <0x00000005e8cca5b8> (a
>> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
>> > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
>> > at
>> >
>> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
>> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
>> > at org.apache.ignite.Ignition.start(Ignition.java:348)
>> > at
>> >
>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
>> >
>> > Until node is in topology, obviously it can't serve any requests.
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > пт, 21 февр. 2020 г. в 01:55, Devin Anderson <danderso@akamai.com
>> <ma...@akamai.com>
>> > <mailto:danderso@akamai.com <ma...@akamai.com>>>:
>> >
>> > Hi Ilya,
>> >
>> > I'm attaching the `jstack` dump to this message. When I took the
>> dump,
>> > there
>> > were ten requests that had been made to the Apache Ignite REST API
>> using
>> > POST
>> > data of the form:
>> >
>> > cmd=add&key=[key]&value=[value]
>> >
>> > ... where [key] and [value] were proper URI encoded data.
>> >
>> > Thanks in advance for any help. I'll take a look as well and try to
>> > figure out
>> > what's going on.
>> >
>> > Devin
>> >
>> > On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
>> > > Hello!
>> > >
>> > > Please collect thread dump (jstack) from affected node, share it
>> with us.
>> > >
>> > > Regards,
>> > > --
>> > > Ilya Kasnacheev
>> > >
>> > >
>> > > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <danderso@akamai.com
>> <ma...@akamai.com>
>> > <mailto:danderso@akamai.com <ma...@akamai.com>>
>> > > <mailto:danderso@akamai.com <ma...@akamai.com>
>> <mailto:danderso@akamai.com <ma...@akamai.com>>>>:
>> > >
>> > > Hi all,
>> > >
>> > > I'm seeing issues wherein the Apache Ignite REST API appears
>> to accept
>> > > requests, but doesn't ever reply. This doesn't always
>> happen; for
>> > > example, if
>> > > I make a request that I expect the API to reject, I get back a
>> > response:
>> > >
>> > > ----------
>> > >
>> > > # curl -v -X GET 'http://127.0.0.1:8080/ignite
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=bT604T6RDd_DwnVyBCW-NHSDUMXvUZq9XsvGzHDbueI&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=Hhk_7aUuf0SyczRr-esByr4-rhF7d6_AdK2FyS_MjFo&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
>> > > * Hostname was NOT found in DNS cache
>> > > * Trying 127.0.0.1...
>> > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
>> > > > GET /ignite HTTP/1.1
>> > > > User-Agent: curl/7.35.0
>> > > > Host: 127.0.0.1:8080
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=xHljiS3IT9e_IL3uT8G8JEJ4wV1eSPx2p4I8oJOPkfc&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
>> > > > Accept: */*
>> > > >
>> > > < HTTP/1.1 400 Bad Request
>> > > < Date: Thu, 20 Feb 2020 11:21:22 GMT
>> > > < Content-Type: application/json;charset=utf-8
>> > > < Content-Length: 0
>> > > * Server Jetty(9.4.11.v20180605) is not blacklisted
>> > > < Server: Jetty(9.4.11.v20180605)
>> > > <
>> > > * Connection #0 to host 127.0.0.1 left intact
>> > >
>> > > ----------
>> > >
>> > > I expect the (above) request above to fail because I don't
>> supply the
>> > > required
>> > > `cmd` parameter.
>> > >
>> > > However, if I make a request that I believe should succeed, I
>> never
>> > > receive a
>> > > response:
>> > >
>> > > ----------
>> > >
>> > > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=ZJdkLEsuKL4bxIqpkGT9pyoxIYEgK40dRUHMJHGytTs&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=b1MZBDEsOiVfh5l7ltu2DgXaYnh_5XtnKPg1RgsDfTI&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
>> > > * Hostname was NOT found in DNS cache
>> > > * Trying 127.0.0.1...
>> > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
>> > > > GET /ignite?cmd=version HTTP/1.1
>> > > > User-Agent: curl/7.35.0
>> > > > Host: 127.0.0.1:8080
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=xHljiS3IT9e_IL3uT8G8JEJ4wV1eSPx2p4I8oJOPkfc&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
>> > > > Accept: */*
>> > > >
>> > >
>> > > ----------
>> > >
>> > > `curl` will wait forever for a response that never arrives.
>> > >
>> > > I tried looking for logs output by the REST API so that I could
>> > debug the
>> > > issue, but failed to find any log entries emitted from the
>> REST API.
>> > >
>> > > Given that I'm new to Apache Ignite, I suspect that I'm almost
>> > certainly
>> > > doing
>> > > something wrong, but I have absolutely no idea what that
>> could be.
>> > >
>> > > I have a few questions:
>> > >
>> > > 1.) Has anyone encountered an issue similar to this with the
>> REST API?
>> > > If so,
>> > > how did you solve it?
>> > > 2.) Can anyone tell me where I might find REST API log entries?
>> > > 3.) I have persistence enabled in the XML configuration.
>> Would that
>> > conflict
>> > > with the REST API?
>> > > 4.) I have additional nodes (3 nodes in total) in the XML
>> > configuration.
>> > > AFAICT from log entries and some liberal use of `ss`, the
>> nodes are
>> > > communicating with each other. Would having multiple nodes
>> > conflict with
>> > > the
>> > > REST API?
>> > > 5.) Is there anything else I'm missing that might not be
>> obvious to me
>> > > because
>> > > I'm new to Apache Ignite?
>> > >
>> > > Other than (3) and (4), I think the configuration file I'm
>> using is
>> > > rather bland:
>> > >
>> > > ----------
>> > >
>> > > <beans xmlns="http://www.springframework.org/schema/beans
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=8xrngwOS5xaTscJxH7HOC8fKhh5BON0ani75xsIpjUo&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
>> > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=7PzeJ7X6RCdABzpGj5jdaon_Eks7xHDjJoBXZ7gyu7o&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=UyopUd5Ot09OGVA6KNi_oTS8JCxjI42wdeGJQbrC2UQ&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
>> > > xsi:schemaLocation="
>> > > http://www.springframework.org/schema/beans
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=8xrngwOS5xaTscJxH7HOC8fKhh5BON0ani75xsIpjUo&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
>> > > http://www.springframework.org/schema/beans/spring-beans.xsd
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=DVCBX8SLQnDmVGJ5Zs9Wb3q4TH7tGwWBK3mxXRiXmOQ&e=>
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=1W2X_AlApjVwM2r9Q9I8hQniVWhjpGvzeGWSbA7z0Oc&e=>
>> > >
>> >
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
>> > > <bean
>> class="org.apache.ignite.configuration.IgniteConfiguration">
>> > >
>> > > <property name="dataStorageConfiguration">
>> > > <bean
>> > > class="org.apache.ignite.configuration.DataStorageConfiguration">
>> > > <!--
>> > > Default memory region that grows endlessly. A
>> cache is bound
>> > > to this
>> > > memory region unless it sets another one in its
>> > > CacheConfiguration/
>> > > -->
>> > > <property name="defaultDataRegionConfiguration">
>> > > <bean
>> > > class="org.apache.ignite.configuration.DataRegionConfiguration">
>> > > <property name="persistenceEnabled" value="true"/>
>> > > <property name="name" value="Default_Region"/>
>> > > <!-- 100 MB memory region with disabled
>> eviction -->
>> > > <property name="initialSize" value="#{100 * 1024 *
>> > 1024}"/>
>> > > </bean>
>> > > </property>
>> > > </bean>
>> > > </property>
>> > >
>> > > <property name="discoverySpi">
>> > > <bean
>> > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>> > > <property name="ipFinder">
>> > > <bean
>> > >
>> >
>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>> > > <property name="addresses">
>> > > <list>
>> > > <value>[other node
>> address]</value><value>[other node
>> > > address]</value>
>> > > </list>
>> > > </property>
>> > > </bean>
>> > > </property>
>> > > </bean>
>> > > </property>
>> > >
>> > > </bean>
>> > > </beans>
>> > >
>> > > ----------
>> > >
>> > > (Obviously, each instance of '[other node address]' is
>> substituted
>> > with a
>> > > real
>> > > IP address in the actual files I'm using.)
>> > >
>> > > Thanks in advance for any help.
>> > >
>> > > --
>> > > Devin
>> > >
>> >
>>
>
Re: REST API requests hang with no response
Posted by Devin Anderson <da...@akamai.com>.
Hi Ilya,
Thank you very much. I suspect (1) is the missing piece of information I was
looking for. I'll add it when I get a chance and will report back.
Thanks again,
Devin
On 2/21/20 4:24 AM, Ilya Kasnacheev wrote:
> Hello!
>
> I'll start in the reverse order:
>
> 3. If everything is OK, the process should take around a second (for
> non-persistent cluster).
> 2. As soon you see "Topology snapshot" in the log, the process is complete.
> 1. From your configuration I see suspicious that it only lists two other
> nodes and not itself. It is recommended that any node can discover itself and
> start a cluster as 1st node. Try adding <value>[own ip address]</value>
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 21 февр. 2020 г. в 14:52, Devin Anderson <danderso@akamai.com
> <ma...@akamai.com>>:
>
> Hi Ilya,
>
> That certainly makes sense, but I'm not totally sure how to act on that
> information yet. A couple questions:
>
> 1.) Given the configuration file I posted (which is basically the same on
> each
> node in the cluster, save that the IP addresses of the nodes to discover are
> different), should each of the nodes eventually join the cluster? Am I
> missing
> anything?
> 2.) Is there a message that I should see in the logs that identifies that
> the
> node has successfully joined the cluster?
> 3.) How long should I expect the process to take with three nodes (just a
> loose
> estimate)?
>
> Thanks for your ongoing help. It's much appreciated. :)
>
> Devin
>
> On 2/21/20 3:23 AM, Ilya Kasnacheev wrote:
> > Hello!
> >
> > Your node has never finished joining to cluster nor was able to
> self-discover
> > to form a cluster of its own, as evident by:
> >
> >
> > "main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
> > condition [0x00007f7e8875f000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
> > at
> >
> org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*(ServerImpl.java:1131)
> > at
> >
> org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*(ServerImpl.java:910)
> > at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
> > at
> >
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
> > at
> >
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> > at
> >
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
> > at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
> > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
> > at
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> > at
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> > - locked <0x00000005e8cca5b8> (a
> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
> > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> > at
> >
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> > at org.apache.ignite.Ignition.start(Ignition.java:348)
> > at
> >
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
> >
> > Until node is in topology, obviously it can't serve any requests.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 21 февр. 2020 г. в 01:55, Devin Anderson <danderso@akamai.com
> <ma...@akamai.com>
> > <mailto:danderso@akamai.com <ma...@akamai.com>>>:
> >
> > Hi Ilya,
> >
> > I'm attaching the `jstack` dump to this message. When I took the dump,
> > there
> > were ten requests that had been made to the Apache Ignite REST API
> using
> > POST
> > data of the form:
> >
> > cmd=add&key=[key]&value=[value]
> >
> > ... where [key] and [value] were proper URI encoded data.
> >
> > Thanks in advance for any help. I'll take a look as well and try to
> > figure out
> > what's going on.
> >
> > Devin
> >
> > On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> > > Hello!
> > >
> > > Please collect thread dump (jstack) from affected node, share it
> with us.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <danderso@akamai.com
> <ma...@akamai.com>
> > <mailto:danderso@akamai.com <ma...@akamai.com>>
> > > <mailto:danderso@akamai.com <ma...@akamai.com>
> <mailto:danderso@akamai.com <ma...@akamai.com>>>>:
> > >
> > > Hi all,
> > >
> > > I'm seeing issues wherein the Apache Ignite REST API appears
> to accept
> > > requests, but doesn't ever reply. This doesn't always
> happen; for
> > > example, if
> > > I make a request that I expect the API to reject, I get back a
> > response:
> > >
> > > ----------
> > >
> > > # curl -v -X GET 'http://127.0.0.1:8080/ignite
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=bT604T6RDd_DwnVyBCW-NHSDUMXvUZq9XsvGzHDbueI&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=Hhk_7aUuf0SyczRr-esByr4-rhF7d6_AdK2FyS_MjFo&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
> > > * Hostname was NOT found in DNS cache
> > > * Trying 127.0.0.1...
> > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > > GET /ignite HTTP/1.1
> > > > User-Agent: curl/7.35.0
> > > > Host: 127.0.0.1:8080
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=xHljiS3IT9e_IL3uT8G8JEJ4wV1eSPx2p4I8oJOPkfc&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > > > Accept: */*
> > > >
> > > < HTTP/1.1 400 Bad Request
> > > < Date: Thu, 20 Feb 2020 11:21:22 GMT
> > > < Content-Type: application/json;charset=utf-8
> > > < Content-Length: 0
> > > * Server Jetty(9.4.11.v20180605) is not blacklisted
> > > < Server: Jetty(9.4.11.v20180605)
> > > <
> > > * Connection #0 to host 127.0.0.1 left intact
> > >
> > > ----------
> > >
> > > I expect the (above) request above to fail because I don't
> supply the
> > > required
> > > `cmd` parameter.
> > >
> > > However, if I make a request that I believe should succeed, I
> never
> > > receive a
> > > response:
> > >
> > > ----------
> > >
> > > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=ZJdkLEsuKL4bxIqpkGT9pyoxIYEgK40dRUHMJHGytTs&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=b1MZBDEsOiVfh5l7ltu2DgXaYnh_5XtnKPg1RgsDfTI&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
> > > * Hostname was NOT found in DNS cache
> > > * Trying 127.0.0.1...
> > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > > GET /ignite?cmd=version HTTP/1.1
> > > > User-Agent: curl/7.35.0
> > > > Host: 127.0.0.1:8080
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=xHljiS3IT9e_IL3uT8G8JEJ4wV1eSPx2p4I8oJOPkfc&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > > > Accept: */*
> > > >
> > >
> > > ----------
> > >
> > > `curl` will wait forever for a response that never arrives.
> > >
> > > I tried looking for logs output by the REST API so that I could
> > debug the
> > > issue, but failed to find any log entries emitted from the
> REST API.
> > >
> > > Given that I'm new to Apache Ignite, I suspect that I'm almost
> > certainly
> > > doing
> > > something wrong, but I have absolutely no idea what that
> could be.
> > >
> > > I have a few questions:
> > >
> > > 1.) Has anyone encountered an issue similar to this with the
> REST API?
> > > If so,
> > > how did you solve it?
> > > 2.) Can anyone tell me where I might find REST API log entries?
> > > 3.) I have persistence enabled in the XML configuration.
> Would that
> > conflict
> > > with the REST API?
> > > 4.) I have additional nodes (3 nodes in total) in the XML
> > configuration.
> > > AFAICT from log entries and some liberal use of `ss`, the
> nodes are
> > > communicating with each other. Would having multiple nodes
> > conflict with
> > > the
> > > REST API?
> > > 5.) Is there anything else I'm missing that might not be
> obvious to me
> > > because
> > > I'm new to Apache Ignite?
> > >
> > > Other than (3) and (4), I think the configuration file I'm
> using is
> > > rather bland:
> > >
> > > ----------
> > >
> > > <beans xmlns="http://www.springframework.org/schema/beans
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=8xrngwOS5xaTscJxH7HOC8fKhh5BON0ani75xsIpjUo&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
> > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=7PzeJ7X6RCdABzpGj5jdaon_Eks7xHDjJoBXZ7gyu7o&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=UyopUd5Ot09OGVA6KNi_oTS8JCxjI42wdeGJQbrC2UQ&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
> > > xsi:schemaLocation="
> > > http://www.springframework.org/schema/beans
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=8xrngwOS5xaTscJxH7HOC8fKhh5BON0ani75xsIpjUo&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
> > > http://www.springframework.org/schema/beans/spring-beans.xsd
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=YDzIr351UmsDB_ZlazckzGvBmGsn219gHDdNtIJyoyM&s=DVCBX8SLQnDmVGJ5Zs9Wb3q4TH7tGwWBK3mxXRiXmOQ&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=1W2X_AlApjVwM2r9Q9I8hQniVWhjpGvzeGWSbA7z0Oc&e=>
> > >
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
> > > <bean
> class="org.apache.ignite.configuration.IgniteConfiguration">
> > >
> > > <property name="dataStorageConfiguration">
> > > <bean
> > > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > > <!--
> > > Default memory region that grows endlessly. A
> cache is bound
> > > to this
> > > memory region unless it sets another one in its
> > > CacheConfiguration/
> > > -->
> > > <property name="defaultDataRegionConfiguration">
> > > <bean
> > > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > > <property name="persistenceEnabled" value="true"/>
> > > <property name="name" value="Default_Region"/>
> > > <!-- 100 MB memory region with disabled eviction -->
> > > <property name="initialSize" value="#{100 * 1024 *
> > 1024}"/>
> > > </bean>
> > > </property>
> > > </bean>
> > > </property>
> > >
> > > <property name="discoverySpi">
> > > <bean
> > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > > <property name="ipFinder">
> > > <bean
> > >
> >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > > <property name="addresses">
> > > <list>
> > > <value>[other node
> address]</value><value>[other node
> > > address]</value>
> > > </list>
> > > </property>
> > > </bean>
> > > </property>
> > > </bean>
> > > </property>
> > >
> > > </bean>
> > > </beans>
> > >
> > > ----------
> > >
> > > (Obviously, each instance of '[other node address]' is
> substituted
> > with a
> > > real
> > > IP address in the actual files I'm using.)
> > >
> > > Thanks in advance for any help.
> > >
> > > --
> > > Devin
> > >
> >
>
Re: REST API requests hang with no response
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I'll start in the reverse order:
3. If everything is OK, the process should take around a second (for
non-persistent cluster).
2. As soon you see "Topology snapshot" in the log, the process is complete.
1. From your configuration I see suspicious that it only lists two other
nodes and not itself. It is recommended that any node can discover itself
and start a cluster as 1st node. Try adding <value>[own ip address]</value>
Regards,
--
Ilya Kasnacheev
пт, 21 февр. 2020 г. в 14:52, Devin Anderson <da...@akamai.com>:
> Hi Ilya,
>
> That certainly makes sense, but I'm not totally sure how to act on that
> information yet. A couple questions:
>
> 1.) Given the configuration file I posted (which is basically the same on
> each
> node in the cluster, save that the IP addresses of the nodes to discover
> are
> different), should each of the nodes eventually join the cluster? Am I
> missing
> anything?
> 2.) Is there a message that I should see in the logs that identifies that
> the
> node has successfully joined the cluster?
> 3.) How long should I expect the process to take with three nodes (just a
> loose
> estimate)?
>
> Thanks for your ongoing help. It's much appreciated. :)
>
> Devin
>
> On 2/21/20 3:23 AM, Ilya Kasnacheev wrote:
> > Hello!
> >
> > Your node has never finished joining to cluster nor was able to
> self-discover
> > to form a cluster of its own, as evident by:
> >
> >
> > "main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
> > condition [0x00007f7e8875f000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
> > at
> >
> org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*(ServerImpl.java:1131)
> > at
> >
> org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*(ServerImpl.java:910)
> > at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
> > at
> >
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
> > at
> >
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> > at
> >
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
> > at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
> > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
> > at
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> > at
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> > - locked <0x00000005e8cca5b8> (a
> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
> > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> > at
> >
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> > at org.apache.ignite.Ignition.start(Ignition.java:348)
> > at
> >
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
> >
> > Until node is in topology, obviously it can't serve any requests.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 21 февр. 2020 г. в 01:55, Devin Anderson <danderso@akamai.com
> > <ma...@akamai.com>>:
> >
> > Hi Ilya,
> >
> > I'm attaching the `jstack` dump to this message. When I took the
> dump,
> > there
> > were ten requests that had been made to the Apache Ignite REST API
> using
> > POST
> > data of the form:
> >
> > cmd=add&key=[key]&value=[value]
> >
> > ... where [key] and [value] were proper URI encoded data.
> >
> > Thanks in advance for any help. I'll take a look as well and try to
> > figure out
> > what's going on.
> >
> > Devin
> >
> > On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> > > Hello!
> > >
> > > Please collect thread dump (jstack) from affected node, share it
> with us.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <danderso@akamai.com
> > <ma...@akamai.com>
> > > <mailto:danderso@akamai.com <ma...@akamai.com>>>:
> > >
> > > Hi all,
> > >
> > > I'm seeing issues wherein the Apache Ignite REST API appears
> to accept
> > > requests, but doesn't ever reply. This doesn't always happen;
> for
> > > example, if
> > > I make a request that I expect the API to reject, I get back a
> > response:
> > >
> > > ----------
> > >
> > > # curl -v -X GET 'http://127.0.0.1:8080/ignite
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=Hhk_7aUuf0SyczRr-esByr4-rhF7d6_AdK2FyS_MjFo&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=
> >'
> > > * Hostname was NOT found in DNS cache
> > > * Trying 127.0.0.1...
> > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > > GET /ignite HTTP/1.1
> > > > User-Agent: curl/7.35.0
> > > > Host: 127.0.0.1:8080
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=
> >
> > > > Accept: */*
> > > >
> > > < HTTP/1.1 400 Bad Request
> > > < Date: Thu, 20 Feb 2020 11:21:22 GMT
> > > < Content-Type: application/json;charset=utf-8
> > > < Content-Length: 0
> > > * Server Jetty(9.4.11.v20180605) is not blacklisted
> > > < Server: Jetty(9.4.11.v20180605)
> > > <
> > > * Connection #0 to host 127.0.0.1 left intact
> > >
> > > ----------
> > >
> > > I expect the (above) request above to fail because I don't
> supply the
> > > required
> > > `cmd` parameter.
> > >
> > > However, if I make a request that I believe should succeed, I
> never
> > > receive a
> > > response:
> > >
> > > ----------
> > >
> > > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=b1MZBDEsOiVfh5l7ltu2DgXaYnh_5XtnKPg1RgsDfTI&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=
> >'
> > > * Hostname was NOT found in DNS cache
> > > * Trying 127.0.0.1...
> > > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > > GET /ignite?cmd=version HTTP/1.1
> > > > User-Agent: curl/7.35.0
> > > > Host: 127.0.0.1:8080
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=
> >
> > > > Accept: */*
> > > >
> > >
> > > ----------
> > >
> > > `curl` will wait forever for a response that never arrives.
> > >
> > > I tried looking for logs output by the REST API so that I could
> > debug the
> > > issue, but failed to find any log entries emitted from the
> REST API.
> > >
> > > Given that I'm new to Apache Ignite, I suspect that I'm almost
> > certainly
> > > doing
> > > something wrong, but I have absolutely no idea what that could
> be.
> > >
> > > I have a few questions:
> > >
> > > 1.) Has anyone encountered an issue similar to this with the
> REST API?
> > > If so,
> > > how did you solve it?
> > > 2.) Can anyone tell me where I might find REST API log entries?
> > > 3.) I have persistence enabled in the XML configuration. Would
> that
> > conflict
> > > with the REST API?
> > > 4.) I have additional nodes (3 nodes in total) in the XML
> > configuration.
> > > AFAICT from log entries and some liberal use of `ss`, the
> nodes are
> > > communicating with each other. Would having multiple nodes
> > conflict with
> > > the
> > > REST API?
> > > 5.) Is there anything else I'm missing that might not be
> obvious to me
> > > because
> > > I'm new to Apache Ignite?
> > >
> > > Other than (3) and (4), I think the configuration file I'm
> using is
> > > rather bland:
> > >
> > > ----------
> > >
> > > <beans xmlns="http://www.springframework.org/schema/beans
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=
> >"
> > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=UyopUd5Ot09OGVA6KNi_oTS8JCxjI42wdeGJQbrC2UQ&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=
> >"
> > > xsi:schemaLocation="
> > > http://www.springframework.org/schema/beans
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=
> >
> > > http://www.springframework.org/schema/beans/spring-beans.xsd
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=1W2X_AlApjVwM2r9Q9I8hQniVWhjpGvzeGWSbA7z0Oc&e=
> >
> > >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=
> >">
> > > <bean
> class="org.apache.ignite.configuration.IgniteConfiguration">
> > >
> > > <property name="dataStorageConfiguration">
> > > <bean
> > > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > > <!--
> > > Default memory region that grows endlessly. A cache
> is bound
> > > to this
> > > memory region unless it sets another one in its
> > > CacheConfiguration/
> > > -->
> > > <property name="defaultDataRegionConfiguration">
> > > <bean
> > > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > > <property name="persistenceEnabled" value="true"/>
> > > <property name="name" value="Default_Region"/>
> > > <!-- 100 MB memory region with disabled eviction
> -->
> > > <property name="initialSize" value="#{100 * 1024 *
> > 1024}"/>
> > > </bean>
> > > </property>
> > > </bean>
> > > </property>
> > >
> > > <property name="discoverySpi">
> > > <bean
> > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > > <property name="ipFinder">
> > > <bean
> > >
> >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > > <property name="addresses">
> > > <list>
> > > <value>[other node
> address]</value><value>[other node
> > > address]</value>
> > > </list>
> > > </property>
> > > </bean>
> > > </property>
> > > </bean>
> > > </property>
> > >
> > > </bean>
> > > </beans>
> > >
> > > ----------
> > >
> > > (Obviously, each instance of '[other node address]' is
> substituted
> > with a
> > > real
> > > IP address in the actual files I'm using.)
> > >
> > > Thanks in advance for any help.
> > >
> > > --
> > > Devin
> > >
> >
>
>
Re: REST API requests hang with no response
Posted by Devin Anderson <da...@akamai.com>.
Hi Ilya,
That certainly makes sense, but I'm not totally sure how to act on that
information yet. A couple questions:
1.) Given the configuration file I posted (which is basically the same on each
node in the cluster, save that the IP addresses of the nodes to discover are
different), should each of the nodes eventually join the cluster? Am I missing
anything?
2.) Is there a message that I should see in the logs that identifies that the
node has successfully joined the cluster?
3.) How long should I expect the process to take with three nodes (just a loose
estimate)?
Thanks for your ongoing help. It's much appreciated. :)
Devin
On 2/21/20 3:23 AM, Ilya Kasnacheev wrote:
> Hello!
>
> Your node has never finished joining to cluster nor was able to self-discover
> to form a cluster of its own, as evident by:
>
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
> condition [0x00007f7e8875f000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
> at
> org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*(ServerImpl.java:1131)
> at
> org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*(ServerImpl.java:910)
> at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
> at
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
> at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> - locked <0x00000005e8cca5b8> (a
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> at org.apache.ignite.Ignition.start(Ignition.java:348)
> at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
>
> Until node is in topology, obviously it can't serve any requests.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 21 февр. 2020 г. в 01:55, Devin Anderson <danderso@akamai.com
> <ma...@akamai.com>>:
>
> Hi Ilya,
>
> I'm attaching the `jstack` dump to this message. When I took the dump,
> there
> were ten requests that had been made to the Apache Ignite REST API using
> POST
> data of the form:
>
> cmd=add&key=[key]&value=[value]
>
> ... where [key] and [value] were proper URI encoded data.
>
> Thanks in advance for any help. I'll take a look as well and try to
> figure out
> what's going on.
>
> Devin
>
> On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> > Hello!
> >
> > Please collect thread dump (jstack) from affected node, share it with us.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <danderso@akamai.com
> <ma...@akamai.com>
> > <mailto:danderso@akamai.com <ma...@akamai.com>>>:
> >
> > Hi all,
> >
> > I'm seeing issues wherein the Apache Ignite REST API appears to accept
> > requests, but doesn't ever reply. This doesn't always happen; for
> > example, if
> > I make a request that I expect the API to reject, I get back a
> response:
> >
> > ----------
> >
> > # curl -v -X GET 'http://127.0.0.1:8080/ignite
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=Hhk_7aUuf0SyczRr-esByr4-rhF7d6_AdK2FyS_MjFo&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
> > * Hostname was NOT found in DNS cache
> > * Trying 127.0.0.1...
> > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > GET /ignite HTTP/1.1
> > > User-Agent: curl/7.35.0
> > > Host: 127.0.0.1:8080
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > > Accept: */*
> > >
> > < HTTP/1.1 400 Bad Request
> > < Date: Thu, 20 Feb 2020 11:21:22 GMT
> > < Content-Type: application/json;charset=utf-8
> > < Content-Length: 0
> > * Server Jetty(9.4.11.v20180605) is not blacklisted
> > < Server: Jetty(9.4.11.v20180605)
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> >
> > ----------
> >
> > I expect the (above) request above to fail because I don't supply the
> > required
> > `cmd` parameter.
> >
> > However, if I make a request that I believe should succeed, I never
> > receive a
> > response:
> >
> > ----------
> >
> > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=b1MZBDEsOiVfh5l7ltu2DgXaYnh_5XtnKPg1RgsDfTI&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
> > * Hostname was NOT found in DNS cache
> > * Trying 127.0.0.1...
> > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > GET /ignite?cmd=version HTTP/1.1
> > > User-Agent: curl/7.35.0
> > > Host: 127.0.0.1:8080
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=VEZ1YiACosFkygqRYUfQg4SBv12OI3TTYzpQDOhT8bE&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > > Accept: */*
> > >
> >
> > ----------
> >
> > `curl` will wait forever for a response that never arrives.
> >
> > I tried looking for logs output by the REST API so that I could
> debug the
> > issue, but failed to find any log entries emitted from the REST API.
> >
> > Given that I'm new to Apache Ignite, I suspect that I'm almost
> certainly
> > doing
> > something wrong, but I have absolutely no idea what that could be.
> >
> > I have a few questions:
> >
> > 1.) Has anyone encountered an issue similar to this with the REST API?
> > If so,
> > how did you solve it?
> > 2.) Can anyone tell me where I might find REST API log entries?
> > 3.) I have persistence enabled in the XML configuration. Would that
> conflict
> > with the REST API?
> > 4.) I have additional nodes (3 nodes in total) in the XML
> configuration.
> > AFAICT from log entries and some liberal use of `ss`, the nodes are
> > communicating with each other. Would having multiple nodes
> conflict with
> > the
> > REST API?
> > 5.) Is there anything else I'm missing that might not be obvious to me
> > because
> > I'm new to Apache Ignite?
> >
> > Other than (3) and (4), I think the configuration file I'm using is
> > rather bland:
> >
> > ----------
> >
> > <beans xmlns="http://www.springframework.org/schema/beans
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=UyopUd5Ot09OGVA6KNi_oTS8JCxjI42wdeGJQbrC2UQ&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
> > xsi:schemaLocation="
> > http://www.springframework.org/schema/beans
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=QS1N4uN0i6B10p6B-YPORLbXGqiH4ve3DC9FLR-9HH4&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
> > http://www.springframework.org/schema/beans/spring-beans.xsd
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=nxds_HJL4Wp8JgHdOCFCwfFGfocNDpGEKHbOigV9HgY&s=1W2X_AlApjVwM2r9Q9I8hQniVWhjpGvzeGWSbA7z0Oc&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
> > <bean class="org.apache.ignite.configuration.IgniteConfiguration">
> >
> > <property name="dataStorageConfiguration">
> > <bean
> > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > <!--
> > Default memory region that grows endlessly. A cache is bound
> > to this
> > memory region unless it sets another one in its
> > CacheConfiguration/
> > -->
> > <property name="defaultDataRegionConfiguration">
> > <bean
> > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > <property name="persistenceEnabled" value="true"/>
> > <property name="name" value="Default_Region"/>
> > <!-- 100 MB memory region with disabled eviction -->
> > <property name="initialSize" value="#{100 * 1024 *
> 1024}"/>
> > </bean>
> > </property>
> > </bean>
> > </property>
> >
> > <property name="discoverySpi">
> > <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > <property name="ipFinder">
> > <bean
> >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > <property name="addresses">
> > <list>
> > <value>[other node address]</value><value>[other node
> > address]</value>
> > </list>
> > </property>
> > </bean>
> > </property>
> > </bean>
> > </property>
> >
> > </bean>
> > </beans>
> >
> > ----------
> >
> > (Obviously, each instance of '[other node address]' is substituted
> with a
> > real
> > IP address in the actual files I'm using.)
> >
> > Thanks in advance for any help.
> >
> > --
> > Devin
> >
>
Re: REST API requests hang with no response
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Your node has never finished joining to cluster nor was able to
self-discover to form a cluster of its own, as evident by:
"main" #1 prio=5 os_prio=0 tid=0x00007f7e8000d000 nid=0x5fed waiting on
condition [0x00007f7e8875f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:7778)
at org.apache.ignite.spi.discovery.tcp.*ServerImpl.sendJoinRequestMessage*
(ServerImpl.java:1131)
at org.apache.ignite.spi.discovery.tcp.*ServerImpl.joinTopology*
(ServerImpl.java:910)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
- locked <0x00000005e8cca5b8> (a
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Until node is in topology, obviously it can't serve any requests.
Regards,
--
Ilya Kasnacheev
пт, 21 февр. 2020 г. в 01:55, Devin Anderson <da...@akamai.com>:
> Hi Ilya,
>
> I'm attaching the `jstack` dump to this message. When I took the dump,
> there
> were ten requests that had been made to the Apache Ignite REST API using
> POST
> data of the form:
>
> cmd=add&key=[key]&value=[value]
>
> ... where [key] and [value] were proper URI encoded data.
>
> Thanks in advance for any help. I'll take a look as well and try to
> figure out
> what's going on.
>
> Devin
>
> On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> > Hello!
> >
> > Please collect thread dump (jstack) from affected node, share it with us.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 20 февр. 2020 г. в 16:17, Devin Anderson <danderso@akamai.com
> > <ma...@akamai.com>>:
> >
> > Hi all,
> >
> > I'm seeing issues wherein the Apache Ignite REST API appears to
> accept
> > requests, but doesn't ever reply. This doesn't always happen; for
> > example, if
> > I make a request that I expect the API to reject, I get back a
> response:
> >
> > ----------
> >
> > # curl -v -X GET 'http://127.0.0.1:8080/ignite
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=
> >'
> > * Hostname was NOT found in DNS cache
> > * Trying 127.0.0.1...
> > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > GET /ignite HTTP/1.1
> > > User-Agent: curl/7.35.0
> > > Host: 127.0.0.1:8080
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=
> >
> > > Accept: */*
> > >
> > < HTTP/1.1 400 Bad Request
> > < Date: Thu, 20 Feb 2020 11:21:22 GMT
> > < Content-Type: application/json;charset=utf-8
> > < Content-Length: 0
> > * Server Jetty(9.4.11.v20180605) is not blacklisted
> > < Server: Jetty(9.4.11.v20180605)
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> >
> > ----------
> >
> > I expect the (above) request above to fail because I don't supply the
> > required
> > `cmd` parameter.
> >
> > However, if I make a request that I believe should succeed, I never
> > receive a
> > response:
> >
> > ----------
> >
> > # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=
> >'
> > * Hostname was NOT found in DNS cache
> > * Trying 127.0.0.1...
> > * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > > GET /ignite?cmd=version HTTP/1.1
> > > User-Agent: curl/7.35.0
> > > Host: 127.0.0.1:8080
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=
> >
> > > Accept: */*
> > >
> >
> > ----------
> >
> > `curl` will wait forever for a response that never arrives.
> >
> > I tried looking for logs output by the REST API so that I could
> debug the
> > issue, but failed to find any log entries emitted from the REST API.
> >
> > Given that I'm new to Apache Ignite, I suspect that I'm almost
> certainly
> > doing
> > something wrong, but I have absolutely no idea what that could be.
> >
> > I have a few questions:
> >
> > 1.) Has anyone encountered an issue similar to this with the REST
> API?
> > If so,
> > how did you solve it?
> > 2.) Can anyone tell me where I might find REST API log entries?
> > 3.) I have persistence enabled in the XML configuration. Would that
> conflict
> > with the REST API?
> > 4.) I have additional nodes (3 nodes in total) in the XML
> configuration.
> > AFAICT from log entries and some liberal use of `ss`, the nodes are
> > communicating with each other. Would having multiple nodes conflict
> with
> > the
> > REST API?
> > 5.) Is there anything else I'm missing that might not be obvious to
> me
> > because
> > I'm new to Apache Ignite?
> >
> > Other than (3) and (4), I think the configuration file I'm using is
> > rather bland:
> >
> > ----------
> >
> > <beans xmlns="http://www.springframework.org/schema/beans
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=
> >"
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=
> >"
> > xsi:schemaLocation="
> > http://www.springframework.org/schema/beans
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=
> >
> > http://www.springframework.org/schema/beans/spring-beans.xsd
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=
> >">
> > <bean class="org.apache.ignite.configuration.IgniteConfiguration">
> >
> > <property name="dataStorageConfiguration">
> > <bean
> > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > <!--
> > Default memory region that grows endlessly. A cache is
> bound
> > to this
> > memory region unless it sets another one in its
> > CacheConfiguration/
> > -->
> > <property name="defaultDataRegionConfiguration">
> > <bean
> > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > <property name="persistenceEnabled" value="true"/>
> > <property name="name" value="Default_Region"/>
> > <!-- 100 MB memory region with disabled eviction -->
> > <property name="initialSize" value="#{100 * 1024 *
> 1024}"/>
> > </bean>
> > </property>
> > </bean>
> > </property>
> >
> > <property name="discoverySpi">
> > <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > <property name="ipFinder">
> > <bean
> >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > <property name="addresses">
> > <list>
> > <value>[other node address]</value><value>[other
> node
> > address]</value>
> > </list>
> > </property>
> > </bean>
> > </property>
> > </bean>
> > </property>
> >
> > </bean>
> > </beans>
> >
> > ----------
> >
> > (Obviously, each instance of '[other node address]' is substituted
> with a
> > real
> > IP address in the actual files I'm using.)
> >
> > Thanks in advance for any help.
> >
> > --
> > Devin
> >
>
>
Re: REST API requests hang with no response
Posted by Devin Anderson <da...@akamai.com>.
Hi Ilya,
I'm attaching the `jstack` dump to this message. When I took the dump, there
were ten requests that had been made to the Apache Ignite REST API using POST
data of the form:
cmd=add&key=[key]&value=[value]
... where [key] and [value] were proper URI encoded data.
Thanks in advance for any help. I'll take a look as well and try to figure out
what's going on.
Devin
On 2/20/20 5:43 AM, Ilya Kasnacheev wrote:
> Hello!
>
> Please collect thread dump (jstack) from affected node, share it with us.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 20 февр. 2020 г. в 16:17, Devin Anderson <danderso@akamai.com
> <ma...@akamai.com>>:
>
> Hi all,
>
> I'm seeing issues wherein the Apache Ignite REST API appears to accept
> requests, but doesn't ever reply. This doesn't always happen; for
> example, if
> I make a request that I expect the API to reject, I get back a response:
>
> ----------
>
> # curl -v -X GET 'http://127.0.0.1:8080/ignite
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Q3T6HlQSi3TnKwaRcPsb0MWMRbFNexaNfrucVQSrTkI&e=>'
> * Hostname was NOT found in DNS cache
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > GET /ignite HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: 127.0.0.1:8080
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > Accept: */*
> >
> < HTTP/1.1 400 Bad Request
> < Date: Thu, 20 Feb 2020 11:21:22 GMT
> < Content-Type: application/json;charset=utf-8
> < Content-Length: 0
> * Server Jetty(9.4.11.v20180605) is not blacklisted
> < Server: Jetty(9.4.11.v20180605)
> <
> * Connection #0 to host 127.0.0.1 left intact
>
> ----------
>
> I expect the (above) request above to fail because I don't supply the
> required
> `cmd` parameter.
>
> However, if I make a request that I believe should succeed, I never
> receive a
> response:
>
> ----------
>
> # curl -v 'http://127.0.0.1:8080/ignite?cmd=version
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080_ignite-3Fcmd-3Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=USDfyc0TCRCJRx4vzXe9ezgsCMJ8YfFh2IK983DMgrc&e=>'
> * Hostname was NOT found in DNS cache
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > GET /ignite?cmd=version HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: 127.0.0.1:8080
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A8080&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=gS2lm03jyjMVp9ZTWC259JI_RG9y4l9QsuvxX4Cz85c&e=>
> > Accept: */*
> >
>
> ----------
>
> `curl` will wait forever for a response that never arrives.
>
> I tried looking for logs output by the REST API so that I could debug the
> issue, but failed to find any log entries emitted from the REST API.
>
> Given that I'm new to Apache Ignite, I suspect that I'm almost certainly
> doing
> something wrong, but I have absolutely no idea what that could be.
>
> I have a few questions:
>
> 1.) Has anyone encountered an issue similar to this with the REST API?
> If so,
> how did you solve it?
> 2.) Can anyone tell me where I might find REST API log entries?
> 3.) I have persistence enabled in the XML configuration. Would that conflict
> with the REST API?
> 4.) I have additional nodes (3 nodes in total) in the XML configuration.
> AFAICT from log entries and some liberal use of `ss`, the nodes are
> communicating with each other. Would having multiple nodes conflict with
> the
> REST API?
> 5.) Is there anything else I'm missing that might not be obvious to me
> because
> I'm new to Apache Ignite?
>
> Other than (3) and (4), I think the configuration file I'm using is
> rather bland:
>
> ----------
>
> <beans xmlns="http://www.springframework.org/schema/beans
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=vdA3V10EDd-kLU4fjTYawM1o1NL-UvHO1iiW4VxHUyw&e=>"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=uORaM4cyX6R3dKSjnm0_K8A4230IGy7LPWUzRAG_IV4&e=>
> http://www.springframework.org/schema/beans/spring-beans.xsd
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.springframework.org_schema_beans_spring-2Dbeans.xsd&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=O-a0zsxmvZZBVCsKS7owS9oPa3AKjd7Xpau8laEriDM&m=0D1iM7MuHAG3uQxX5V8Iv_9kNyO25FNACjcqQnHaaLs&s=Reg5uEVfbW-0vucPqzp0oUT2MSnUoevTyriLXqXkAkY&e=>">
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
>
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <!--
> Default memory region that grows endlessly. A cache is bound
> to this
> memory region unless it sets another one in its
> CacheConfiguration/
> -->
> <property name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="persistenceEnabled" value="true"/>
> <property name="name" value="Default_Region"/>
> <!-- 100 MB memory region with disabled eviction -->
> <property name="initialSize" value="#{100 * 1024 * 1024}"/>
> </bean>
> </property>
> </bean>
> </property>
>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>[other node address]</value><value>[other node
> address]</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
>
> </bean>
> </beans>
>
> ----------
>
> (Obviously, each instance of '[other node address]' is substituted with a
> real
> IP address in the actual files I'm using.)
>
> Thanks in advance for any help.
>
> --
> Devin
>
Re: REST API requests hang with no response
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Please collect thread dump (jstack) from affected node, share it with us.
Regards,
--
Ilya Kasnacheev
чт, 20 февр. 2020 г. в 16:17, Devin Anderson <da...@akamai.com>:
> Hi all,
>
> I'm seeing issues wherein the Apache Ignite REST API appears to accept
> requests, but doesn't ever reply. This doesn't always happen; for
> example, if
> I make a request that I expect the API to reject, I get back a response:
>
> ----------
>
> # curl -v -X GET 'http://127.0.0.1:8080/ignite'
> * Hostname was NOT found in DNS cache
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > GET /ignite HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: 127.0.0.1:8080
> > Accept: */*
> >
> < HTTP/1.1 400 Bad Request
> < Date: Thu, 20 Feb 2020 11:21:22 GMT
> < Content-Type: application/json;charset=utf-8
> < Content-Length: 0
> * Server Jetty(9.4.11.v20180605) is not blacklisted
> < Server: Jetty(9.4.11.v20180605)
> <
> * Connection #0 to host 127.0.0.1 left intact
>
> ----------
>
> I expect the (above) request above to fail because I don't supply the
> required
> `cmd` parameter.
>
> However, if I make a request that I believe should succeed, I never
> receive a
> response:
>
> ----------
>
> # curl -v 'http://127.0.0.1:8080/ignite?cmd=version'
> * Hostname was NOT found in DNS cache
> * Trying 127.0.0.1...
> * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> > GET /ignite?cmd=version HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: 127.0.0.1:8080
> > Accept: */*
> >
>
> ----------
>
> `curl` will wait forever for a response that never arrives.
>
> I tried looking for logs output by the REST API so that I could debug the
> issue, but failed to find any log entries emitted from the REST API.
>
> Given that I'm new to Apache Ignite, I suspect that I'm almost certainly
> doing
> something wrong, but I have absolutely no idea what that could be.
>
> I have a few questions:
>
> 1.) Has anyone encountered an issue similar to this with the REST API? If
> so,
> how did you solve it?
> 2.) Can anyone tell me where I might find REST API log entries?
> 3.) I have persistence enabled in the XML configuration. Would that
> conflict
> with the REST API?
> 4.) I have additional nodes (3 nodes in total) in the XML configuration.
> AFAICT from log entries and some liberal use of `ss`, the nodes are
> communicating with each other. Would having multiple nodes conflict with
> the
> REST API?
> 5.) Is there anything else I'm missing that might not be obvious to me
> because
> I'm new to Apache Ignite?
>
> Other than (3) and (4), I think the configuration file I'm using is rather
> bland:
>
> ----------
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd">
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
>
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <!--
> Default memory region that grows endlessly. A cache is bound to
> this
> memory region unless it sets another one in its
> CacheConfiguration/
> -->
> <property name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="persistenceEnabled" value="true"/>
> <property name="name" value="Default_Region"/>
> <!-- 100 MB memory region with disabled eviction -->
> <property name="initialSize" value="#{100 * 1024 * 1024}"/>
> </bean>
> </property>
> </bean>
> </property>
>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>[other node address]</value><value>[other node
> address]</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
>
> </bean>
> </beans>
>
> ----------
>
> (Obviously, each instance of '[other node address]' is substituted with a
> real
> IP address in the actual files I'm using.)
>
> Thanks in advance for any help.
>
> --
> Devin
>
>