Fwd: ENAMETOOLONG on mkdir() leading to failed startup with exception org.apache.ignite.IgniteCheckedException: Failed to create page store work directory: ...

[Arthur Naseef <ar...@apache.org>]

Forwarding this one.  Looks like there is a new problem introduced to the
code base since 2.14 here.

We are running into it again while working on the current development
branch.

Art

---------- Forwarded message ---------
From: Arthur Naseef <ar...@apache.org>
Date: Wed, Jul 6, 2022 at 5:14 PM
Subject: ENAMETOOLONG on mkdir() leading to failed startup with exception
org.apache.ignite.IgniteCheckedException: Failed to create page store work
directory: ...
To: <us...@ignite.apache.org>


I started using the latest master brannch in local testing of Apache Ignite
and ran into the following exception (note IP addresses and path elements
replaced redacted with asterisks to prevent possible security concerns):

Caused by: org.apache.ignite.IgniteCheckedException: Failed to create page
store work directory:
/home/art/****/******/*******/*******/POC/ignite/distributed-scheduling-poc/ignite/work/db/*_*_*_*_*_*_*_*_**_***_*_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_***_*_***_***_***_***_*_***_***_**_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_**_*_***_***_**_*_****_****_***_****_*_*_*_*_**_************_*****


Using strace, found the following underlying cause:

mkdir("/home/art/****/******/*******/*******/POC/ignite/distributed-scheduling-poc/ignite/work/db/*_*_*_*_*_*_*_*_**_***_*_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_***_*_***_***_***_***_*_***_***_**_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_**_*_***_***_**_*_****_****_***_****_*_*_*_*_**_************_*****",
0777) = -1 ENAMETOOLONG (File name too long)


Debugging through the code, I *THINK* the following change may be involved:

ticket https://issues.apache.org/jira/browse/IGNITE-16938
commit
https://github.com/apache/ignite/commit/cdf8200fa02da8e88b6ca5a9fd9e89bb28f95c62


It appears the path name includes all of the IP address of all the
interfaces on the system, and since this system is running docker and
kubernetes, the number of those interfaces is ... large.

I'll go back to a released version for my DEV purposes for now.

Feedback is welcome.

Art

Re: Fwd: ENAMETOOLONG on mkdir() leading to failed startup with exception org.apache.ignite.IgniteCheckedException: Failed to create page store work directory: ...

[Łukasz Dywicki <lu...@code-house.org>]

I can confirm this issue happening also in my local development 
environment. I had ~30 network interfaces (don't ask why) which caused 
fairly long filenames generated for persistent store path. I figured it 
out while debugging startup issue of tests. Arthur pointed out his 
earlier mail, but I am not entirely sure if IGNITE-16938 is related.

Thinking of consistent node identifier and its storage path, are these 
IP addresses in generated path are relevant for third party tooling? If 
not, why not getting host addresses, sorting them and then calculating a 
hash? It would be consistent, short and meaningless.

Best,
Łukasz

On 27.10.2022 19:12, Arthur Naseef wrote:
> Forwarding this one.  Looks like there is a new problem introduced to 
> the code base since 2.14 here.
> 
> We are running into it again while working on the current development 
> branch.
> 
> Art
> 
> ---------- Forwarded message ---------
> From: *Arthur Naseef* <artnaseef@apache.org <ma...@apache.org>>
> Date: Wed, Jul 6, 2022 at 5:14 PM
> Subject: ENAMETOOLONG on mkdir() leading to failed startup with 
> exception org.apache.ignite.IgniteCheckedException: Failed to create 
> page store work directory: ...
> To: <user@ignite.apache.org <ma...@ignite.apache.org>>
> 
> 
> I started using the latest master brannch in local testing of Apache 
> Ignite and ran into the following exception (note IP addresses and path 
> elements replaced redacted with asterisks to prevent possible security 
> concerns):
> 
>     Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>     create page store work directory:
>     /home/art/****/******/*******/*******/POC/ignite/distributed-scheduling-poc/ignite/work/db/*_*_*_*_*_*_*_*_**_***_*_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_***_*_***_***_***_***_*_***_***_**_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_**_*_***_***_**_*_****_****_***_****_*_*_*_*_**_************_*****
> 
> 
> Using strace, found the following underlying cause:
> 
>     mkdir("/home/art/****/******/*******/*******/POC/ignite/distributed-scheduling-poc/ignite/work/db/*_*_*_*_*_*_*_*_**_***_*_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_**_*_*_***_***_*_***_***_***_***_*_***_***_**_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_***_*_***_***_**_*_***_***_**_*_****_****_***_****_*_*_*_*_**_************_*****", 0777) = -1 ENAMETOOLONG (File name too long)
> 
> 
> Debugging through the code, I *THINK* the following change may be involved:
> 
>     ticket https://issues.apache.org/jira/browse/IGNITE-16938
>     <https://issues.apache.org/jira/browse/IGNITE-16938>
>     commit
>     https://github.com/apache/ignite/commit/cdf8200fa02da8e88b6ca5a9fd9e89bb28f95c62 <https://github.com/apache/ignite/commit/cdf8200fa02da8e88b6ca5a9fd9e89bb28f95c62>
> 
> 
> It appears the path name includes all of the IP address of all the 
> interfaces on the system, and since this system is running docker and 
> kubernetes, the number of those interfaces is ... large.
> 
> I'll go back to a released version for my DEV purposes for now.
> 
> Feedback is welcome.
> 
> Art
>