You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@harmony.apache.org by "Tim Ellison (JIRA)" <ji...@apache.org> on 2009/06/18 17:28:08 UTC

[jira] Created: (HARMONY-6238) [classlib][luni] File path limit set to 1K chars for all file systems

[classlib][luni] File path limit set to 1K chars for all file systems
---------------------------------------------------------------------

                 Key: HARMONY-6238
                 URL: https://issues.apache.org/jira/browse/HARMONY-6238
             Project: Harmony
          Issue Type: Sub-task
          Components: Classlib
    Affects Versions: 5.0M10
            Reporter: Tim Ellison


In Harmony's portlib  (modules/portlib/src/main/native/include/shared/hyport.h)  we set the maximum path length to 1024

/** HyMaxPath was chosen from unix MAXPATHLEN.  Override in platform
  * specific hyfile implementations if needed.
  */
#define HyMaxPath   1024

This causes some long file path operations to fail with an IOException on Harmony which do not fail on the RI.

Harmony should allow for file systems that handle longer paths.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Re: [jira] Created: (HARMONY-6238) [classlib][luni] File path limit set to 1K chars for all file systems

Posted by Oliver Deakin <ol...@googlemail.com>.
Tim Ellison wrote:
> Regis wrote:
>   
>> Tim Ellison wrote:
>>     
>>> Oliver Deakin wrote:
>>>       
>>>> I've had a quick look around for the "right" way to do this, and it
>>>> seems there are 2 options:
>>>> 1) Determine the max path length from a system macro, probably PATH_MAX
>>>> in limits.h.
>>>> 2) Use the unix system call pathconf() to get the max path length.
>>>>
>>>> I think using a build time constant, as in 1, is preferable, however
>>>> this macro is not defined on all platforms. On zOS, for example, only
>>>> _POSIX_PATH_MAX is defined, and that is set to a paltry 256 (which is
>>>> not the real max path length - pathconf() tells me it is 1024).
>>>>
>>>> Perhaps to start with we can include /usr/include/linux/limits.h and use
>>>> PATH_MAX for linux platforms (I'm assuming that file exists for linux
>>>> distros in general here - perhaps someone can clarify this?) and on
>>>> non-linux platforms fall back to a default of 1024. We can add other
>>>> platforms in as we find the correct includes/macro definitions for them.
>>>>         
>>>> From the Harmony code I was looking at, the max path length is used to
>>>>         
>>> allocate char buffers so we can do path manipulations before passing
>>> through to OS calls.  We fail early if the path is greater than the
>>> expected max path length.
>>>
>>> Would could, as you say try to set the max path length based on the OS
>>> we are using, but even that is going to be a guess since who knows where
>>> the file path points (e.g. network drives etc.)?
>>>
>>> I wonder if we can be a bit lazier and set the Harmony max path length
>>> to a large number for the path manipulations, then let the OS call fail
>>> if the path is too long?
>>>       
>> agree
>>
>>     
>>> (caveat: I've not looked at all uses of the HyMaxPath const to see if
>>> that is a good idea or not)
>>>
>>> Regards,
>>> Tim
>>>
>>>       
>> I guess we have max path length because we used char array as buffer.
>> It's not efficient and a kind of waste memory if the buffer size was the
>> same as OS limit (on Windows it's 32,767 characters).
>>
>> So I wonder if we can set a threshold value, like 1024, if path length
>> is smaller than the value, we allocate char array as buffer, just like
>> we have done; otherwise dynamic malloc memory as buffer, and then let OS
>> tell us if the path is too long?
>>     
>
> Sounds good to me.
>
>   

And to me - I didn't realise it was being used to allocate a buffer. 
This sounds like a good solution to me.

Regards,
Oliver

> Tim
>
>
>   

-- 
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: [jira] Created: (HARMONY-6238) [classlib][luni] File path limit set to 1K chars for all file systems

Posted by Tim Ellison <t....@gmail.com>.
Regis wrote:
> Tim Ellison wrote:
>> Oliver Deakin wrote:
>>> I've had a quick look around for the "right" way to do this, and it
>>> seems there are 2 options:
>>> 1) Determine the max path length from a system macro, probably PATH_MAX
>>> in limits.h.
>>> 2) Use the unix system call pathconf() to get the max path length.
>>>
>>> I think using a build time constant, as in 1, is preferable, however
>>> this macro is not defined on all platforms. On zOS, for example, only
>>> _POSIX_PATH_MAX is defined, and that is set to a paltry 256 (which is
>>> not the real max path length - pathconf() tells me it is 1024).
>>>
>>> Perhaps to start with we can include /usr/include/linux/limits.h and use
>>> PATH_MAX for linux platforms (I'm assuming that file exists for linux
>>> distros in general here - perhaps someone can clarify this?) and on
>>> non-linux platforms fall back to a default of 1024. We can add other
>>> platforms in as we find the correct includes/macro definitions for them.
>>
>>> From the Harmony code I was looking at, the max path length is used to
>> allocate char buffers so we can do path manipulations before passing
>> through to OS calls.  We fail early if the path is greater than the
>> expected max path length.
>>
>> Would could, as you say try to set the max path length based on the OS
>> we are using, but even that is going to be a guess since who knows where
>> the file path points (e.g. network drives etc.)?
>>
>> I wonder if we can be a bit lazier and set the Harmony max path length
>> to a large number for the path manipulations, then let the OS call fail
>> if the path is too long?
> 
> agree
> 
>>
>> (caveat: I've not looked at all uses of the HyMaxPath const to see if
>> that is a good idea or not)
>>
>> Regards,
>> Tim
>>
> 
> I guess we have max path length because we used char array as buffer.
> It's not efficient and a kind of waste memory if the buffer size was the
> same as OS limit (on Windows it's 32,767 characters).
> 
> So I wonder if we can set a threshold value, like 1024, if path length
> is smaller than the value, we allocate char array as buffer, just like
> we have done; otherwise dynamic malloc memory as buffer, and then let OS
> tell us if the path is too long?

Sounds good to me.

Tim


Re: [jira] Created: (HARMONY-6238) [classlib][luni] File path limit set to 1K chars for all file systems

Posted by Regis <xu...@gmail.com>.
Tim Ellison wrote:
> Oliver Deakin wrote:
>> I've had a quick look around for the "right" way to do this, and it
>> seems there are 2 options:
>> 1) Determine the max path length from a system macro, probably PATH_MAX
>> in limits.h.
>> 2) Use the unix system call pathconf() to get the max path length.
>>
>> I think using a build time constant, as in 1, is preferable, however
>> this macro is not defined on all platforms. On zOS, for example, only
>> _POSIX_PATH_MAX is defined, and that is set to a paltry 256 (which is
>> not the real max path length - pathconf() tells me it is 1024).
>>
>> Perhaps to start with we can include /usr/include/linux/limits.h and use
>> PATH_MAX for linux platforms (I'm assuming that file exists for linux
>> distros in general here - perhaps someone can clarify this?) and on
>> non-linux platforms fall back to a default of 1024. We can add other
>> platforms in as we find the correct includes/macro definitions for them.
> 
>>>From the Harmony code I was looking at, the max path length is used to
> allocate char buffers so we can do path manipulations before passing
> through to OS calls.  We fail early if the path is greater than the
> expected max path length.
> 
> Would could, as you say try to set the max path length based on the OS
> we are using, but even that is going to be a guess since who knows where
> the file path points (e.g. network drives etc.)?
> 
> I wonder if we can be a bit lazier and set the Harmony max path length
> to a large number for the path manipulations, then let the OS call fail
> if the path is too long?

agree

> 
> (caveat: I've not looked at all uses of the HyMaxPath const to see if
> that is a good idea or not)
> 
> Regards,
> Tim
> 

I guess we have max path length because we used char array as buffer. It's not 
efficient and a kind of waste memory if the buffer size was the same as OS limit 
(on Windows it's 32,767 characters).

So I wonder if we can set a threshold value, like 1024, if path length is 
smaller than the value, we allocate char array as buffer, just like we have 
done; otherwise dynamic malloc memory as buffer, and then let OS tell us if the 
path is too long?

-- 
Best Regards,
Regis.

Re: [jira] Created: (HARMONY-6238) [classlib][luni] File path limit set to 1K chars for all file systems

Posted by Tim Ellison <t....@gmail.com>.
Oliver Deakin wrote:
> I've had a quick look around for the "right" way to do this, and it
> seems there are 2 options:
> 1) Determine the max path length from a system macro, probably PATH_MAX
> in limits.h.
> 2) Use the unix system call pathconf() to get the max path length.
> 
> I think using a build time constant, as in 1, is preferable, however
> this macro is not defined on all platforms. On zOS, for example, only
> _POSIX_PATH_MAX is defined, and that is set to a paltry 256 (which is
> not the real max path length - pathconf() tells me it is 1024).
> 
> Perhaps to start with we can include /usr/include/linux/limits.h and use
> PATH_MAX for linux platforms (I'm assuming that file exists for linux
> distros in general here - perhaps someone can clarify this?) and on
> non-linux platforms fall back to a default of 1024. We can add other
> platforms in as we find the correct includes/macro definitions for them.

>From the Harmony code I was looking at, the max path length is used to
allocate char buffers so we can do path manipulations before passing
through to OS calls.  We fail early if the path is greater than the
expected max path length.

Would could, as you say try to set the max path length based on the OS
we are using, but even that is going to be a guess since who knows where
the file path points (e.g. network drives etc.)?

I wonder if we can be a bit lazier and set the Harmony max path length
to a large number for the path manipulations, then let the OS call fail
if the path is too long?

(caveat: I've not looked at all uses of the HyMaxPath const to see if
that is a good idea or not)

Regards,
Tim

Re: [jira] Created: (HARMONY-6238) [classlib][luni] File path limit set to 1K chars for all file systems

Posted by Oliver Deakin <ol...@googlemail.com>.
Hi Tim,

I've had a quick look around for the "right" way to do this, and it 
seems there are 2 options:
1) Determine the max path length from a system macro, probably PATH_MAX 
in limits.h.
2) Use the unix system call pathconf() to get the max path length.

I think using a build time constant, as in 1, is preferable, however 
this macro is not defined on all platforms. On zOS, for example, only 
_POSIX_PATH_MAX is defined, and that is set to a paltry 256 (which is 
not the real max path length - pathconf() tells me it is 1024).

Perhaps to start with we can include /usr/include/linux/limits.h and use 
PATH_MAX for linux platforms (I'm assuming that file exists for linux 
distros in general here - perhaps someone can clarify this?) and on 
non-linux platforms fall back to a default of 1024. We can add other 
platforms in as we find the correct includes/macro definitions for them.

Regards,
Oliver


Tim Ellison (JIRA) wrote:
> [classlib][luni] File path limit set to 1K chars for all file systems
> ---------------------------------------------------------------------
>
>                  Key: HARMONY-6238
>                  URL: https://issues.apache.org/jira/browse/HARMONY-6238
>              Project: Harmony
>           Issue Type: Sub-task
>           Components: Classlib
>     Affects Versions: 5.0M10
>             Reporter: Tim Ellison
>
>
> In Harmony's portlib  (modules/portlib/src/main/native/include/shared/hyport.h)  we set the maximum path length to 1024
>
> /** HyMaxPath was chosen from unix MAXPATHLEN.  Override in platform
>   * specific hyfile implementations if needed.
>   */
> #define HyMaxPath   1024
>
> This causes some long file path operations to fail with an IOException on Harmony which do not fail on the RI.
>
> Harmony should allow for file systems that handle longer paths.
>
>
>   

-- 
Oliver Deakin
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU