You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Roger Whitcomb <Ro...@actian.com> on 2015/11/16 23:11:50 UTC
[VFS] Further changes to HDFS Provider for alternate configuration
support
Hi all,
In trying to solve some customer issues we're having, mainly to do with trying to browse HDFS files when the Name Node is configured for High-Availability, I found I needed to make some more changes/additions to the VFS HDFS Provider. I have attached a diff/patch file. But, a couple of questions:
* This is a follow-on (basically) to my earlier patch for VFS-555, should I open a new JIRA, or just reopen the existing one?
* Since this actually changes an API (but which is not released yet), is that an acceptable thing to do?
* The new properties I have added to HdfsFileSystemConfigBuilder are not actually symmetrical, so they don't fit the definition of a Java "property", that is the setters are "setXXX" while the getters are "getXXXs" (plural). The logic behind this is that the setters are called by user code, and each can be called multiple times. The getters are only (meant to be) called by the HdfsFileSystem object (that is internally to the HdfsProvider), and thus it is a lot easier to get all the settings at once than to have to make multiple calls, with potentially iterators, etc.
* Is this too big a change to fit in last-minute before the release of VFS 2.1?
Patch is attached.
Thanks,
~Roger Whitcomb
RE: [VFS] Further changes to HDFS Provider for alternate
configuration support
Posted by Roger Whitcomb <Ro...@actian.com>.
I have opened issue https://issues.apache.org/jira/browse/VFS-586 with my proposed patch attached, and these questions included in the issue. Updated the patch with some things I noticed after the initial post also.
Thanks,
~Roger
-----Original Message-----
From: dlmarion@comcast.net [mailto:dlmarion@comcast.net]
Sent: Monday, November 16, 2015 3:11 PM
To: 'Commons Developers List' <de...@commons.apache.org>
Subject: RE: [VFS] Further changes to HDFS Provider for alternate configuration support
I don't see the patch. It might be stripped off by the mail server.
From: Roger Whitcomb [mailto:Roger.Whitcomb@actian.com]
Sent: Monday, November 16, 2015 5:12 PM
To: Commons Developers List
Subject: [VFS] Further changes to HDFS Provider for alternate configuration support
Hi all,
In trying to solve some customer issues we're having, mainly to do with trying to browse HDFS files when the Name Node is configured for High-Availability, I found I needed to make some more changes/additions to the VFS HDFS Provider. I have attached a diff/patch file. But, a couple of
questions:
. This is a follow-on (basically) to my earlier patch for VFS-555,
should I open a new JIRA, or just reopen the existing one?
. Since this actually changes an API (but which is not released
yet), is that an acceptable thing to do?
. The new properties I have added to HdfsFileSystemConfigBuilder are
not actually symmetrical, so they don't fit the definition of a Java "property", that is the setters are "setXXX" while the getters are "getXXXs"
(plural). The logic behind this is that the setters are called by user code, and each can be called multiple times. The getters are only (meant to be) called by the HdfsFileSystem object (that is internally to the HdfsProvider), and thus it is a lot easier to get all the settings at once than to have to make multiple calls, with potentially iterators, etc.
. Is this too big a change to fit in last-minute before the release
of VFS 2.1?
Patch is attached.
Thanks,
~Roger Whitcomb
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [VFS] Further changes to HDFS Provider for alternate
configuration support
Posted by Roger Whitcomb <Ro...@actian.com>.
I have created https://issues.apache.org/jira/browse/VFS-586 and attached the proposed patch there for your consideration.
The original questions I had still apply, and will be added to the issue.
Thanks,
~Roger Whitcomb
-----Original Message-----
From: Pascal Schumacher [mailto:pascalschumacher@gmx.net]
Sent: Tuesday, November 17, 2015 9:48 AM
To: Commons Developers List <de...@commons.apache.org>
Subject: Re: [VFS] Further changes to HDFS Provider for alternate configuration support
Yes, the mailing list server does not allow attachments. Please attach the patch to a JIRA issue.
Thanks!
Am 17.11.2015 um 00:11 schrieb dlmarion@comcast.net:
> I don't see the patch. It might be stripped off by the mail server.
>
>
>
> From: Roger Whitcomb [mailto:Roger.Whitcomb@actian.com]
> Sent: Monday, November 16, 2015 5:12 PM
> To: Commons Developers List
> Subject: [VFS] Further changes to HDFS Provider for alternate
> configuration support
>
>
>
> Hi all,
>
> In trying to solve some customer issues we're having,
> mainly to do with trying to browse HDFS files when the Name Node is
> configured for High-Availability, I found I needed to make some more
> changes/additions to the VFS HDFS Provider. I have attached a
> diff/patch file. But, a couple of
> questions:
>
> . This is a follow-on (basically) to my earlier patch for VFS-555,
> should I open a new JIRA, or just reopen the existing one?
>
> . Since this actually changes an API (but which is not released
> yet), is that an acceptable thing to do?
>
> . The new properties I have added to HdfsFileSystemConfigBuilder are
> not actually symmetrical, so they don't fit the definition of a Java
> "property", that is the setters are "setXXX" while the getters are "getXXXs"
> (plural). The logic behind this is that the setters are called by user
> code, and each can be called multiple times. The getters are only
> (meant to be) called by the HdfsFileSystem object (that is internally
> to the HdfsProvider), and thus it is a lot easier to get all the
> settings at once than to have to make multiple calls, with potentially iterators, etc.
>
> . Is this too big a change to fit in last-minute before the release
> of VFS 2.1?
>
>
>
> Patch is attached.
>
>
>
> Thanks,
>
> ~Roger Whitcomb
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS] Further changes to HDFS Provider for alternate
configuration support
Posted by Roger Whitcomb <Ro...@actian.com>.
Okay, I'll go ahead and make a new JIRA issue, then, and attach the patch there.
Thanks,
~Roger
________________________________________
From: Pascal Schumacher <pa...@gmx.net>
Sent: Tuesday, November 17, 2015 9:48 AM
To: Commons Developers List
Subject: Re: [VFS] Further changes to HDFS Provider for alternate configuration support
Yes, the mailing list server does not allow attachments. Please attach
the patch to a JIRA issue.
Thanks!
Am 17.11.2015 um 00:11 schrieb dlmarion@comcast.net:
> I don't see the patch. It might be stripped off by the mail server.
>
>
>
> From: Roger Whitcomb [mailto:Roger.Whitcomb@actian.com]
> Sent: Monday, November 16, 2015 5:12 PM
> To: Commons Developers List
> Subject: [VFS] Further changes to HDFS Provider for alternate configuration
> support
>
>
>
> Hi all,
>
> In trying to solve some customer issues we're having, mainly
> to do with trying to browse HDFS files when the Name Node is configured for
> High-Availability, I found I needed to make some more changes/additions to
> the VFS HDFS Provider. I have attached a diff/patch file. But, a couple of
> questions:
>
> . This is a follow-on (basically) to my earlier patch for VFS-555,
> should I open a new JIRA, or just reopen the existing one?
>
> . Since this actually changes an API (but which is not released
> yet), is that an acceptable thing to do?
>
> . The new properties I have added to HdfsFileSystemConfigBuilder are
> not actually symmetrical, so they don't fit the definition of a Java
> "property", that is the setters are "setXXX" while the getters are "getXXXs"
> (plural). The logic behind this is that the setters are called by user code,
> and each can be called multiple times. The getters are only (meant to be)
> called by the HdfsFileSystem object (that is internally to the
> HdfsProvider), and thus it is a lot easier to get all the settings at once
> than to have to make multiple calls, with potentially iterators, etc.
>
> . Is this too big a change to fit in last-minute before the release
> of VFS 2.1?
>
>
>
> Patch is attached.
>
>
>
> Thanks,
>
> ~Roger Whitcomb
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS] Further changes to HDFS Provider for alternate
configuration support
Posted by Pascal Schumacher <pa...@gmx.net>.
Yes, the mailing list server does not allow attachments. Please attach
the patch to a JIRA issue.
Thanks!
Am 17.11.2015 um 00:11 schrieb dlmarion@comcast.net:
> I don't see the patch. It might be stripped off by the mail server.
>
>
>
> From: Roger Whitcomb [mailto:Roger.Whitcomb@actian.com]
> Sent: Monday, November 16, 2015 5:12 PM
> To: Commons Developers List
> Subject: [VFS] Further changes to HDFS Provider for alternate configuration
> support
>
>
>
> Hi all,
>
> In trying to solve some customer issues we're having, mainly
> to do with trying to browse HDFS files when the Name Node is configured for
> High-Availability, I found I needed to make some more changes/additions to
> the VFS HDFS Provider. I have attached a diff/patch file. But, a couple of
> questions:
>
> . This is a follow-on (basically) to my earlier patch for VFS-555,
> should I open a new JIRA, or just reopen the existing one?
>
> . Since this actually changes an API (but which is not released
> yet), is that an acceptable thing to do?
>
> . The new properties I have added to HdfsFileSystemConfigBuilder are
> not actually symmetrical, so they don't fit the definition of a Java
> "property", that is the setters are "setXXX" while the getters are "getXXXs"
> (plural). The logic behind this is that the setters are called by user code,
> and each can be called multiple times. The getters are only (meant to be)
> called by the HdfsFileSystem object (that is internally to the
> HdfsProvider), and thus it is a lot easier to get all the settings at once
> than to have to make multiple calls, with potentially iterators, etc.
>
> . Is this too big a change to fit in last-minute before the release
> of VFS 2.1?
>
>
>
> Patch is attached.
>
>
>
> Thanks,
>
> ~Roger Whitcomb
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [VFS] Further changes to HDFS Provider for alternate configuration support
Posted by dl...@comcast.net.
I don't see the patch. It might be stripped off by the mail server.
From: Roger Whitcomb [mailto:Roger.Whitcomb@actian.com]
Sent: Monday, November 16, 2015 5:12 PM
To: Commons Developers List
Subject: [VFS] Further changes to HDFS Provider for alternate configuration
support
Hi all,
In trying to solve some customer issues we're having, mainly
to do with trying to browse HDFS files when the Name Node is configured for
High-Availability, I found I needed to make some more changes/additions to
the VFS HDFS Provider. I have attached a diff/patch file. But, a couple of
questions:
. This is a follow-on (basically) to my earlier patch for VFS-555,
should I open a new JIRA, or just reopen the existing one?
. Since this actually changes an API (but which is not released
yet), is that an acceptable thing to do?
. The new properties I have added to HdfsFileSystemConfigBuilder are
not actually symmetrical, so they don't fit the definition of a Java
"property", that is the setters are "setXXX" while the getters are "getXXXs"
(plural). The logic behind this is that the setters are called by user code,
and each can be called multiple times. The getters are only (meant to be)
called by the HdfsFileSystem object (that is internally to the
HdfsProvider), and thus it is a lot easier to get all the settings at once
than to have to make multiple calls, with potentially iterators, etc.
. Is this too big a change to fit in last-minute before the release
of VFS 2.1?
Patch is attached.
Thanks,
~Roger Whitcomb