You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "CenluChen (via GitHub)" <gi...@apache.org> on 2025/04/29 03:24:15 UTC
[I] Support for Baidu advanced file system (AFS) [arrow]
CenluChen opened a new issue, #46256:
URL: https://github.com/apache/arrow/issues/46256
### Describe the enhancement requested
Baidu advanced file system (AFS) is HDFS-compatible file system and implements all HDFS interfaces which is used by Baidu, Inc. The only deferent is AFS start with afs://, so we need to follow the HDFS processing when encountering a file system starting with afs:// in arrow.
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Baidu advanced file system (AFS) [arrow]
Posted by "CenluChen (via GitHub)" <gi...@apache.org>.
CenluChen commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-2856869835
> Do we have any policy on adding new URI schemes? (maybe relevant based on the latest flight conversations [#46194](https://github.com/apache/arrow/pull/46194))
>
> Should we be worried about clashing file system URI schemes (i.e. https://en.wikipedia.org/wiki/Andrew_File_System). Not worried about Andrew File System but just as an example about one that matches the URI protocol which is IANA registered.
>
> [@zeroshade](https://github.com/zeroshade) [@lidavidm](https://github.com/lidavidm) [@pitrou](https://github.com/pitrou) [@bkietz](https://github.com/bkietz)
That's indeed a problem, afs schema may introduce conflict in the future. Currently we complile and install pyarrow in local, it will cost some time when we upgrade a new version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Baidu advanced file system (AFS) [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-2838104469
I'm also curious why, if it's HDFS compatible, it doesn't just use the HDFS scheme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Baidu advanced file system (AFS) [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-2837997145
Do we have any policy on adding new URI schemes? (maybe relevant based on the latest flight conversations https://github.com/apache/arrow/pull/46194)
Should we be worried about clashing file system URI schemes (i.e. https://en.wikipedia.org/wiki/Andrew_File_System). Not worried about Andrew File System but just as an example about one that matches the URI protocol which is IANA registered.
@zeroshade @lidavidm @pitrou @bkietz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Baidu advanced file system (AFS) [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-2856850688
But it still uses the HDFS protocol, right?
I wonder if the solution is just to allow overriding URI schemes in the filesystem layer instead of hardcoding all the possible schemes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Baidu advanced file system (AFS) [arrow]
Posted by "CenluChen (via GitHub)" <gi...@apache.org>.
CenluChen commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-2856840584
> I'm also curious why, if it's HDFS compatible, it doesn't just use the HDFS scheme.
We use HDFS at first, then we've developed a new file system that implements the HDFS interface for business and tech needs. To distinguish it from the hdfs, we used AFS as the new scheme header. Another reason is we still maintain a small number of hdfs clusters, afs schema can distinguish them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Support for Baidu advanced file system (AFS) [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #46256: [C++] Support for Baidu advanced file system (AFS)
URL: https://github.com/apache/arrow/issues/46256
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Baidu advanced file system (AFS) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-2857269712
> We use HDFS at first, then we've developed a new file system that implements the HDFS interface for business and tech needs. To distinguish it from the hdfs, we used AFS as the new scheme header.
The "hdfs" scheme is only about the _protocol_ used to communicate with the server, it doesn't say anything about the server's implementation.
Just like the "s3" scheme only tells that you're talking S3 to the server, it doesn't mandate a particular server implementation (it can be AWS, Minio or others still).
If your filesystem is talking the HDFS protocol, you shouldn't be adding a new URI scheme for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Support for Baidu advanced file system (AFS) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46256:
URL: https://github.com/apache/arrow/issues/46256#issuecomment-4642416128
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org