You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2024/04/10 14:36:40 UTC

[PR] ORC-1669: [C++] Deprecate HDFS support [orc]

wgtmac opened a new pull request, #1885:
URL: https://github.com/apache/orc/pull/1885

   ### What changes were proposed in this pull request?
   Mark readHdfsFile as deprecated.
   
   ### Why are the changes needed?
   Reading ORC on HDFS was introduced in https://github.com/apache/orc/pull/134 without any test. It has not been updated for 7 years and updating libhdfspp will result in extra dependency like boost library. Staying at an old version of libhdfspp will also prohibit us from updating other libraries like protobuf.
   
   ### How was this patch tested?
   It does not need test.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2047734845

   cc @dongjoon-hyun @stiga-huang @ffacs 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "stiga-huang (via GitHub)" <gi...@apache.org>.
stiga-huang commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2048801787

   +1 for choosing the protobuf version based on the `BUILD_LIBHDFSPP` option. It might have less complaints.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2048882547

   I'm not sure what you mean by that. May I ask what is broken in `libprotobuf` PR, #1857, specifically , @wgtmac ?
   
   > Does it mean that we cannot upgrade libprotobuf like #1857 in 2.x? Or at least we can choose appropriate protobuf version according to the `BUILD_LIBHDFSPP` option.
   
   Apache ORC 2 has two layer protections.
   - One is `orc-format` which exists independently from this repo. `apache/orc` repository and PRs cannot break any ORC file format structure because `orc_proto.proto` doesn't belong to this repo. It exists in `apache/orc-format` without any change.
     - https://github.com/apache/orc-format/blob/main/src/main/proto/orc/proto/orc_proto.proto 
   
   - The other is that `ProtocolBuffer`. ProtocolBuffer is a layer to provide a language and platform neutral binary compatibility, isn't it? #1857 only changes from 3.5.1 to 3.21.12. I don't think they broken anything there because the major version is the same.
   
   FYI, `Semantic Versioning` is defined here. You can do whatever you want if you follow the policy. In this case, my concern is the part of `a backward compatible manner`. In general, `Removal` is not an option for backward compatibility in general. That's the reason why I told you that we can do only **Deprecatation**. Deprecation doesn't mean `Removal`.
   - https://semver.org
   ```
   Given a version number MAJOR.MINOR.PATCH, increment the:
   
   MAJOR version when you make incompatible API changes
   MINOR version when you add functionality in a backward compatible manner
   PATCH version when you make backward compatible bug fixes
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2048746441

   Does it mean that we cannot upgrade libprotobuf like https://github.com/apache/orc/pull/1857 in 2.x? Or at least we can choose appropriate protobuf version according to the `BUILD_LIBHDFSPP` option. @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2048908475

   > https://github.com/apache/orc/pull/1857 only changes from 3.5.1 to 3.21.12. The major version is the same. Do you mean they break anything there?
   
   Yes, libhdfspp does not link with 3.21.12. For example:
   ```
   [ 84%] Linking CXX executable orc-test
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(namenode_operations.cc.o): in function `hadoop::hdfs::GetFsStatusRequestProto::~GetFsStatusRequestProto()':
   /root/build/libhdfspp_ep-prefix/src/libhdfspp_ep-build/lib/proto/ClientNamenodeProtocol.pb.h:9833: undefined reference to `google::protobuf::internal::ZeroFieldsBase::~ZeroFieldsBase()'
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(namenode_operations.cc.o): in function `hadoop::hdfs::DisallowSnapshotResponseProto::~DisallowSnapshotResponseProto()':
   /root/build/libhdfspp_ep-prefix/src/libhdfspp_ep-build/lib/proto/ClientNamenodeProtocol.pb.h:23777: undefined reference to `google::protobuf::internal::ZeroFieldsBase::~ZeroFieldsBase()'
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(namenode_operations.cc.o): in function `hadoop::hdfs::AllowSnapshotResponseProto::~AllowSnapshotResponseProto()':
   /root/build/libhdfspp_ep-prefix/src/libhdfspp_ep-build/lib/proto/ClientNamenodeProtocol.pb.h:23487: undefined reference to `google::protobuf::internal::ZeroFieldsBase::~ZeroFieldsBase()'
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(namenode_operations.cc.o): in function `hadoop::hdfs::RenameSnapshotResponseProto::~RenameSnapshotResponseProto()':
   /root/build/libhdfspp_ep-prefix/src/libhdfspp_ep-build/lib/proto/ClientNamenodeProtocol.pb.h:23197: undefined reference to `google::protobuf::internal::ZeroFieldsBase::~ZeroFieldsBase()'
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(namenode_operations.cc.o): in function `hadoop::hdfs::DeleteSnapshotResponseProto::~DeleteSnapshotResponseProto()':
   /root/build/libhdfspp_ep-prefix/src/libhdfspp_ep-build/lib/proto/ClientNamenodeProtocol.pb.h:24090: undefined reference to `google::protobuf::internal::ZeroFieldsBase::~ZeroFieldsBase()'
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(namenode_operations.cc.o):/root/build/libhdfspp_ep-prefix/src/libhdfspp_ep-build/lib/proto/ClientNamenodeProtocol.pb.h:4055: more undefined references to `google::protobuf::internal::ZeroFieldsBase::~ZeroFieldsBase()' follow
   /usr/bin/ld: ../libs/thirdparty/libhdfspp_ep-install/lib/libhdfspp_static.a(ClientNamenodeProtocol.pb.cc.o):
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2048922873

   Although we don't want to touch that file, I guess we are able to update the source code like we did in the following PR. WDTY?
   - #1685


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2049597361

   Right, I feel very reluctant to touch that file. I'll look into it later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2048127937

   cc @williamhyun , @pavibhai , @deshanxiao , too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2047738076

   Also cc @omalley in case there is any blocker to this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1885: ORC-1669: [C++] Deprecate HDFS support
URL: https://github.com/apache/orc/pull/1885


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1669: [C++] Deprecate HDFS support [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1885:
URL: https://github.com/apache/orc/pull/1885#issuecomment-2054085136

   Thanks @dongjoon-hyun!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org