You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/06/14 01:11:34 UTC

[GitHub] [arrow] ctring opened a new pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

ctring opened a new pull request #7430:
URL: https://github.com/apache/arrow/pull/7430


   In Linux, the configuration step of Boost allows selecting a subset of the libraries being built using the `--with-libraries=` flag. However, this flag does not exist in the configure script on Windows, so now we're building all libraries, including those being trimmed:
   ```
   Component configuration:
   
    - atomic : building
    - chrono : building
    - container : building
    - date_time : building
    - exception : building
    - filesystem : building
    - headers : building
    - iostreams : building
    - locale : building
    - log : building
    - mpi : building
    - program_options : building
    - python : building
    - random : building
    - regex : building
    - serialization : building
    - system : building
    - test : building
    - thread : building
    - timer : building
    - wave : building
   ```
   
   To select only the necessary libraries on Windows, we can provide the `--with-<library>` flag to `./b2`. Doing so helped getting me pass the boost building step but the thrift building step failed. This time, it is still due to a header file `typeof/incr_registration_group.hpp` missing in Boost (this file is only required on Windows but not on Linux); thus, added that header file to the `trim-boost.sh` script. 
   
   I verified that this can now build on both Windows and Linux with the commands:
   ```
   mkdir build
   cd build
   cmake .. -DARROW_PARQUET=ON
   cmake --build .
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644412255


   It will work but it may increase CI time for "C++ / AMD64 Windows 2019 C++".
   Increasing CI time may not be acceptable.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-645015432


   OK. Could you update the bundle and then re-run the build?
   
   And could you add me ( https://bintray.com/kou ) to https://bintray.com/ursalabs 's members?
   I may want to upload something to there in the future.
   Or should we create https://bintray.com/apache/arrow/boost and migrate https://bintray.com/ursalabs/arrow-boost/arrow-boost to https://bintray.com/apache/arrow/boost ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644450282


   OK. Let's try it in this pull request. If it doesn't increase build time much, we can change the current job.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644413986


   I suspect it wouldn't increase build time much, at least not enough to block things. Where would you prefer to run the test? Do we run windows nightlies on crossbow?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-645084134


   > Done.
   
   Thanks!
   Now, I'm a member of https://bintray.com/ursalabs .
   (Please ignore my meaningless join request...)
   
   > We could I guess, though I don't think I have sufficient permissions on the org to do that.
   
   You can join https://bintray.com/apache/arrow/ by requesting it to INFRA like https://issues.apache.org/jira/browse/INFRA-18698 .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou closed pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou closed pull request #7430:
URL: https://github.com/apache/arrow/pull/7430


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644410813


   Maybe adding `BOOST_SOURCE: BUNDLED` around [here](https://github.com/apache/arrow/blob/master/.github/workflows/cpp.yml#L179) would be sufficient? The job is already building Thrift.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-645041249


   @github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] ctring commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
ctring commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644468486


   I believe you need to regenerate and reupload the trimmed Boost bundle. This failed due to the same reason I mentioned above since it is still downloading the faulty bundle.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644468001


   Failed. Do we need to re-generate our Boost archive?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644207536


   Thanks @ctring! 
   
   Since the builds have been green, we must not have any CI jobs that test building with bundled boost and thrift on MSVC. Should we add one (nightly perhaps)? @kou thoughts?
   
   To be clear, if we add such a job, it should fail until the trimmed boost is rebuild, assuming that that typeof/incr_registration_group.hpp is required as reported here. How exactly did you confirm that these changes work as expected, @ctring?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-645041382


   Uploaded and re-triggered the failing build for confirmation that it's working now. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kou commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644397052


   > Since the builds have been green, we must not have any CI jobs that test building with bundled boost and thrift on MSVC. Should we add one (nightly perhaps)? @kou thoughts?
   
   Yes.
   We should add a job for the case to our nightly test jobs.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-643703705


   https://issues.apache.org/jira/browse/ARROW-9126


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644849969


   Yes, I expected that it should fail, thus confirming the issue. I can update the bundle and then re-run the build, and when that passes, 🎉 we've fixed the issue :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nealrichardson commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-645021672


   > OK. Could you update the bundle and then re-run the build?
   
   Yes, will do
   
   > And could you add me ( https://bintray.com/kou ) to https://bintray.com/ursalabs 's members?
   > I may want to upload something to there in the future.
   
   Done.
   
   > Or should we create https://bintray.com/apache/arrow/boost and migrate https://bintray.com/ursalabs/arrow-boost/arrow-boost to https://bintray.com/apache/arrow/boost ?
   
   We could I guess, though I don't think I have sufficient permissions on the org to do that.  
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] ctring commented on pull request #7430: ARROW-9126: [C++] Fix building trimmed Boost bundle on Windows

Posted by GitBox <gi...@apache.org>.
ctring commented on pull request #7430:
URL: https://github.com/apache/arrow/pull/7430#issuecomment-644373157


   @nealrichardson Once I made all the changes in the CMake file, I could pass the boost_ep build phase but failed at thrift_ep with the error that `typeof/incr_registration_group.hpp` was missing.
   ```
   ...\build\boost_ep-prefix\src\boost_ep\boost/scope_exit.hpp(257,10): fatal error C1083: Cannot open include file: 'boost/typeof/incr_registration_group.hpp': No such file or directory (compiling source file C:\source\arrow\cpp\build\thrift_ep-prefix\src\thrift_ep\lib\cpp\src\thrift\windows\OverlappedSubmissionThread.cpp) [C:\source\arrow\cpp\build\thrift_ep-prefix\src\thrift_ep-build\lib\cpp\thrift_static.vcxproj]
   
   ```
   From the error, you can tell the reason why this happened on Windows but not Linux: Thrift would additionally compile the file `windows\OverlappedSubmissionThread.cpp` on Windows, which required `boost/scope_exit.hpp`, which included `boost/typeof/incr_registration_group.hpp`
   
   I then copied the `boost/typeof/incr_registration_group.hpp` from the original Boost source to the boost_ep folder in the build tree. After that, I could build everything successfully, which means that header was the only missing file.
   
   I also ran the `trim-boost.sh` script to make sure the missing header file would be in the new bundle.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org