[jira] [Commented] (ARROW-9616) [C++] Support LTO for R

["Jeroen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-9616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341603#comment-17341603 ] 

Jeroen commented on ARROW-9616:
-------------------------------

Testing this with the new GCC-10.3 UCRT toolchain, we get more specific linking errors when using LTO on Windows. Does this give a hint what might be wrong? I'm not seeing this for any other R package / C++ library.

 
{code:java}
g++ -std=gnu++11 -shared -s -static-libgcc -o arrow.dll tmp.def array.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o runtimeinfo.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-4.0.0/lib-10.2.0/x64 -L../windows/arrow-4.0.0/lib/x64-ucrt -lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lutf8proc -lthrift -lsnappy -lz -lzstd -llz4 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -laws-cpp-sdk-config -laws-cpp-sdk-transfer -laws-cpp-sdk-identity-management -laws-cpp-sdk-cognito-identity -laws-cpp-sdk-sts -laws-cpp-sdk-s3 -laws-cpp-sdk-core -laws-c-event-stream -laws-checksums -laws-c-common -lUserenv -lversion -lws2_32 -lBcrypt -lWininet -lwinhttp -L/ucrt64/lib/x64 -L/ucrt64/lib -LC:/PROGRA~1/R/R-devel/bin/x64 -lR
`_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE' referenced in section `.rdata$_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE]' of io.o (symbol from plugin)
`_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE' referenced in section `.rdata$_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE]' of io.o (symbol from plugin)
`_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE' referenced in section `.rdata$_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTTN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTCN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE0_NS0_11InputStreamE]' of io.o (symbol from plugin)
`_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx[_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv[_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv]' of io.o (symbol from plugin)
`_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv' referenced in section `.rdata$_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE[_ZTVN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEEE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libparquet.a(schema.cc.obj): defined in discarded section `.gnu.linkonce.t._ZNK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv[_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.text' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow_dataset.a(file_base.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.text' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow_dataset.a(file_base.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.rdata$_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE[_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv' referenced in section `.rdata$_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE[_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv[_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv]' of io.o (symbol from plugin)
`_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv' referenced in section `.rdata$_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE[_ZTCN5arrow2io12BufferReaderE0_NS0_8internal34RandomAccessFileConcurrencyWrapperIS1_EE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZNK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv[_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv]' of io.o (symbol from plugin)
`_ZThn24_N5arrow2io12BufferReaderD1Ev' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io12BufferReaderD1Ev[_ZThn24_N5arrow2io12BufferReaderD1Ev]' of io.o (symbol from plugin)
`_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx[_ZThn24_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4SeekEx]' of io.o (symbol from plugin)
`_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv[_ZTv0_n32_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5CloseEv]' of io.o (symbol from plugin)
`_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZN5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv[_ZTv0_n40_N5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE5AbortEv]' of io.o (symbol from plugin)
`_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv' referenced in section `.rdata$_ZTVN5arrow2io12BufferReaderE[_ZTVN5arrow2io12BufferReaderE]' of ../windows/arrow-4.0.0/lib/x64-ucrt/libarrow.a(memory.cc.obj): defined in discarded section `.gnu.linkonce.t._ZNK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv[_ZTv0_n48_NK5arrow2io8internal34RandomAccessFileConcurrencyWrapperINS0_12BufferReaderEE4TellEv]' of io.o (symbol from plugin)
collect2.exe: error: ld returned 1 exit status
no DLL was created

 {code}

> [C++] Support LTO for R
> -----------------------
>
>                 Key: ARROW-9616
>                 URL: https://issues.apache.org/jira/browse/ARROW-9616
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, R
>    Affects Versions: 1.0.0
>            Reporter: Jeroen
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> The next version of R might enable LTO on Windows, i.e. R packages will be compiled with {{-flto}} by default. This works out of the box for most packages, but for arrow, the linker crashes as below. 
> {code}
>  C:/rtools40/mingw64/bin/g++ -shared -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -flto -s -static-libgcc -o arrow.dll tmp.def array.o array_from_vector.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o -L../windows//lib-8.3.0/x64 -L../windows//lib/x64 -lparquet -larrow_dataset -larrow -lthrift -lsnappy -lz -lzstd -llz4 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lws2_32 -LC:/PROGRA~1/R/R-devel/bin/x64 -lR
>  lto1.exe: internal compiler error: in add_symbol_to_partition_1, at lto/lto-partition.c:153
>  libbacktrace could not find executable to open
>  Please submit a full bug report,
>  with preprocessed source if appropriate.
>  See <[https://github.com/r-windows]> for instructions.
>  lto-wrapper.exe: fatal error: C:\rtools40\mingw64\bin\g++.exe returned 1 exit status
>  compilation terminated.
>  C:/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: error: lto-wrapper failed
> {code}
> You can reproduce this in R on Windows for example like so:
> {code:r}
> dir.create("~/.R")
> writeLines("CPPFLAGS=-flto", file = "~/.R/Makevars")
> install.packages("arrow", type = 'source')
> {code}
> I am not sure if this is a bug in the toolchain, or in arrow. I tried with both gcc-8.3.0 and gcc-9.3.0, and the result is the same. I did find [this issue|https://github.com/cycfi/elements/pull/56] in another project which suggests to enable `INTERPROCEDURAL_OPTIMIZATION` in cmake, when mixing lto code with non-lto code (which is the case when we only build the r bindings with lto, but not the c++ library).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)