You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Neal Richardson (Jira)" <ji...@apache.org> on 2021/08/11 13:03:00 UTC

[jira] [Created] (ARROW-13606) [R] Actually disable LTO

Neal Richardson created ARROW-13606:
---------------------------------------

             Summary: [R] Actually disable LTO
                 Key: ARROW-13606
                 URL: https://issues.apache.org/jira/browse/ARROW-13606
             Project: Apache Arrow
          Issue Type: New Feature
          Components: R
            Reporter: Neal Richardson
             Fix For: 6.0.0


Background: ARROW-12853 reported that if the C++ code in the R package is built with {{-flto}}, it compiles and links successfully, but then the package segfaults on load. ARROW-13199 attempted to fix this by adding {{-fno-lto}} to the build, and it did work for the ubuntu 21.04 CI job. However, the LTO check on CRAN failed. 

The reason for the difference between the passing ubuntu 21.04 CI job and the CRAN check turns out to be that ARROW-13199 only actually added {{-fno-lto}} to the link step. It intended to add them to the CXXFLAGS for compiling too, but it set the wrong variable, so they were not passed in. Yet, the build was still successful because the CXXFLAGS used on the ubuntu 21.04 build included {{-ffat-lto-objects}} in addition to {{-flto}}. fat-lto-objects can be linked [with or without LTO|https://stackoverflow.com/questions/13799452/what-is-the-difference-in-gcc-between-lto-and-fat-lto-objects], so the {{-fno-lto}} at link time was able to work. The CRAN machine configuration does not have {{-ffat-lto-objects}} as a compile flag, so the {{-fno-lto}} in the linker meant that linking would fail.

There are two solutions: fix the segfault-on-load and allow LTO or properly disable LTO. This issue is to do the latter; we are continuing to work on the former in a separate issue. ARROW-13507 tried to fix this by setting {{UseLTO: false}} in {{DESCRIPTION}}, but it turns out that the CRAN check overrides that flag and forces LTO anyway, so that isn't a viable option.

Example output from the ubuntu-21.04 build (compiles with {{-flto=auto -ffat-lto-objects}}, links with {{-flto=10 -fno-lto}}, result is successful non-LTO library

{code}
...
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I/arrow/r/check/arrow.Rcheck/00_pkg_src/arrow/libarrow/arrow-5.0.0.9000/include  -Werror -DARROW_R_WITH_ARROW -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -I'/usr/local/lib/R/site-library/cpp11/include'    -fpic  -g -O2 -ffile-prefix-map=/build/r-base-aXXzqd/r-base-4.1.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c type_infer.cpp -o type_infer.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/arrow/r/check/arrow.Rcheck/00_pkg_src/arrow/libarrow/arrow-5.0.0.9000/lib -larrow_dataset -lparquet -larrow -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -fno-lto -L/usr/lib/R/lib -lR
...
{code}

Example output from the CRAN check (compiles with {{-flto=10}}, links with {{-fno-lto}}, so it fails:

{code}
...
g++ -std=gnu++11 -I"/data/gannet/ripley/R/R-devel/include" -DNDEBUG -I/data/gannet/ripley/R/packages/tests-LTO/arrow/libarrow/arrow-5.0.0/include  -DARROW_R_WITH_ARROW -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -I'/data/gannet/ripley/R/test-4.2/cpp11/include' -I/usr/local/include   -fpic  -g -O2 -Wall -pedantic -mtune=native -Wno-ignored-attributes -Wno-parentheses -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong -fstack-clash-protection -fcf-protection -flto=10 -c type_infer.cpp -o type_infer.o
g++ -std=gnu++11 -shared -L/usr/local/lib64 -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o config.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/data/gannet/ripley/R/packages/tests-LTO/arrow/libarrow/arrow-5.0.0/lib -larrow_dataset -lparquet -larrow -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -fno-lto
/usr/bin/ld: RTasks.o: plugin needed to handle lto object
/usr/bin/ld: RTasks.o: plugin needed to handle lto object
/usr/bin/ld: altrep.o: plugin needed to handle lto object
...
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)