You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Zhuo Peng <br...@gmail.com> on 2019/10/04 16:05:02 UTC
Should Arrow adopt C++14 / 17?
Dear Arrow maintainers,
Sorry if this was raised before. I did search the mailing list but "C++" matched too many results..
With manylinux1 (GCC4.8) being sunset, both Conda and Pypa are providing a modern enough toolchain (Conda Forge - GCC7; Pypa manylinux2010 docker - devtoolset-8(GCC8)). And full C++17 support has been included in GCC7 [1]. I wonder what are the concerns of adopting a newer standard?
C++14 might not bring a whole lot of interesting features, but C++17 brings:
std::string_view
std::optional
std::variant (the newly added Result class is based on some form of variant implementation I suppose?)
and many syntax sugar.. (like emplace_back() returning back(), so you can do RETURN_NOT_OK(CreateArray(my_array_sp_vector.emplace_back())))
And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
[1] https://gcc.gnu.org/projects/cxx-status.html
Re: Should Arrow adopt C++14 / 17?
Posted by Antoine Pitrou <an...@python.org>.
C++14 isn't very interesting. C++17 is, but it's probably too young
given the diversity of platform and toolchain requirements that constrai us.
Regards
Antoine.
Le 04/10/2019 à 18:13, Neal Richardson a écrit :
> We do have to care about more than just conda and Python. For R, for
> example, C++14 support is limited:
> https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-C_002b_002b14-code
>
> That's not to say that we can't try (if C++ devs wanted to, and I
> can't speak for them), but it might be time-consuming (or worse) to
> figure out what features are supported on all of the platforms and
> languages that Arrow C++ intends to work for. That said, hopefully our
> CI coverage is sufficient to let someone try it out and see what
> breaks.
>
> Neal
>
> On Fri, Oct 4, 2019 at 9:05 AM Zhuo Peng <br...@gmail.com> wrote:
>>
>> Dear Arrow maintainers,
>>
>> Sorry if this was raised before. I did search the mailing list but "C++" matched too many results..
>>
>> With manylinux1 (GCC4.8) being sunset, both Conda and Pypa are providing a modern enough toolchain (Conda Forge - GCC7; Pypa manylinux2010 docker - devtoolset-8(GCC8)). And full C++17 support has been included in GCC7 [1]. I wonder what are the concerns of adopting a newer standard?
>>
>> C++14 might not bring a whole lot of interesting features, but C++17 brings:
>>
>> std::string_view
>> std::optional
>> std::variant (the newly added Result class is based on some form of variant implementation I suppose?)
>>
>> and many syntax sugar.. (like emplace_back() returning back(), so you can do RETURN_NOT_OK(CreateArray(my_array_sp_vector.emplace_back())))
>>
>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
>>
>> [1] https://gcc.gnu.org/projects/cxx-status.html
>>
Re: Should Arrow adopt C++14 / 17?
Posted by Neal Richardson <ne...@gmail.com>.
We do have to care about more than just conda and Python. For R, for
example, C++14 support is limited:
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-C_002b_002b14-code
That's not to say that we can't try (if C++ devs wanted to, and I
can't speak for them), but it might be time-consuming (or worse) to
figure out what features are supported on all of the platforms and
languages that Arrow C++ intends to work for. That said, hopefully our
CI coverage is sufficient to let someone try it out and see what
breaks.
Neal
On Fri, Oct 4, 2019 at 9:05 AM Zhuo Peng <br...@gmail.com> wrote:
>
> Dear Arrow maintainers,
>
> Sorry if this was raised before. I did search the mailing list but "C++" matched too many results..
>
> With manylinux1 (GCC4.8) being sunset, both Conda and Pypa are providing a modern enough toolchain (Conda Forge - GCC7; Pypa manylinux2010 docker - devtoolset-8(GCC8)). And full C++17 support has been included in GCC7 [1]. I wonder what are the concerns of adopting a newer standard?
>
> C++14 might not bring a whole lot of interesting features, but C++17 brings:
>
> std::string_view
> std::optional
> std::variant (the newly added Result class is based on some form of variant implementation I suppose?)
>
> and many syntax sugar.. (like emplace_back() returning back(), so you can do RETURN_NOT_OK(CreateArray(my_array_sp_vector.emplace_back())))
>
> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
>
> [1] https://gcc.gnu.org/projects/cxx-status.html
>
Re: Should Arrow adopt C++14 / 17?
Posted by Wes McKinney <we...@gmail.com>.
I opened https://issues.apache.org/jira/browse/ARROW-6800.
On Fri, Oct 4, 2019 at 3:42 PM Wes McKinney <we...@gmail.com> wrote:
>
> On Fri, Oct 4, 2019 at 3:18 PM Zhuo Peng <br...@gmail.com> wrote:
> >
> >
> >
> > On 2019/10/04 19:43:04, Wes McKinney <we...@gmail.com> wrote:
> > > On Fri, Oct 4, 2019 at 12:45 PM Zhuo Peng <br...@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > On 2019/10/04 17:05:00, Antoine Pitrou <an...@python.org> wrote:
> > > > >
> > > > > Le 04/10/2019 à 19:01, Zhuo Peng a écrit :
> > > > > >
> > > > > > backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
> > > > >
> > > > > This is true unfortunately.
> > > > >
> > > > > >>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
> > > > > >>
> > > > > >> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
> > > > > > https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
> > > > > >
> > > > > > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
> > > > >
> > > > > Right, so this is a CMake decision. I think we require only plain C++11
> > > > > (but we may enable additional features on some compilers, provided
> > > > > there's a fallback).
> > > > Extensions can be disabled through:
> > > > set(CMAKE_CXX_EXTENSIONS OFF)
> > > >
> > > > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_EXTENSIONS.html
> > > >
> > > > Is that something more desirable than the current state?
> > >
> > > Yes, I think so, I don't think we need to be relying on GNU gcc
> > > extensions, but we should open a JIRA issue about disabling it in case
> > > some tests break because of something we didn't realize we were
> > > depending on.
> > sg. I'll create one then.
> > >
> > > As far as C++14/17 upgrading, it seems like it will be at least 2
> > > years before we could upgrade to C++17 given the state of compiler
> > > support across the spectrum. Using C++17 would mean requiring at least
> > > VS 2017 on Windows, since at least in the Python world I think
> > > everything is on VS 2015.
> > >
> > > Are there ways we could create defines to switch between backports and
> > > STL things (like string_view, optional, etc.) so that developers using
> > > the Arrow library in a C++17 application can use the built-in types?
> > This is dangerous unless they build the Arrow library from source with C++17. if libarrow takes arrow::string_view but the user gives it a std::string_view, it's UB.
> >
> > If we are talking about allowing users to build Arrow with C++17 and support transparently the new STL types in the public APIs, the ABSL[1] library could be something to consider.. absl::{string_view,optional,variant} becomes their std:: counterparts when compiled under C++17, e.g. [2].
> >
>
> Yes, the presumption would be a monotoolchain environment, and Arrow
> would need to have some CMake options to build in C++17 mode
>
> > And inline namespaces are used [3] to make sure different libraries can depend on different version of absl.
> >
> > [1] https://abseil.io/
> > [2] https://github.com/abseil/abseil-cpp/blob/25597bdfc148e91e27678ec30efa52f4fc8c164f/absl/strings/string_view.h#L38
> > [3] https://github.com/abseil/abseil-cpp/blob/aa844899c937bde5d2b24f276b59997e5b668bde/absl/strings/string_view.h#L38
> > >
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > > >
> > >
Re: Should Arrow adopt C++14 / 17?
Posted by Wes McKinney <we...@gmail.com>.
On Fri, Oct 4, 2019 at 3:18 PM Zhuo Peng <br...@gmail.com> wrote:
>
>
>
> On 2019/10/04 19:43:04, Wes McKinney <we...@gmail.com> wrote:
> > On Fri, Oct 4, 2019 at 12:45 PM Zhuo Peng <br...@gmail.com> wrote:
> > >
> > >
> > >
> > > On 2019/10/04 17:05:00, Antoine Pitrou <an...@python.org> wrote:
> > > >
> > > > Le 04/10/2019 à 19:01, Zhuo Peng a écrit :
> > > > >
> > > > > backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
> > > >
> > > > This is true unfortunately.
> > > >
> > > > >>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
> > > > >>
> > > > >> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
> > > > > https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
> > > > >
> > > > > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
> > > >
> > > > Right, so this is a CMake decision. I think we require only plain C++11
> > > > (but we may enable additional features on some compilers, provided
> > > > there's a fallback).
> > > Extensions can be disabled through:
> > > set(CMAKE_CXX_EXTENSIONS OFF)
> > >
> > > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_EXTENSIONS.html
> > >
> > > Is that something more desirable than the current state?
> >
> > Yes, I think so, I don't think we need to be relying on GNU gcc
> > extensions, but we should open a JIRA issue about disabling it in case
> > some tests break because of something we didn't realize we were
> > depending on.
> sg. I'll create one then.
> >
> > As far as C++14/17 upgrading, it seems like it will be at least 2
> > years before we could upgrade to C++17 given the state of compiler
> > support across the spectrum. Using C++17 would mean requiring at least
> > VS 2017 on Windows, since at least in the Python world I think
> > everything is on VS 2015.
> >
> > Are there ways we could create defines to switch between backports and
> > STL things (like string_view, optional, etc.) so that developers using
> > the Arrow library in a C++17 application can use the built-in types?
> This is dangerous unless they build the Arrow library from source with C++17. if libarrow takes arrow::string_view but the user gives it a std::string_view, it's UB.
>
> If we are talking about allowing users to build Arrow with C++17 and support transparently the new STL types in the public APIs, the ABSL[1] library could be something to consider.. absl::{string_view,optional,variant} becomes their std:: counterparts when compiled under C++17, e.g. [2].
>
Yes, the presumption would be a monotoolchain environment, and Arrow
would need to have some CMake options to build in C++17 mode
> And inline namespaces are used [3] to make sure different libraries can depend on different version of absl.
>
> [1] https://abseil.io/
> [2] https://github.com/abseil/abseil-cpp/blob/25597bdfc148e91e27678ec30efa52f4fc8c164f/absl/strings/string_view.h#L38
> [3] https://github.com/abseil/abseil-cpp/blob/aa844899c937bde5d2b24f276b59997e5b668bde/absl/strings/string_view.h#L38
> >
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> >
Re: Should Arrow adopt C++14 / 17?
Posted by Zhuo Peng <br...@gmail.com>.
On 2019/10/04 19:43:04, Wes McKinney <we...@gmail.com> wrote:
> On Fri, Oct 4, 2019 at 12:45 PM Zhuo Peng <br...@gmail.com> wrote:
> >
> >
> >
> > On 2019/10/04 17:05:00, Antoine Pitrou <an...@python.org> wrote:
> > >
> > > Le 04/10/2019 à 19:01, Zhuo Peng a écrit :
> > > >
> > > > backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
> > >
> > > This is true unfortunately.
> > >
> > > >>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
> > > >>
> > > >> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
> > > > https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
> > > >
> > > > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
> > >
> > > Right, so this is a CMake decision. I think we require only plain C++11
> > > (but we may enable additional features on some compilers, provided
> > > there's a fallback).
> > Extensions can be disabled through:
> > set(CMAKE_CXX_EXTENSIONS OFF)
> >
> > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_EXTENSIONS.html
> >
> > Is that something more desirable than the current state?
>
> Yes, I think so, I don't think we need to be relying on GNU gcc
> extensions, but we should open a JIRA issue about disabling it in case
> some tests break because of something we didn't realize we were
> depending on.
sg. I'll create one then.
>
> As far as C++14/17 upgrading, it seems like it will be at least 2
> years before we could upgrade to C++17 given the state of compiler
> support across the spectrum. Using C++17 would mean requiring at least
> VS 2017 on Windows, since at least in the Python world I think
> everything is on VS 2015.
>
> Are there ways we could create defines to switch between backports and
> STL things (like string_view, optional, etc.) so that developers using
> the Arrow library in a C++17 application can use the built-in types?
This is dangerous unless they build the Arrow library from source with C++17. if libarrow takes arrow::string_view but the user gives it a std::string_view, it's UB.
If we are talking about allowing users to build Arrow with C++17 and support transparently the new STL types in the public APIs, the ABSL[1] library could be something to consider.. absl::{string_view,optional,variant} becomes their std:: counterparts when compiled under C++17, e.g. [2].
And inline namespaces are used [3] to make sure different libraries can depend on different version of absl.
[1] https://abseil.io/
[2] https://github.com/abseil/abseil-cpp/blob/25597bdfc148e91e27678ec30efa52f4fc8c164f/absl/strings/string_view.h#L38
[3] https://github.com/abseil/abseil-cpp/blob/aa844899c937bde5d2b24f276b59997e5b668bde/absl/strings/string_view.h#L38
>
> > >
> > > Regards
> > >
> > > Antoine.
> > >
>
Re: Should Arrow adopt C++14 / 17?
Posted by Wes McKinney <we...@gmail.com>.
On Fri, Oct 4, 2019 at 12:45 PM Zhuo Peng <br...@gmail.com> wrote:
>
>
>
> On 2019/10/04 17:05:00, Antoine Pitrou <an...@python.org> wrote:
> >
> > Le 04/10/2019 à 19:01, Zhuo Peng a écrit :
> > >
> > > backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
> >
> > This is true unfortunately.
> >
> > >>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
> > >>
> > >> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
> > > https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
> > >
> > > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
> >
> > Right, so this is a CMake decision. I think we require only plain C++11
> > (but we may enable additional features on some compilers, provided
> > there's a fallback).
> Extensions can be disabled through:
> set(CMAKE_CXX_EXTENSIONS OFF)
>
> https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_EXTENSIONS.html
>
> Is that something more desirable than the current state?
Yes, I think so, I don't think we need to be relying on GNU gcc
extensions, but we should open a JIRA issue about disabling it in case
some tests break because of something we didn't realize we were
depending on.
As far as C++14/17 upgrading, it seems like it will be at least 2
years before we could upgrade to C++17 given the state of compiler
support across the spectrum. Using C++17 would mean requiring at least
VS 2017 on Windows, since at least in the Python world I think
everything is on VS 2015.
Are there ways we could create defines to switch between backports and
STL things (like string_view, optional, etc.) so that developers using
the Arrow library in a C++17 application can use the built-in types?
> >
> > Regards
> >
> > Antoine.
> >
Re: Should Arrow adopt C++14 / 17?
Posted by Zhuo Peng <br...@gmail.com>.
On 2019/10/04 17:05:00, Antoine Pitrou <an...@python.org> wrote:
>
> Le 04/10/2019 à 19:01, Zhuo Peng a écrit :
> >
> > backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
>
> This is true unfortunately.
>
> >>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
> >>
> >> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
> > https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
> >
> > https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
>
> Right, so this is a CMake decision. I think we require only plain C++11
> (but we may enable additional features on some compilers, provided
> there's a fallback).
Extensions can be disabled through:
set(CMAKE_CXX_EXTENSIONS OFF)
https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_EXTENSIONS.html
Is that something more desirable than the current state?
>
> Regards
>
> Antoine.
>
Re: Should Arrow adopt C++14 / 17?
Posted by Antoine Pitrou <an...@python.org>.
Le 04/10/2019 à 19:01, Zhuo Peng a écrit :
>
> backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
This is true unfortunately.
>>> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
>>
>> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
> https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
>
> https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
Right, so this is a CMake decision. I think we require only plain C++11
(but we may enable additional features on some compilers, provided
there's a fallback).
Regards
Antoine.
Re: Should Arrow adopt C++14 / 17?
Posted by Zhuo Peng <br...@gmail.com>.
On 2019/10/04 16:53:59, Antoine Pitrou <an...@python.org> wrote:
>
> Le 04/10/2019 à 18:05, Zhuo Peng a écrit :
> > Dear Arrow maintainers,
> >
> > Sorry if this was raised before. I did search the mailing list but "C++" matched too many results..
> >
> > With manylinux1 (GCC4.8) being sunset, both Conda and Pypa are providing a modern enough toolchain (Conda Forge - GCC7; Pypa manylinux2010 docker - devtoolset-8(GCC8)). And full C++17 support has been included in GCC7 [1]. I wonder what are the concerns of adopting a newer standard?
> >
> > C++14 might not bring a whole lot of interesting features, but C++17 brings:
> >
> > std::string_view
> > std::optional
> > std::variant (the newly added Result class is based on some form of variant implementation I suppose?)
>
> We already have `string_view` and `variant` backports. We could
> reasonably add a `optional` backport.
>
backports are cool for internal use, but probably not so if a public API accepts it? (because you vendor the headers in (i.e. namespace, symbol names unchanged), they might clash with headers that a client uses).
> > And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
>
> None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
https://github.com/apache/arrow/blob/3129e3ed90219ecfffe2a25ce5820eec8cc947d0/cpp/cmake_modules/SetupCxxFlags.cmake#L33
https://cmake.org/cmake/help/v3.1/prop_tgt/CXX_STANDARD.html
>
> Regards
>
> Antoine.
>
Re: Should Arrow adopt C++14 / 17?
Posted by Antoine Pitrou <an...@python.org>.
Le 04/10/2019 à 18:05, Zhuo Peng a écrit :
> Dear Arrow maintainers,
>
> Sorry if this was raised before. I did search the mailing list but "C++" matched too many results..
>
> With manylinux1 (GCC4.8) being sunset, both Conda and Pypa are providing a modern enough toolchain (Conda Forge - GCC7; Pypa manylinux2010 docker - devtoolset-8(GCC8)). And full C++17 support has been included in GCC7 [1]. I wonder what are the concerns of adopting a newer standard?
>
> C++14 might not bring a whole lot of interesting features, but C++17 brings:
>
> std::string_view
> std::optional
> std::variant (the newly added Result class is based on some form of variant implementation I suppose?)
We already have `string_view` and `variant` backports. We could
reasonably add a `optional` backport.
> And btw, was -std=gnu++11 an intentional choice? what gnu extensions does the library rely on?
None, AFAIK. Arrow compiles on MSVC fine. Where is -std=gnu++11 added?
Regards
Antoine.