You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/08 19:06:08 UTC
[GitHub] [arrow] pitrou opened a new pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
pitrou opened a new pull request #9657:
URL: https://github.com/apache/arrow/pull/9657
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-792999306
@jonkeane @uwe I don't know if you can easily test this on a x86 Mac.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793821658
Shouldn't native show `neon`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#discussion_r590126902
##########
File path: cpp/src/arrow/util/cpu_info.cc
##########
@@ -330,6 +346,29 @@ void CpuInfo::Init() {
if (QueryPerformanceFrequency(&performance_frequency)) {
max_mhz = static_cast<float>(performance_frequency.QuadPart);
}
+#elif defined(__APPLE__)
+ // On macOS, get CPU information from system information base
+ struct SysCtlCpuFeature {
+ const char* name;
+ int64_t flag;
+ };
+ for (auto feature : std::vector<SysCtlCpuFeature>{
+ {"hw.optional.sse4_2", SSSE3 | SSE4_1 | SSE4_2 | POPCNT},
+ {"hw.optional.avx1_0", AVX},
+ {"hw.optional.avx2_0", AVX2},
+ {"hw.optional.bmi1", BMI1},
+ {"hw.optional.bmi2", BMI2},
+ {"hw.optional.avx512f", AVX512F},
+ {"hw.optional.avx512cd", AVX512CD},
+ {"hw.optional.avx512dq", AVX512DQ},
+ {"hw.optional.avx512bw", AVX512BW},
+ {"hw.optional.avx512vl", AVX512VL},
Review comment:
Can do :-) By the way, is there a difference between "Neon" and "ASIMD"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #9657:
URL: https://github.com/apache/arrow/pull/9657
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793029067
On the system I ran that I've got an Intel Xeon CPU E5-1650 v2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793684791
On an M1 I get the following output:
```
In [2]: pa.runtime_info()
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.sse4_2'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx1_0'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx2_0'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.bmi1'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.bmi2'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512f'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512cd'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512dq'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512bw'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512vl'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.l3cachesize'. Detail: [errno 22] Invalid argument
Out[2]: RuntimeInfo(simd_level='none', detected_simd_level='none')
```
> jonkeane uwe I don't know if you can easily test this on a x86 Mac.
Different Uwe in that handle ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on a change in pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#discussion_r591225133
##########
File path: cpp/src/arrow/util/cpu_info.cc
##########
@@ -330,6 +346,29 @@ void CpuInfo::Init() {
if (QueryPerformanceFrequency(&performance_frequency)) {
max_mhz = static_cast<float>(performance_frequency.QuadPart);
}
+#elif defined(__APPLE__)
+ // On macOS, get CPU information from system information base
+ struct SysCtlCpuFeature {
+ const char* name;
+ int64_t flag;
+ };
+ for (auto feature : std::vector<SysCtlCpuFeature>{
+ {"hw.optional.sse4_2", SSSE3 | SSE4_1 | SSE4_2 | POPCNT},
+ {"hw.optional.avx1_0", AVX},
+ {"hw.optional.avx2_0", AVX2},
+ {"hw.optional.bmi1", BMI1},
+ {"hw.optional.bmi2", BMI2},
+ {"hw.optional.avx512f", AVX512F},
+ {"hw.optional.avx512cd", AVX512CD},
+ {"hw.optional.avx512dq", AVX512DQ},
+ {"hw.optional.avx512bw", AVX512BW},
+ {"hw.optional.avx512vl", AVX512VL},
Review comment:
Double checked with an expert from Arm. Actually **Neon == Advanced SIMD**.
From Arm Architecture Reference Manual (ArmARM):
`The Advanced SIMD architecture, its associated implementations, and supporting software, are commonly referred
to as NEON™ technology". `
`Neon` was the codename that became public and more commonly used than the "proper name" (Advanced SIMD).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793027775
What CPU do you have, by the way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793027510
It works! Thanks
This is with the R code from #9641, but c++ from this branch:
```r
jkeane@het r % R -e "arrow:::runtime_info()"
> arrow:::runtime_info()
[1] "avx" "avx"
>
jkeane@het r % ARROW_USER_SIMD_LEVEL=sse4_2 R -e "arrow:::runtime_info()"
> arrow:::runtime_info()
[1] "sse4_2" "avx"
>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy edited a comment on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
xhochy edited a comment on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793684791
On an M1 I get the following output for `a397fc3cbebf44f0e0b3bd1e3eccfbd69eb507eb`
```
In [2]: pa.runtime_info()
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.sse4_2'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx1_0'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx2_0'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.bmi1'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.bmi2'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512f'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512cd'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512dq'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512bw'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.optional.avx512vl'. Detail: [errno 45] Operation not supported
../src/arrow/util/cpu_info.cc:92: IOError: sysctlbyname failed for 'hw.l3cachesize'. Detail: [errno 22] Invalid argument
Out[2]: RuntimeInfo(simd_level='none', detected_simd_level='none')
```
> jonkeane uwe I don't know if you can easily test this on a x86 Mac.
Different Uwe in that handle ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793710137
> Thank you @xhochy .
> The joke is that ENOTSUP isn't mentioned in Apple's own [manual page](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysctlbyname.3.html)...
Also not mentioned on the manpage on the actual system. In general, I have the feeling that near to none of the internal workings / changes of macos-on-M1 vs macos-on-Intel has been published yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793818399
Ok, thank you. So it seems to work now. I'll wait for CI and then merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#discussion_r590127868
##########
File path: cpp/src/arrow/util/cpu_info.cc
##########
@@ -330,6 +346,29 @@ void CpuInfo::Init() {
if (QueryPerformanceFrequency(&performance_frequency)) {
max_mhz = static_cast<float>(performance_frequency.QuadPart);
}
+#elif defined(__APPLE__)
+ // On macOS, get CPU information from system information base
+ struct SysCtlCpuFeature {
+ const char* name;
+ int64_t flag;
+ };
+ for (auto feature : std::vector<SysCtlCpuFeature>{
+ {"hw.optional.sse4_2", SSSE3 | SSE4_1 | SSE4_2 | POPCNT},
+ {"hw.optional.avx1_0", AVX},
+ {"hw.optional.avx2_0", AVX2},
+ {"hw.optional.bmi1", BMI1},
+ {"hw.optional.bmi2", BMI2},
+ {"hw.optional.avx512f", AVX512F},
+ {"hw.optional.avx512cd", AVX512CD},
+ {"hw.optional.avx512dq", AVX512DQ},
+ {"hw.optional.avx512bw", AVX512BW},
+ {"hw.optional.avx512vl", AVX512VL},
Review comment:
Also, here are the available values on a M1:
https://gist.github.com/pitrou/fa1527038fea2d7a4cf367ecba7a2d2c
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793697629
Also, can you try under Rosetta?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793015116
👀
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793813011
For 662182a:
Rosetta 2:
```
In [2]: pa.runtime_info()
Out[2]: RuntimeInfo(simd_level='sse4_2', detected_simd_level='sse4_2')
```
Native:
```
In [2]: pa.runtime_info()
Out[2]: RuntimeInfo(simd_level='none', detected_simd_level='none')
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793822986
`simd_level` corresponds to the values allowed in `ARROW_USER_SIMD_LEVEL`. Currently we don't have dynamic dispatch for ARM64 (and I don't think we allow compiling with Neon disabled).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on a change in pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#discussion_r589898691
##########
File path: cpp/src/arrow/util/cpu_info.cc
##########
@@ -330,6 +346,29 @@ void CpuInfo::Init() {
if (QueryPerformanceFrequency(&performance_frequency)) {
max_mhz = static_cast<float>(performance_frequency.QuadPart);
}
+#elif defined(__APPLE__)
+ // On macOS, get CPU information from system information base
+ struct SysCtlCpuFeature {
+ const char* name;
+ int64_t flag;
+ };
+ for (auto feature : std::vector<SysCtlCpuFeature>{
+ {"hw.optional.sse4_2", SSSE3 | SSE4_1 | SSE4_2 | POPCNT},
+ {"hw.optional.avx1_0", AVX},
+ {"hw.optional.avx2_0", AVX2},
+ {"hw.optional.bmi1", BMI1},
+ {"hw.optional.bmi2", BMI2},
+ {"hw.optional.avx512f", AVX512F},
+ {"hw.optional.avx512cd", AVX512CD},
+ {"hw.optional.avx512dq", AVX512DQ},
+ {"hw.optional.avx512bw", AVX512BW},
+ {"hw.optional.avx512vl", AVX512VL},
Review comment:
Also add M1 arm64 flags?
`{"hw.optional.neon", ASIMD}`
I haven't tested this flag myself, just references https://github.com/golang/go/issues/42747#issuecomment-731208964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793029880
Indeed, it doesn't have AVX2. Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on a change in pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#discussion_r590136872
##########
File path: cpp/src/arrow/util/cpu_info.cc
##########
@@ -330,6 +346,29 @@ void CpuInfo::Init() {
if (QueryPerformanceFrequency(&performance_frequency)) {
max_mhz = static_cast<float>(performance_frequency.QuadPart);
}
+#elif defined(__APPLE__)
+ // On macOS, get CPU information from system information base
+ struct SysCtlCpuFeature {
+ const char* name;
+ int64_t flag;
+ };
+ for (auto feature : std::vector<SysCtlCpuFeature>{
+ {"hw.optional.sse4_2", SSSE3 | SSE4_1 | SSE4_2 | POPCNT},
+ {"hw.optional.avx1_0", AVX},
+ {"hw.optional.avx2_0", AVX2},
+ {"hw.optional.bmi1", BMI1},
+ {"hw.optional.bmi2", BMI2},
+ {"hw.optional.avx512f", AVX512F},
+ {"hw.optional.avx512cd", AVX512CD},
+ {"hw.optional.avx512dq", AVX512DQ},
+ {"hw.optional.avx512bw", AVX512BW},
+ {"hw.optional.avx512vl", AVX512VL},
Review comment:
Per my knowledge, `NEON` exists since 32 bit Arm, and `ASIMD` extends it to 64 bit system, with additional instructions and wider registers. `ASIMD` should be the correct term for Arm64 world, but obviously `NEON` is well known and widely used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793696499
Thank you @xhochy .
The joke is that ENOTSUP isn't mentioned in Apple's own [manual page](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysctlbyname.3.html)...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9657: ARROW-11905: [C++] Fix SIMD detection on macOS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9657:
URL: https://github.com/apache/arrow/pull/9657#issuecomment-793021262
https://issues.apache.org/jira/browse/ARROW-11905
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org