You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Shant Hovsepian (Jira)" <ji...@apache.org> on 2020/04/24 21:35:00 UTC
[jira] [Commented] (IMPALA-9690) Bump minimum x86-64 CPU
requirements
[ https://issues.apache.org/jira/browse/IMPALA-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091914#comment-17091914 ]
Shant Hovsepian commented on IMPALA-9690:
-----------------------------------------
In complete support of this for the sake of code simplification. Only caveat is occasionally with virtual machines and cloud machine instances "virtualizing" the SSE4.2 and later instruction set isn't enabled by default, even though the underlying host CPU supports the instruction set.
For software VMs this is usually just a quick config change. For things like Amazon's EC2 it means you must always run with HVM AMIs.
This makes the scenario more common than the actual number of older CPU types out there, but a good error message or disclaimer in the documentation should be more than sufficient.
> Bump minimum x86-64 CPU requirements
> ------------------------------------
>
> Key: IMPALA-9690
> URL: https://issues.apache.org/jira/browse/IMPALA-9690
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: performance
>
> We still have a minimum CPU requirement of SSSE3 support https://impala.apache.org/docs/build/html/topics/impala_prereqs.html. I.e. we don't assume SSE4.2 or AVX or AVX2.
> There is a lot of legacy code to support CPUs without SSE4.2 and various other extensions. As a start, here are all the locations in the code where we branch based on CPU feature:
> {noformat}
> :~/impala/impala$ git grep CpuInfo::IsSupport
> be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) suite32.BENCH(uint32_t, CRC);
> be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
> be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_1)) {
> be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
> be/src/benchmarks/string-compare-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
> be/src/exec/delimited-text-parser.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
> be/src/exec/delimited-text-parser.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
> be/src/exec/delimited-text-parser.inline.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/exec/delimited-text-parser.inline.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSE4_2))) {
> be/src/runtime/io/disk-io-mgr.cc: if (!CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
> be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(CpuInfo::SSSE3)) {
> be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
> be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(cpu_info_flag)) {
> be/src/util/bit-util-test.cc:// CpuInfo::IsSupported() checks. This doesn't test the bug precisely but is a canary for
> be/src/util/bit-util.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
> be/src/util/bit-util.cc: } else if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSSE3))) {
> be/src/util/bit-util.cc: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSSE3))) {
> be/src/util/bit-util.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::POPCNT))) {
> be/src/util/bloom-filter.cc: if (CpuInfo::IsSupported(CpuInfo::AVX)) {
> be/src/util/bloom-filter.h: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
> be/src/util/bloom-filter.h: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
> be/src/util/cpu-info.cc: if (!CpuInfo::IsSupported(CpuInfo::SSSE3)) {
> be/src/util/cpu-info.h: /// // line, CpuInfo::IsSupported(CpuInfo::AVX2) will return false.
> be/src/util/cpu-info.h: : feature_(feature), reenable_(CpuInfo::IsSupported(feature)) {
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
> be/src/util/hash-util.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSE4_2))) {
> be/src/util/openssl-util.cc: return (CpuInfo::IsSupported(CpuInfo::PCLMULQDQ)
> {noformat}
> We also ship two versions of the codegen module, one of which (nosse42) is essentially never used.
> I think it would be uncontroversial to bump the minimum requirement to SSE4.2, which would allow us to delete some old fallbacks. I think the last time Intel or AMD shipped a processor without this was 2010 or 2011. Jumping to AVX is probably almost as uncontroversial, since it looks like that has been universal for nearly as long: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX.
> Some older lower-performance cloud instance types don't support AVX, it looks like, but I think this is an edge case.
> It would be very nice to require AVX2 because that could remove a bunch of conditional code (I think the contributors adding ARM support might want to keep the scalar fallback code though, potentially). It looks like most Intel and AMD processors have supported it since 2013 and 2015 respectively, except some low-end Intel processors: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2.
> In the past we've been a bit conservative about this - people do sometimes use very old hardware as a test cluster because it was the hardware that they could get their hands on (e.g. I remember someone complaining several years back that they couldn't get Impala running on some old AMD CPUs that were missing SSSE3 support).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org