You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/11/29 00:49:59 UTC
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20737
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
IMPALA-11157: Use native-toolchain hadoop for aarch64
Uses hadoop binaries from native-toolchain for aarch64 rather than
pre-built binaries from an unknown source hosted in GitHub.
TODO: update with final native-toolchain build. Current built is from
https://gerrit.cloudera.org/c/20724/.
Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 14 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/20737/1
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 2:
(2 comments)
I've tried running a core-mode test build with this change, but dataload seems to be quite flaky:
- https://jenkins.impala.io/view/Ubuntu%2020/job/ubuntu-20.04-from-scratch-ARM/62/ crashed with symptoms indicating an Impala crash (Transport endpoint is not connected):
00:52:56.956 ERROR: INSERT OVERWRITE TABLE functional_parquet.alltypessmall partition (year, month)
00:52:56.956 SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col, year, month
00:52:56.956 FROM functional.alltypessmall
00:52:56.956 Traceback (most recent call last):
00:52:56.956 File "/home/ubuntu/Impala/bin/load-data.py", line 189, in exec_impala_query_from_file
00:52:56.956 result = impala_client.execute(query)
00:52:56.956 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 191, in execute
00:52:56.956 handle = self.__execute_query(query_string.strip(), user=user)
00:52:56.956 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 369, in __execute_query
00:52:56.956 self.wait_for_finished(handle)
00:52:56.956 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 390, in wait_for_finished
00:52:56.956 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
00:52:56.956 ImpalaBeeswaxException: ImpalaBeeswaxException:
00:52:56.956 Query aborted:Exec() rpc failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Another build on a private instance (similar size with Ubuntu 20.04) crashed in a different location during dataload, leaving a number of JVM crash reports in /var/crash, and several hs_err_pidNNN.logs in the Impala working directory.
(These are pretty large; I'm happy to send them to you directly if you think they would be helpful).
My primary suspicion is the version mismatch between CDP_HADOOP_VERSION and IMPALA_HADOOP_CLIENT_BINARY_VERSION, see the comments.
http://gerrit.cloudera.org:8080/#/c/20737/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20737/2/bin/impala-config.sh@207
PS2, Line 207: 3.3.6
I'm afraid this version is too new, and much too different from the regular Hadoop version, see below
http://gerrit.cloudera.org:8080/#/c/20737/2/bin/impala-config.sh@253
PS2, Line 253: 3.1.1.7.2.18.0-369
I'm afraid this version of Hadoop is not really compatible with the significantly newer 3.3.6 that's specified for the ARM native libraries above.
3.1.1 was released 5 years ago; 3.3.6 is quite fresh, and it has recently received a full upgrade to the AWS Java SDK v2, among other changes.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2023 17:58:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3:
> Patch Set 3:
>
> (1 comment)
>
> https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/59/ was my test run for this patch.
This turned out to be a false alarm after all, my apologies for the turbulence. The real culprit was indeed a debug build paired with the lack of IMPALA-11542.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Dec 2023 18:40:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14552/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Nov 2023 23:10:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2023 18:39:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3:
Checking the build logs yielded an impalad minidump captured during dataload, which suggests that the crash is probably unrelated to the native Hadoop libs.
This is the abbreviated backtrace (with registers omitted):
Crash reason: SIGABRT
Crash address: 0x3e80000fdc5
Process uptime: not available
Thread 634 (crashed)
0 libc.so.6!gsignal + 0xe0
Found by: given as instruction pointer in context
1 libc.so.6!abort + 0x110
Found by: call frame info
2 libc.so.6!uselocale + 0x324
Found by: call frame info
3 libc.so.6!__assert_fail + 0x48
Found by: call frame info
4 impalad!llvm::RuntimeDyldELF::resolveAArch64Relocation(llvm::SectionEntry const&, unsigned long, unsigned long, unsigned int, long) [RuntimeDyldELF.cpp : 400 + 0x20]
Found by: call frame info
5 impalad!llvm::RuntimeDyldImpl::resolveRelocationList(llvm::SmallVector<llvm::RelocationEntry, 64u> const&, unsigned long) [RuntimeDyld.cpp : 957 + 0xc]
Found by: call frame info
6 impalad!llvm::RuntimeDyldImpl::resolveRelocations() [RuntimeDyld.cpp : 145 + 0xc]
Found by: call frame info
7 impalad!llvm::MCJIT::finalizeLoadedModules() [MCJIT.cpp : 237 + 0x4]
Found by: call frame info
8 impalad!llvm::MCJIT::finalizeObject() [MCJIT.cpp : 261 + 0x4]
Found by: call frame info
9 impalad!impala::LlvmCodeGen::FinalizeModule() [llvm-codegen.cc : 1361 + 0x0]
Found by: call frame info
10 impalad!impala::FragmentState::CodegenHelper(impala::RuntimeProfile::EventSequence*) [fragment-state.cc : 178 + 0x4]
Found by: call frame info
11 impalad!impala::FragmentState::InvokeCodegen(impala::RuntimeProfile::EventSequence*) [fragment-state.cc : 139 + 0xc]
Found by: call frame info
12 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc : 417 + 0xc]
Found by: call frame info
13 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc : 95 + 0x8]
Found by: call frame info
14 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) [query-state.cc : 967 + 0x8]
Found by: call frame info
15 impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke [query-state.cc : 872 + 0x0]
Found by: call frame info
16 impalad!impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) [function_template.hpp : 763 + 0x8]
It suggests IMPALA-11542 to me, for which the patch has just been merged. I'll re-test with this patch rebased on top of it.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2023 20:38:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2023 18:40:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3:
(1 comment)
https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/59/ was my test run for this patch.
http://gerrit.cloudera.org:8080/#/c/20737/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20737/2/bin/impala-config.sh@253
PS2, Line 253: 3.1.1.7.2.18.0-369
> I'm afraid this version of Hadoop is not really compatible with the signifi
I wouldn't expect that to impact the C APIs. Do you have a version you'd recommend using?
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2023 18:49:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
IMPALA-11157: Use native-toolchain hadoop for aarch64
Uses hadoop binaries from native-toolchain for aarch64 rather than
pre-built binaries from an individual's GitHub repo.
Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Reviewed-on: http://gerrit.cloudera.org:8080/20737
Reviewed-by: Laszlo Gaal <la...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 15 insertions(+), 11 deletions(-)
Approvals:
Laszlo Gaal: Looks good to me, approved
Michael Smith: Verified
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Removed Verified+1 by Michael Smith <mi...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20737
to look at the new patch set (#2).
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
IMPALA-11157: Use native-toolchain hadoop for aarch64
Uses hadoop binaries from native-toolchain for aarch64 rather than
pre-built binaries from an individual's GitHub repo.
Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
---
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
3 files changed, 15 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/20737/2
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3:
https://jenkins.impala.io/view/Ubuntu%2020/job/ubuntu-20.04-from-scratch-ARM/64/ is running on the rebase
It passed dataload on my local build, so I'm hopeful.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2023 21:48:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 3: Code-Review+2
The rebased code passed build, dataload, FE_TEST and BE_TEST; there were two test failures.
The CUSTOM_CLUSTER test failed with the same symptom that was observed in post-commit tests on x86, while the EE_TEST failure looked more like flakiness.
Running the build again with only the failing EE_TEST step and the CUSTOM_CLUSTER suite passed, so I'm chalking up both of the above failures to flakiness.
This means that the Hadoop native libs sourced from the toolchain are very likely good enough for Impala on ARM.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Dec 2023 18:37:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2023 22:54:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9975/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2023 18:27:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 2:
> Patch Set 2:
>
> (2 comments)
>
> I've tried running a core-mode test build with this change, but dataload seems to be quite flaky:
> - https://jenkins.impala.io/view/Ubuntu%2020/job/ubuntu-20.04-from-scratch-ARM/62/ crashed with symptoms indicating an Impala crash (Transport endpoint is not connected):
>
> 00:52:56.956 ERROR: INSERT OVERWRITE TABLE functional_parquet.alltypessmall partition (year, month)
> 00:52:56.956 SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col, year, month
> 00:52:56.956 FROM functional.alltypessmall
> 00:52:56.956 Traceback (most recent call last):
> 00:52:56.956 File "/home/ubuntu/Impala/bin/load-data.py", line 189, in exec_impala_query_from_file
> 00:52:56.956 result = impala_client.execute(query)
> 00:52:56.956 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 191, in execute
> 00:52:56.956 handle = self.__execute_query(query_string.strip(), user=user)
> 00:52:56.956 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 369, in __execute_query
> 00:52:56.956 self.wait_for_finished(handle)
> 00:52:56.956 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 390, in wait_for_finished
> 00:52:56.956 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> 00:52:56.956 ImpalaBeeswaxException: ImpalaBeeswaxException:
> 00:52:56.956 Query aborted:Exec() rpc failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
>
> Another build on a private instance (similar size with Ubuntu 20.04) crashed in a different location during dataload, leaving a number of JVM crash reports in /var/crash, and several hs_err_pidNNN.logs in the Impala working directory.
> (These are pretty large; I'm happy to send them to you directly if you think they would be helpful).
>
> My primary suspicion is the version mismatch between CDP_HADOOP_VERSION and IMPALA_HADOOP_CLIENT_BINARY_VERSION, see the comments.
DEBUG builds on ARM are still flaky until https://gerrit.cloudera.org/c/20692/ is merged.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2023 18:00:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11157: Use native-toolchain hadoop for aarch64
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20737 )
Change subject: IMPALA-11157: Use native-toolchain hadoop for aarch64
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14539/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20737
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ide5ad327d6ce7c2a6b7d0ec4cf1dd53fef987720
Gerrit-Change-Number: 20737
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Nov 2023 01:18:57 +0000
Gerrit-HasComments: No