You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Wang, Youwei A" <yo...@intel.com> on 2016/05/12 13:16:06 UTC
Symbol not found issue when working on IMPALA-3504
Greetings All. I am working on IMPALA-3504 right now.
https://issues.cloudera.org/browse/IMPALA-3504
This JIRA basically wants to implement a UDF timestamp function. Following is my process when working on this JIRA:
1. Back-end source code:
I have added an function entry in the file timestamp-functions.h:
static TimestampVal UtcTimestamp(FunctionContext* context);
and added corresponding implementation in the file timestamp-functions.cc.
2. Build back-end:
And after that, I first build impalad binary using make_debug.sh script.
After the impalad binary is generated, I run Linux "strings" command to get the binary signature of the function like: _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE.
And this entry is also added to the Impala/common/function-registry/impala_functions.py file like:
[['utc_timestamp'], 'TIMESTAMP', [], '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'],
3. Overall build:
Then I execute the buildall.sh script to initiate build all front-end, back-end and third-party tools.
Issue:
When the build process is done, I use the command:"start-impala-cluster.py -r --build_type=debug" to start the Impala cluster. However, the impalad process fails to run due to the error which is copied from Impala/logs/cluster/impalad.INFO:
I0512 20:35:46.621750 31842 init.cc:204] Physical Memory: 11.69 GB
I0512 20:35:46.621752 31842 init.cc:205] OS version: Linux version 4.5.3 (root@debian) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Tue May 10 09:19:36 EDT 2016
Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
I0512 20:35:46.621754 31842 init.cc:206] Process ID: 31842
I0512 20:35:47.307698 31842 status.cc:40] Kudu is not supported on this operating system.
@ 0x10de550 impala::Status::Status()
@ 0x10e1984 impala::CheckKuduAvailability()
@ 0x10e18f7 impala::KuduIsAvailable()
@ 0x10d89c1 impala::InitCommonRuntime()
@ 0x1446db1 ImpaladMain()
@ 0x1088720 main
@ 0x7f214e93ab45 (unknown)
@ 0x108859c (unknown)
I0512 20:35:47.689296 31842 hbase-table-scanner.cc:161] Detected HBase version >= 0.95.2
I0512 20:35:47.700866 31842 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I0512 20:35:47.702227 31842 JniFrontend.java:125] Authorization is 'DISABLED'.
I0512 20:35:47.702286 31842 JniFrontend.java:127] Java Version Info: Java(TM) SE Runtime Environment (1.8.0_65-b17)
W0512 20:35:47.871737 31842 HiveConf.java:2721] HiveConf of name hive.access.conf.url does not exist
I0512 20:35:47.958432 31842 fe-support.cc:201] Could not find symbol '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE' in:
Unable to find _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE
dlerror: /root/Impala/be/build/debug/service/impalad: undefined symbol: _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE
F0512 20:35:47.959112 31842 frontend.cc:107] java.lang.RuntimeException: Builtin symbol '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'[] not found!
at com.cloudera.impala.catalog.ScalarFunction.createBuiltin(ScalarFunction.java:79)
at com.cloudera.impala.catalog.Db.addScalarBuiltin(Db.java:389)
at com.cloudera.impala.catalog.Db.addScalarBuiltin(Db.java:379)
at com.cloudera.impala.builtins.ScalarBuiltins.initBuiltins(ScalarBuiltins.java:124)
at com.cloudera.impala.catalog.BuiltinsDb.initBuiltins(BuiltinsDb.java:58)
at com.cloudera.impala.catalog.BuiltinsDb.<init>(BuiltinsDb.java:39)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.ImpaladCatalog.<init>(ImpaladCatalog.java:92)
at com.cloudera.impala.service.Frontend.<init>(Frontend.java:170)
at com.cloudera.impala.service.JniFrontend.<init>(JniFrontend.java:129)
Caused by: com.cloudera.impala.common.AnalysisException: Could not find symbol '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE' in:
at com.cloudera.impala.catalog.Function.lookupSymbol(Function.java:423)
at com.cloudera.impala.catalog.ScalarFunction.createBuiltin(ScalarFunction.java:75)
... 9 more
This is really weird since such symbol "ZN6impala18TimestampFunctions12UtcTimestampEPN10impala" is indeed in the impalad binary and I have double-checked that. Actually I have some successful experience of developing some UDF before. So this issue really makes me feel confused. I have removed all built binaries and replay the steps above. This issue still remains there.
Thank you so much for any advice or hint. :)
RE: Symbol not found issue when working on IMPALA-3504
Posted by "Wang, Youwei A" <yo...@intel.com>.
Hi Tim. It is so great to hear from you.
I have tried these two commands you have provided:
nm -g be/build/debug/service/libfesupport.so | grep ZN6impala18TimestampFunctions12UtcTimestampEPN10impala
nm -g be/build/debug/service/impalad | grep ZN6impala18TimestampFunctions12UtcTimestampEPN10impala
Both of them return NOTHING.
But if I use these two commands, I can get:
strings be/build/debug/service/libfesupport.so | grep ZN6impala18TimestampFunctions12UtcTimestampEPN10impala
_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE
strings be/build/debug/service/impalad | grep ZN6impala18TimestampFunctions12UtcTimestampEPN10impala
_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE
I will take a deeper investigation about this.
As for the clean build, what I have done is:
1. remove all things under Impala/be/build directory;
2. call make clean; call Impala/bin/clean.sh
3. call Impala/buildall.sh
Not sure step 1 and 2 can make the environment clean enough now.
Thank you so much for your help. I think I have a clue now. :)
-----Original Message-----
From: Tim Armstrong [mailto:tarmstrong@cloudera.com]
Sent: Friday, May 13, 2016 4:11 AM
To: dev@impala.incubator.apache.org
Subject: Re: Symbol not found issue when working on IMPALA-3504
Hi Youwei,
Your steps seem right. One thing is that it might be libfesupport.so that it is trying to resolve the binary in:
nm -g be/build/debug/service/libfesupport.so
nm -g be/build/debug/service/impalad
Make something just wasn't rebuilt. Did you try doing a clean buidl?
- Tim
On Thu, May 12, 2016 at 6:16 AM, Wang, Youwei A <yo...@intel.com>
wrote:
> Greetings All. I am working on IMPALA-3504 right now.
> https://issues.cloudera.org/browse/IMPALA-3504
> This JIRA basically wants to implement a UDF timestamp function.
> Following is my process when working on this JIRA:
>
> 1. Back-end source code:
> I have added an function entry in the file timestamp-functions.h:
> static TimestampVal UtcTimestamp(FunctionContext* context); and added
> corresponding implementation in the file timestamp-functions.cc.
>
> 2. Build back-end:
> And after that, I first build impalad binary using make_debug.sh script.
> After the impalad binary is generated, I run Linux "strings" command
> to get the binary signature of the function like:
> _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE.
> And this entry is also added to the
> Impala/common/function-registry/impala_functions.py file like:
> [['utc_timestamp'], 'TIMESTAMP', [],
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15Function
> ContextE'],
>
> 3. Overall build:
> Then I execute the buildall.sh script to initiate build all front-end,
> back-end and third-party tools.
>
> Issue:
> When the build process is done, I use the
> command:"start-impala-cluster.py -r --build_type=debug" to start the
> Impala cluster. However, the impalad process fails to run due to the
> error which is copied from Impala/logs/cluster/impalad.INFO:
>
> I0512 20:35:46.621750 31842 init.cc:204] Physical Memory: 11.69 GB
> I0512 20:35:46.621752 31842 init.cc:205] OS version: Linux version
> 4.5.3
> (root@debian) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Tue May 10
> 09:19:36 EDT 2016
> Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
> I0512 20:35:46.621754 31842 init.cc:206] Process ID: 31842
> I0512 20:35:47.307698 31842 status.cc:40] Kudu is not supported on
> this operating system.
> @ 0x10de550 impala::Status::Status()
> @ 0x10e1984 impala::CheckKuduAvailability()
> @ 0x10e18f7 impala::KuduIsAvailable()
> @ 0x10d89c1 impala::InitCommonRuntime()
> @ 0x1446db1 ImpaladMain()
> @ 0x1088720 main
> @ 0x7f214e93ab45 (unknown)
> @ 0x108859c (unknown)
> I0512 20:35:47.689296 31842 hbase-table-scanner.cc:161] Detected HBase
> version >= 0.95.2
> I0512 20:35:47.700866 31842 GlogAppender.java:123] Logging initialized.
> Impala: VLOG, All other: INFO
> I0512 20:35:47.702227 31842 JniFrontend.java:125] Authorization is
> 'DISABLED'.
> I0512 20:35:47.702286 31842 JniFrontend.java:127] Java Version Info:
> Java(TM) SE Runtime Environment (1.8.0_65-b17)
> W0512 20:35:47.871737 31842 HiveConf.java:2721] HiveConf of name
> hive.access.conf.url does not exist
> I0512 20:35:47.958432 31842 fe-support.cc:201] Could not find symbol
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'
> in:
> Unable to find
> _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionC
> ontextE
> dlerror: /root/Impala/be/build/debug/service/impalad: undefined symbol:
> _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionC
> ontextE
> F0512 20:35:47.959112 31842 frontend.cc:107] java.lang.RuntimeException:
> Builtin symbol
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15Function
> ContextE'[]
> not found!
> at
> com.cloudera.impala.catalog.ScalarFunction.createBuiltin(ScalarFunction.java:79)
> at com.cloudera.impala.catalog.Db.addScalarBuiltin(Db.java:389)
> at com.cloudera.impala.catalog.Db.addScalarBuiltin(Db.java:379)
> at
> com.cloudera.impala.builtins.ScalarBuiltins.initBuiltins(ScalarBuiltins.java:124)
> at
> com.cloudera.impala.catalog.BuiltinsDb.initBuiltins(BuiltinsDb.java:58)
> at
> com.cloudera.impala.catalog.BuiltinsDb.<init>(BuiltinsDb.java:39)
> at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
> at
> com.cloudera.impala.catalog.ImpaladCatalog.<init>(ImpaladCatalog.java:92)
> at
> com.cloudera.impala.service.Frontend.<init>(Frontend.java:170)
> at
> com.cloudera.impala.service.JniFrontend.<init>(JniFrontend.java:129)
> Caused by: com.cloudera.impala.common.AnalysisException: Could not
> find symbol
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'
> in:
> at
> com.cloudera.impala.catalog.Function.lookupSymbol(Function.java:423)
> at
> com.cloudera.impala.catalog.ScalarFunction.createBuiltin(ScalarFunction.java:75)
> ... 9 more
>
> This is really weird since such symbol
> "ZN6impala18TimestampFunctions12UtcTimestampEPN10impala" is indeed in
> the impalad binary and I have double-checked that. Actually I have
> some successful experience of developing some UDF before. So this
> issue really makes me feel confused. I have removed all built binaries
> and replay the steps above. This issue still remains there.
>
> Thank you so much for any advice or hint. :)
>
>
>
Re: Symbol not found issue when working on IMPALA-3504
Posted by Tim Armstrong <ta...@cloudera.com>.
Hi Youwei,
Your steps seem right. One thing is that it might be libfesupport.so that
it is trying to resolve the binary in:
nm -g be/build/debug/service/libfesupport.so
nm -g be/build/debug/service/impalad
Make something just wasn't rebuilt. Did you try doing a clean buidl?
- Tim
On Thu, May 12, 2016 at 6:16 AM, Wang, Youwei A <yo...@intel.com>
wrote:
> Greetings All. I am working on IMPALA-3504 right now.
> https://issues.cloudera.org/browse/IMPALA-3504
> This JIRA basically wants to implement a UDF timestamp function. Following
> is my process when working on this JIRA:
>
> 1. Back-end source code:
> I have added an function entry in the file timestamp-functions.h:
> static TimestampVal UtcTimestamp(FunctionContext* context);
> and added corresponding implementation in the file timestamp-functions.cc.
>
> 2. Build back-end:
> And after that, I first build impalad binary using make_debug.sh script.
> After the impalad binary is generated, I run Linux "strings" command to
> get the binary signature of the function like:
> _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE.
> And this entry is also added to the
> Impala/common/function-registry/impala_functions.py file like:
> [['utc_timestamp'], 'TIMESTAMP', [],
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'],
>
> 3. Overall build:
> Then I execute the buildall.sh script to initiate build all front-end,
> back-end and third-party tools.
>
> Issue:
> When the build process is done, I use the
> command:"start-impala-cluster.py -r --build_type=debug" to start the
> Impala cluster. However, the impalad process fails to run due to the error
> which is copied from Impala/logs/cluster/impalad.INFO:
>
> I0512 20:35:46.621750 31842 init.cc:204] Physical Memory: 11.69 GB
> I0512 20:35:46.621752 31842 init.cc:205] OS version: Linux version 4.5.3
> (root@debian) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Tue May 10
> 09:19:36 EDT 2016
> Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
> I0512 20:35:46.621754 31842 init.cc:206] Process ID: 31842
> I0512 20:35:47.307698 31842 status.cc:40] Kudu is not supported on this
> operating system.
> @ 0x10de550 impala::Status::Status()
> @ 0x10e1984 impala::CheckKuduAvailability()
> @ 0x10e18f7 impala::KuduIsAvailable()
> @ 0x10d89c1 impala::InitCommonRuntime()
> @ 0x1446db1 ImpaladMain()
> @ 0x1088720 main
> @ 0x7f214e93ab45 (unknown)
> @ 0x108859c (unknown)
> I0512 20:35:47.689296 31842 hbase-table-scanner.cc:161] Detected HBase
> version >= 0.95.2
> I0512 20:35:47.700866 31842 GlogAppender.java:123] Logging initialized.
> Impala: VLOG, All other: INFO
> I0512 20:35:47.702227 31842 JniFrontend.java:125] Authorization is
> 'DISABLED'.
> I0512 20:35:47.702286 31842 JniFrontend.java:127] Java Version Info:
> Java(TM) SE Runtime Environment (1.8.0_65-b17)
> W0512 20:35:47.871737 31842 HiveConf.java:2721] HiveConf of name
> hive.access.conf.url does not exist
> I0512 20:35:47.958432 31842 fe-support.cc:201] Could not find symbol
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'
> in:
> Unable to find
> _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE
> dlerror: /root/Impala/be/build/debug/service/impalad: undefined symbol:
> _ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE
> F0512 20:35:47.959112 31842 frontend.cc:107] java.lang.RuntimeException:
> Builtin symbol
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'[]
> not found!
> at
> com.cloudera.impala.catalog.ScalarFunction.createBuiltin(ScalarFunction.java:79)
> at com.cloudera.impala.catalog.Db.addScalarBuiltin(Db.java:389)
> at com.cloudera.impala.catalog.Db.addScalarBuiltin(Db.java:379)
> at
> com.cloudera.impala.builtins.ScalarBuiltins.initBuiltins(ScalarBuiltins.java:124)
> at
> com.cloudera.impala.catalog.BuiltinsDb.initBuiltins(BuiltinsDb.java:58)
> at
> com.cloudera.impala.catalog.BuiltinsDb.<init>(BuiltinsDb.java:39)
> at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
> at
> com.cloudera.impala.catalog.ImpaladCatalog.<init>(ImpaladCatalog.java:92)
> at
> com.cloudera.impala.service.Frontend.<init>(Frontend.java:170)
> at
> com.cloudera.impala.service.JniFrontend.<init>(JniFrontend.java:129)
> Caused by: com.cloudera.impala.common.AnalysisException: Could not find
> symbol
> '_ZN6impala18TimestampFunctions12UtcTimestampEPN10impala_udf15FunctionContextE'
> in:
> at
> com.cloudera.impala.catalog.Function.lookupSymbol(Function.java:423)
> at
> com.cloudera.impala.catalog.ScalarFunction.createBuiltin(ScalarFunction.java:75)
> ... 9 more
>
> This is really weird since such symbol
> "ZN6impala18TimestampFunctions12UtcTimestampEPN10impala" is indeed in the
> impalad binary and I have double-checked that. Actually I have some
> successful experience of developing some UDF before. So this issue really
> makes me feel confused. I have removed all built binaries and replay the
> steps above. This issue still remains there.
>
> Thank you so much for any advice or hint. :)
>
>
>