You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Pradeep Nayak <pr...@gmail.com> on 2016/09/12 20:06:31 UTC
How to install the custom built impala
This question may sound noob, but apologies in advance.
I have changes in the impala source code (both backend and frontend
components). How do I go about installing this version of Impala on my
cloudera test cluster ? I know the path where I can replace the front end
built jar file, but not sure of the backend parts.
Regards,
Pradeep Nayak
http://pradeepnayak.in
http://twitter.com/_prdp
Re: How to install the custom built impala
Posted by Jim Apple <jb...@cloudera.com>.
Also, questions like this are welcomed on this list.
On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
wrote:
> I don't think this is documented anywhere. You would need to replace the
> impalad binary and libfesupport.so under sbin-retail or sbin-debug for the
> release or debug builds respectively. You also need to replace
> impala-frontend-*.jar and impala-data-source-api*.jar.
>
> You also need to make sure the statestored and catalogd symlinks point to
> the impalad binary,
>
> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
> > This question may sound noob, but apologies in advance.
> >
> > I have changes in the impala source code (both backend and frontend
> > components). How do I go about installing this version of Impala on my
> > cloudera test cluster ? I know the path where I can replace the front end
> > built jar file, but not sure of the backend parts.
> >
> > Regards,
> > Pradeep Nayak
> >
> > http://pradeepnayak.in
> > http://twitter.com/_prdp
> >
>
Re: How to install the custom built impala
Posted by Jim Apple <jb...@cloudera.com>.
What have you tried? Did you try building with the old Cloudera Impala
repo at https://github.com/cloudera/impala/tree/cdh5-2.3.0_5.5.4?
On Wed, Oct 5, 2016 at 10:48 AM, Pradeep Nayak <pr...@gmail.com> wrote:
> Hi - I have verified my changes in the dev cluster which I had. However the
> test/staging cluster is at 5.5.4. Is there a quick way to build it for that
> version. ?
>
> Cheers!
> Pradeep
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>
> On Tue, Sep 13, 2016 at 4:54 PM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> It looks like you built the debug Impala with dynamic linking
>> (-build_shared_libs or -so) whereas the CDH build uses static linking -
>> that might explain why the release build worked ok.
>>
>> It's expected that your binary is much larger, in the CDH packages the
>> debug symbols were stripped out and put in a separate impalad.debug file,
>> whereas just building a dev environment doesn't do that step. Should work
>> fine though.
>>
>> On Tue, Sep 13, 2016 at 4:44 PM, Pradeep Nayak <pr...@gmail.com>
>> wrote:
>>
>>> Well when I built the debug one and replaced it my cluster it seemed to
>>> have more dependencies which it was not able to find;
>>>
>>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>>> 5.8.0.p0.42/lib/impala/sbin-debug$ ldd impalad
>>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
>>> `GLIBCXX_3.4.20' not found (required by ./impalad)
>>> linux-vdso.so.1 => (0x00007ffd87dc6000)
>>> libjsig.so => not found
>>> libTestUtil.so => not found
>>> libUtil.so => not found
>>> libRuntime.so => not found
>>> libExec.so => not found
>>> libCodeGen.so => not found
>>> libExprs.so => not found
>>> libRpc.so => not found
>>> libService.so => not found
>>> libStatestore.so => not found
>>> libCatalog.so => not found
>>> libResourceBroker.so => not found
>>> libImpalaThrift.so => not found
>>> libGlobalFlags.so => not found
>>> libCommon.so => not found
>>> libUdf.so => not found
>>> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
>>> (0x00007fd4661be000)
>>> libThriftSaslTransport.so => not found
>>> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> (0x00007fd465f5f000)
>>> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>> (0x00007fd465b83000)
>>> libgutil.so => not found
>>> libhdfs.so.0.0.0 => not found
>>> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000)
>>> libjvm.so => not found
>>> libkudu_client.so.0 => not found
>>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
>>> (0x00007fd465762000)
>>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
>>> (0x00007fd46555e000)
>>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> (0x00007fd46525a000)
>>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000)
>>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> (0x00007fd464d3e000)
>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>>> (0x00007fd464b20000)
>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000)
>>> /lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000)
>>>
>>>
>>>
>>> I went ahead and build the retail one and that was OKAY.
>>>
>>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>>> 5.8.0.p0.42/lib/impala/sbin-retail$ ldd impalad
>>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
>>> `CXXABI_1.3.8' not found (required by ./impalad)
>>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
>>> `GLIBCXX_3.4.20' not found (required by ./impalad)
>>> linux-vdso.so.1 => (0x00007fffc2679000)
>>> libjsig.so => not found
>>> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
>>> (0x00007f153629d000)
>>> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> (0x00007f153603e000)
>>> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>> (0x00007f1535c62000)
>>> libjvm.so => not found
>>> libkudu_client.so.0 => not found
>>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
>>> (0x00007f1535a5a000)
>>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
>>> (0x00007f1535856000)
>>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> (0x00007f1535552000)
>>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000)
>>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> (0x00007f1535036000)
>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>>> (0x00007f1534e18000)
>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000)
>>> /lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000)
>>>
>>>
>>> However I see the sizes being different. The one which came with cloudera
>>> parcels was of 37M and one which I built was 384MB!! Why is that ?
>>>
>>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh impalad
>>> 348M impalad
>>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/impala.retail.orig
>>> 37M /home/ubuntu/impala.retail.orig
>>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh libfesupport.so
>>> 346M libfesupport.so
>>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/libfesupport.so.retail.orig
>>> 39M /home/ubuntu/libfesupport.so.retail.orig
>>>
>>>
>>> To conclude I am able to verify the changes which I made in the retail
>>> build but not with the debug build. We can probably document what needs to
>>> be replaced to test changes in a real cluster with someone's custom impala
>>> binaries.
>>>
>>> 1. impalad in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impa
>>> la/sbin-retail
>>> 2. libfesupport.so in /opt/cloudera/parcels/CDH-5
>>> .8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail
>>> 3. impala-frontend-0.1-SNAPSHOT.jar in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/.
>>> The impala-frontend-0.1-SNAPSHOT.jar file in
>>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/
>>> symlinks to the one in /opt/cloudera/parcels/CDH-5.8.
>>> 0-1.cdh5.8.0.p0.42/jars/.
>>>
>>> This is all which is need to test your changes.
>>>
>>> Regards,
>>> Pradeep Nayak
>>>
>>> http://pradeepnayak.in
>>> http://twitter.com/_prdp
>>>
>>> On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <pr...@gmail.com>
>>> wrote:
>>>
>>>> Okay thanks Tim. I tried that.
>>>>
>>>> I am seeing the below error, once I repalced the impalad and
>>>> libfesupport.so. Same errors for statestore and catalogd.
>>>>
>>>> + true
>>>> + export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
>>>> + IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
>>>> + [[ true = '' ]]
>>>> + '[' impalad = impalad ']'
>>>> + exec /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags
>>>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: error while loading shared libraries: libTestUtil.so: cannot open shared object file: No such file or directory
>>>>
>>>>
>>>> Regards,
>>>> Pradeep Nayak
>>>>
>>>> http://pradeepnayak.in
>>>> http://twitter.com/_prdp
>>>>
>>>> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <ta...@cloudera.com>
>>>> wrote:
>>>>
>>>>> If the statestored and catalogd symlinks point to the new impalad
>>>>> binary then you're good.
>>>>>
>>>>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> My confusion should I update the bin location or the sbin location
>>>>>> here : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd
>>>>>> and statestored appear to be symlinks to impalad.
>>>>>>
>>>>>> Regards,
>>>>>> Pradeep Nayak
>>>>>>
>>>>>> http://pradeepnayak.in
>>>>>> http://twitter.com/_prdp
>>>>>>
>>>>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5
>>>>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built.
>>>>>>> How about catalogd and statestored ?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Pradeep Nayak
>>>>>>>
>>>>>>> http://pradeepnayak.in
>>>>>>> http://twitter.com/_prdp
>>>>>>>
>>>>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>>>>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>>>>>>>
>>>>>>>> If you go through this process again with a build after that change
>>>>>>>> goes in, then you'll need to update that jar.
>>>>>>>>
>>>>>>>> You can build it by running mvn package in Impala/ext-data-source.
>>>>>>>>
>>>>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <
>>>>>>>> tarmstrong@cloudera.com> wrote:
>>>>>>>> > It looks like impala-data-source* may not be built in all case. It
>>>>>>>> doesn't
>>>>>>>> > really change much if at all so I doubt you will need to update it
>>>>>>>> anyway.
>>>>>>>> >
>>>>>>>> > P.S. I agree with Jim, this is exactly the kind of question this
>>>>>>>> list is
>>>>>>>> > for.
>>>>>>>> >
>>>>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <
>>>>>>>> pradeep1288@gmail.com>
>>>>>>>> > wrote:
>>>>>>>> >
>>>>>>>> >> So here is what I understand:
>>>>>>>> >>
>>>>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>>>>>>>> >> build/debug/service
>>>>>>>> >>
>>>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>>>>>>>> >> total 330928
>>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25
>>>>>>>> query-options-test
>>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25
>>>>>>>> session-expiry-test
>>>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>>>>>>> >>
>>>>>>>> >> and replace it with one on my test cluster for backend part.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> For the frontend, i need to pick the below file:
>>>>>>>> >>
>>>>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> I couldn't find the impala-data-source-api*.jar file
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Regards,
>>>>>>>> >> Pradeep Nayak
>>>>>>>> >>
>>>>>>>> >> http://pradeepnayak.in
>>>>>>>> >> http://twitter.com/_prdp
>>>>>>>> >>
>>>>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>>>>>>>> tarmstrong@cloudera.com>
>>>>>>>> >> wrote:
>>>>>>>> >>
>>>>>>>> >>> I don't think this is documented anywhere. You would need to
>>>>>>>> replace the
>>>>>>>> >>> impalad binary and libfesupport.so under sbin-retail or
>>>>>>>> sbin-debug for the
>>>>>>>> >>> release or debug builds respectively. You also need to replace
>>>>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>>>>>>> >>>
>>>>>>>> >>> You also need to make sure the statestored and catalogd symlinks
>>>>>>>> point to
>>>>>>>> >>> the impalad binary,
>>>>>>>> >>>
>>>>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <
>>>>>>>> pradeep1288@gmail.com>
>>>>>>>> >>> wrote:
>>>>>>>> >>>
>>>>>>>> >>>> This question may sound noob, but apologies in advance.
>>>>>>>> >>>>
>>>>>>>> >>>> I have changes in the impala source code (both backend and
>>>>>>>> frontend
>>>>>>>> >>>> components). How do I go about installing this version of
>>>>>>>> Impala on my
>>>>>>>> >>>> cloudera test cluster ? I know the path where I can replace the
>>>>>>>> front end
>>>>>>>> >>>> built jar file, but not sure of the backend parts.
>>>>>>>> >>>>
>>>>>>>> >>>> Regards,
>>>>>>>> >>>> Pradeep Nayak
>>>>>>>> >>>>
>>>>>>>> >>>> http://pradeepnayak.in
>>>>>>>> >>>> http://twitter.com/_prdp
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: How to install the custom built impala
Posted by Pradeep Nayak <pr...@gmail.com>.
Hi - I have verified my changes in the dev cluster which I had. However the
test/staging cluster is at 5.5.4. Is there a quick way to build it for that
version. ?
Cheers!
Pradeep
http://pradeepnayak.in
http://twitter.com/_prdp
On Tue, Sep 13, 2016 at 4:54 PM, Tim Armstrong <ta...@cloudera.com>
wrote:
> It looks like you built the debug Impala with dynamic linking
> (-build_shared_libs or -so) whereas the CDH build uses static linking -
> that might explain why the release build worked ok.
>
> It's expected that your binary is much larger, in the CDH packages the
> debug symbols were stripped out and put in a separate impalad.debug file,
> whereas just building a dev environment doesn't do that step. Should work
> fine though.
>
> On Tue, Sep 13, 2016 at 4:44 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
>> Well when I built the debug one and replaced it my cluster it seemed to
>> have more dependencies which it was not able to find;
>>
>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>> 5.8.0.p0.42/lib/impala/sbin-debug$ ldd impalad
>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
>> `GLIBCXX_3.4.20' not found (required by ./impalad)
>> linux-vdso.so.1 => (0x00007ffd87dc6000)
>> libjsig.so => not found
>> libTestUtil.so => not found
>> libUtil.so => not found
>> libRuntime.so => not found
>> libExec.so => not found
>> libCodeGen.so => not found
>> libExprs.so => not found
>> libRpc.so => not found
>> libService.so => not found
>> libStatestore.so => not found
>> libCatalog.so => not found
>> libResourceBroker.so => not found
>> libImpalaThrift.so => not found
>> libGlobalFlags.so => not found
>> libCommon.so => not found
>> libUdf.so => not found
>> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
>> (0x00007fd4661be000)
>> libThriftSaslTransport.so => not found
>> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> (0x00007fd465f5f000)
>> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> (0x00007fd465b83000)
>> libgutil.so => not found
>> libhdfs.so.0.0.0 => not found
>> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000)
>> libjvm.so => not found
>> libkudu_client.so.0 => not found
>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
>> (0x00007fd465762000)
>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
>> (0x00007fd46555e000)
>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> (0x00007fd46525a000)
>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000)
>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
>> (0x00007fd464d3e000)
>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x00007fd464b20000)
>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000)
>>
>>
>>
>> I went ahead and build the retail one and that was OKAY.
>>
>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>> 5.8.0.p0.42/lib/impala/sbin-retail$ ldd impalad
>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
>> `CXXABI_1.3.8' not found (required by ./impalad)
>> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
>> `GLIBCXX_3.4.20' not found (required by ./impalad)
>> linux-vdso.so.1 => (0x00007fffc2679000)
>> libjsig.so => not found
>> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
>> (0x00007f153629d000)
>> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> (0x00007f153603e000)
>> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> (0x00007f1535c62000)
>> libjvm.so => not found
>> libkudu_client.so.0 => not found
>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
>> (0x00007f1535a5a000)
>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
>> (0x00007f1535856000)
>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> (0x00007f1535552000)
>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000)
>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
>> (0x00007f1535036000)
>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x00007f1534e18000)
>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000)
>>
>>
>> However I see the sizes being different. The one which came with cloudera
>> parcels was of 37M and one which I built was 384MB!! Why is that ?
>>
>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh impalad
>> 348M impalad
>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/impala.retail.orig
>> 37M /home/ubuntu/impala.retail.orig
>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh libfesupport.so
>> 346M libfesupport.so
>> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh
>> 5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/libfesupport.so.retail.orig
>> 39M /home/ubuntu/libfesupport.so.retail.orig
>>
>>
>> To conclude I am able to verify the changes which I made in the retail
>> build but not with the debug build. We can probably document what needs to
>> be replaced to test changes in a real cluster with someone's custom impala
>> binaries.
>>
>> 1. impalad in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impa
>> la/sbin-retail
>> 2. libfesupport.so in /opt/cloudera/parcels/CDH-5
>> .8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail
>> 3. impala-frontend-0.1-SNAPSHOT.jar in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/.
>> The impala-frontend-0.1-SNAPSHOT.jar file in
>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/
>> symlinks to the one in /opt/cloudera/parcels/CDH-5.8.
>> 0-1.cdh5.8.0.p0.42/jars/.
>>
>> This is all which is need to test your changes.
>>
>> Regards,
>> Pradeep Nayak
>>
>> http://pradeepnayak.in
>> http://twitter.com/_prdp
>>
>> On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <pr...@gmail.com>
>> wrote:
>>
>>> Okay thanks Tim. I tried that.
>>>
>>> I am seeing the below error, once I repalced the impalad and
>>> libfesupport.so. Same errors for statestore and catalogd.
>>>
>>> + true
>>> + export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
>>> + IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
>>> + [[ true = '' ]]
>>> + '[' impalad = impalad ']'
>>> + exec /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags
>>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: error while loading shared libraries: libTestUtil.so: cannot open shared object file: No such file or directory
>>>
>>>
>>> Regards,
>>> Pradeep Nayak
>>>
>>> http://pradeepnayak.in
>>> http://twitter.com/_prdp
>>>
>>> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <ta...@cloudera.com>
>>> wrote:
>>>
>>>> If the statestored and catalogd symlinks point to the new impalad
>>>> binary then you're good.
>>>>
>>>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> My confusion should I update the bin location or the sbin location
>>>>> here : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd
>>>>> and statestored appear to be symlinks to impalad.
>>>>>
>>>>> Regards,
>>>>> Pradeep Nayak
>>>>>
>>>>> http://pradeepnayak.in
>>>>> http://twitter.com/_prdp
>>>>>
>>>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5
>>>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built.
>>>>>> How about catalogd and statestored ?
>>>>>>
>>>>>> Regards,
>>>>>> Pradeep Nayak
>>>>>>
>>>>>> http://pradeepnayak.in
>>>>>> http://twitter.com/_prdp
>>>>>>
>>>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>>>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>>>>>>
>>>>>>> If you go through this process again with a build after that change
>>>>>>> goes in, then you'll need to update that jar.
>>>>>>>
>>>>>>> You can build it by running mvn package in Impala/ext-data-source.
>>>>>>>
>>>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <
>>>>>>> tarmstrong@cloudera.com> wrote:
>>>>>>> > It looks like impala-data-source* may not be built in all case. It
>>>>>>> doesn't
>>>>>>> > really change much if at all so I doubt you will need to update it
>>>>>>> anyway.
>>>>>>> >
>>>>>>> > P.S. I agree with Jim, this is exactly the kind of question this
>>>>>>> list is
>>>>>>> > for.
>>>>>>> >
>>>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <
>>>>>>> pradeep1288@gmail.com>
>>>>>>> > wrote:
>>>>>>> >
>>>>>>> >> So here is what I understand:
>>>>>>> >>
>>>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>>>>>>> >> build/debug/service
>>>>>>> >>
>>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>>>>>>> >> total 330928
>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25
>>>>>>> query-options-test
>>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25
>>>>>>> session-expiry-test
>>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>>>>>> >>
>>>>>>> >> and replace it with one on my test cluster for backend part.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> For the frontend, i need to pick the below file:
>>>>>>> >>
>>>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> I couldn't find the impala-data-source-api*.jar file
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Regards,
>>>>>>> >> Pradeep Nayak
>>>>>>> >>
>>>>>>> >> http://pradeepnayak.in
>>>>>>> >> http://twitter.com/_prdp
>>>>>>> >>
>>>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>>>>>>> tarmstrong@cloudera.com>
>>>>>>> >> wrote:
>>>>>>> >>
>>>>>>> >>> I don't think this is documented anywhere. You would need to
>>>>>>> replace the
>>>>>>> >>> impalad binary and libfesupport.so under sbin-retail or
>>>>>>> sbin-debug for the
>>>>>>> >>> release or debug builds respectively. You also need to replace
>>>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>>>>>> >>>
>>>>>>> >>> You also need to make sure the statestored and catalogd symlinks
>>>>>>> point to
>>>>>>> >>> the impalad binary,
>>>>>>> >>>
>>>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <
>>>>>>> pradeep1288@gmail.com>
>>>>>>> >>> wrote:
>>>>>>> >>>
>>>>>>> >>>> This question may sound noob, but apologies in advance.
>>>>>>> >>>>
>>>>>>> >>>> I have changes in the impala source code (both backend and
>>>>>>> frontend
>>>>>>> >>>> components). How do I go about installing this version of
>>>>>>> Impala on my
>>>>>>> >>>> cloudera test cluster ? I know the path where I can replace the
>>>>>>> front end
>>>>>>> >>>> built jar file, but not sure of the backend parts.
>>>>>>> >>>>
>>>>>>> >>>> Regards,
>>>>>>> >>>> Pradeep Nayak
>>>>>>> >>>>
>>>>>>> >>>> http://pradeepnayak.in
>>>>>>> >>>> http://twitter.com/_prdp
>>>>>>> >>>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How to install the custom built impala
Posted by Tim Armstrong <ta...@cloudera.com>.
It looks like you built the debug Impala with dynamic linking
(-build_shared_libs or -so) whereas the CDH build uses static linking -
that might explain why the release build worked ok.
It's expected that your binary is much larger, in the CDH packages the
debug symbols were stripped out and put in a separate impalad.debug file,
whereas just building a dev environment doesn't do that step. Should work
fine though.
On Tue, Sep 13, 2016 at 4:44 PM, Pradeep Nayak <pr...@gmail.com>
wrote:
> Well when I built the debug one and replaced it my cluster it seemed to
> have more dependencies which it was not able to find;
>
> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.
> cdh5.8.0.p0.42/lib/impala/sbin-debug$ ldd impalad
> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
> `GLIBCXX_3.4.20' not found (required by ./impalad)
> linux-vdso.so.1 => (0x00007ffd87dc6000)
> libjsig.so => not found
> libTestUtil.so => not found
> libUtil.so => not found
> libRuntime.so => not found
> libExec.so => not found
> libCodeGen.so => not found
> libExprs.so => not found
> libRpc.so => not found
> libService.so => not found
> libStatestore.so => not found
> libCatalog.so => not found
> libResourceBroker.so => not found
> libImpalaThrift.so => not found
> libGlobalFlags.so => not found
> libCommon.so => not found
> libUdf.so => not found
> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
> (0x00007fd4661be000)
> libThriftSaslTransport.so => not found
> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
> (0x00007fd465f5f000)
> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> (0x00007fd465b83000)
> libgutil.so => not found
> libhdfs.so.0.0.0 => not found
> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000)
> libjvm.so => not found
> libkudu_client.so.0 => not found
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
> (0x00007fd465762000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x00007fd46555e000)
> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (0x00007fd46525a000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x00007fd464d3e000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007fd464b20000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000)
>
>
>
> I went ahead and build the retail one and that was OKAY.
>
> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.
> cdh5.8.0.p0.42/lib/impala/sbin-retail$ ldd impalad
> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
> `CXXABI_1.3.8' not found (required by ./impalad)
> ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
> `GLIBCXX_3.4.20' not found (required by ./impalad)
> linux-vdso.so.1 => (0x00007fffc2679000)
> libjsig.so => not found
> libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
> (0x00007f153629d000)
> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
> (0x00007f153603e000)
> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> (0x00007f1535c62000)
> libjvm.so => not found
> libkudu_client.so.0 => not found
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
> (0x00007f1535a5a000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x00007f1535856000)
> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (0x00007f1535552000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x00007f1535036000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f1534e18000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000)
>
>
> However I see the sizes being different. The one which came with cloudera
> parcels was of 37M and one which I built was 384MB!! Why is that ?
>
> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.
> cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh impalad
> 348M impalad
> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.
> cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/impala.retail.orig
> 37M /home/ubuntu/impala.retail.orig
> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.
> cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh libfesupport.so
> 346M libfesupport.so
> ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.
> cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh
> ~/libfesupport.so.retail.orig
> 39M /home/ubuntu/libfesupport.so.retail.orig
>
>
> To conclude I am able to verify the changes which I made in the retail
> build but not with the debug build. We can probably document what needs to
> be replaced to test changes in a real cluster with someone's custom impala
> binaries.
>
> 1. impalad in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/
> impala/sbin-retail
> 2. libfesupport.so in /opt/cloudera/parcels/CDH-
> 5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail
> 3. impala-frontend-0.1-SNAPSHOT.jar in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/.
> The impala-frontend-0.1-SNAPSHOT.jar file in
> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/ symlinks
> to the one in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/.
>
> This is all which is need to test your changes.
>
> Regards,
> Pradeep Nayak
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>
> On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
>> Okay thanks Tim. I tried that.
>>
>> I am seeing the below error, once I repalced the impalad and
>> libfesupport.so. Same errors for statestore and catalogd.
>>
>> + true
>> + export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
>> + IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
>> + [[ true = '' ]]
>> + '[' impalad = impalad ']'
>> + exec /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags
>> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: error while loading shared libraries: libTestUtil.so: cannot open shared object file: No such file or directory
>>
>>
>> Regards,
>> Pradeep Nayak
>>
>> http://pradeepnayak.in
>> http://twitter.com/_prdp
>>
>> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <ta...@cloudera.com>
>> wrote:
>>
>>> If the statestored and catalogd symlinks point to the new impalad binary
>>> then you're good.
>>>
>>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pr...@gmail.com>
>>> wrote:
>>>
>>>> My confusion should I update the bin location or the sbin location here
>>>> : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd and
>>>> statestored appear to be symlinks to impalad.
>>>>
>>>> Regards,
>>>> Pradeep Nayak
>>>>
>>>> http://pradeepnayak.in
>>>> http://twitter.com/_prdp
>>>>
>>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5
>>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built.
>>>>> How about catalogd and statestored ?
>>>>>
>>>>> Regards,
>>>>> Pradeep Nayak
>>>>>
>>>>> http://pradeepnayak.in
>>>>> http://twitter.com/_prdp
>>>>>
>>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>>>>>
>>>>>> If you go through this process again with a build after that change
>>>>>> goes in, then you'll need to update that jar.
>>>>>>
>>>>>> You can build it by running mvn package in Impala/ext-data-source.
>>>>>>
>>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <
>>>>>> tarmstrong@cloudera.com> wrote:
>>>>>> > It looks like impala-data-source* may not be built in all case. It
>>>>>> doesn't
>>>>>> > really change much if at all so I doubt you will need to update it
>>>>>> anyway.
>>>>>> >
>>>>>> > P.S. I agree with Jim, this is exactly the kind of question this
>>>>>> list is
>>>>>> > for.
>>>>>> >
>>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <
>>>>>> pradeep1288@gmail.com>
>>>>>> > wrote:
>>>>>> >
>>>>>> >> So here is what I understand:
>>>>>> >>
>>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>>>>>> >> build/debug/service
>>>>>> >>
>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>>>>>> >> total 330928
>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
>>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25
>>>>>> session-expiry-test
>>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>>>>> >>
>>>>>> >> and replace it with one on my test cluster for backend part.
>>>>>> >>
>>>>>> >>
>>>>>> >> For the frontend, i need to pick the below file:
>>>>>> >>
>>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>>>>> >>
>>>>>> >>
>>>>>> >> I couldn't find the impala-data-source-api*.jar file
>>>>>> >>
>>>>>> >>
>>>>>> >> Regards,
>>>>>> >> Pradeep Nayak
>>>>>> >>
>>>>>> >> http://pradeepnayak.in
>>>>>> >> http://twitter.com/_prdp
>>>>>> >>
>>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>>>>>> tarmstrong@cloudera.com>
>>>>>> >> wrote:
>>>>>> >>
>>>>>> >>> I don't think this is documented anywhere. You would need to
>>>>>> replace the
>>>>>> >>> impalad binary and libfesupport.so under sbin-retail or
>>>>>> sbin-debug for the
>>>>>> >>> release or debug builds respectively. You also need to replace
>>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>>>>> >>>
>>>>>> >>> You also need to make sure the statestored and catalogd symlinks
>>>>>> point to
>>>>>> >>> the impalad binary,
>>>>>> >>>
>>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <
>>>>>> pradeep1288@gmail.com>
>>>>>> >>> wrote:
>>>>>> >>>
>>>>>> >>>> This question may sound noob, but apologies in advance.
>>>>>> >>>>
>>>>>> >>>> I have changes in the impala source code (both backend and
>>>>>> frontend
>>>>>> >>>> components). How do I go about installing this version of Impala
>>>>>> on my
>>>>>> >>>> cloudera test cluster ? I know the path where I can replace the
>>>>>> front end
>>>>>> >>>> built jar file, but not sure of the backend parts.
>>>>>> >>>>
>>>>>> >>>> Regards,
>>>>>> >>>> Pradeep Nayak
>>>>>> >>>>
>>>>>> >>>> http://pradeepnayak.in
>>>>>> >>>> http://twitter.com/_prdp
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How to install the custom built impala
Posted by Pradeep Nayak <pr...@gmail.com>.
Well when I built the debug one and replaced it my cluster it seemed to
have more dependencies which it was not able to find;
ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug$
ldd impalad
./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
`GLIBCXX_3.4.20' not found (required by ./impalad)
linux-vdso.so.1 => (0x00007ffd87dc6000)
libjsig.so => not found
libTestUtil.so => not found
libUtil.so => not found
libRuntime.so => not found
libExec.so => not found
libCodeGen.so => not found
libExprs.so => not found
libRpc.so => not found
libService.so => not found
libStatestore.so => not found
libCatalog.so => not found
libResourceBroker.so => not found
libImpalaThrift.so => not found
libGlobalFlags.so => not found
libCommon.so => not found
libUdf.so => not found
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
(0x00007fd4661be000)
libThriftSaslTransport.so => not found
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007fd465f5f000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007fd465b83000)
libgutil.so => not found
libhdfs.so.0.0.0 => not found
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000)
libjvm.so => not found
libkudu_client.so.0 => not found
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd465762000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd46555e000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007fd46525a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007fd464d3e000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fd464b20000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000)
I went ahead and build the retail one and that was OKAY.
ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail$
ldd impalad
./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8'
not found (required by ./impalad)
./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
`GLIBCXX_3.4.20' not found (required by ./impalad)
linux-vdso.so.1 => (0x00007fffc2679000)
libjsig.so => not found
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
(0x00007f153629d000)
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f153603e000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f1535c62000)
libjvm.so => not found
libkudu_client.so.0 => not found
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1535a5a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1535856000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f1535552000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f1535036000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f1534e18000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000)
/lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000)
However I see the sizes being different. The one which came with cloudera
parcels was of 37M and one which I built was 384MB!! Why is that ?
ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail$
du -sh impalad
348M impalad
ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail$
du -sh ~/impala.retail.orig
37M /home/ubuntu/impala.retail.orig
ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail$
du -sh libfesupport.so
346M libfesupport.so
ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail$
du -sh ~/libfesupport.so.retail.orig
39M /home/ubuntu/libfesupport.so.retail.orig
To conclude I am able to verify the changes which I made in the retail
build but not with the debug build. We can probably document what needs to
be replaced to test changes in a real cluster with someone's custom impala
binaries.
1. impalad
in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail
2. libfesupport.so
in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail
3. impala-frontend-0.1-SNAPSHOT.jar in
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/. The
impala-frontend-0.1-SNAPSHOT.jar file in
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/ symlinks
to the one in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/.
This is all which is need to test your changes.
Regards,
Pradeep Nayak
http://pradeepnayak.in
http://twitter.com/_prdp
On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <pr...@gmail.com>
wrote:
> Okay thanks Tim. I tried that.
>
> I am seeing the below error, once I repalced the impalad and
> libfesupport.so. Same errors for statestore and catalogd.
>
> + true
> + export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
> + IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
> + [[ true = '' ]]
> + '[' impalad = impalad ']'
> + exec /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags
> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: error while loading shared libraries: libTestUtil.so: cannot open shared object file: No such file or directory
>
>
> Regards,
> Pradeep Nayak
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>
> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> If the statestored and catalogd symlinks point to the new impalad binary
>> then you're good.
>>
>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pr...@gmail.com>
>> wrote:
>>
>>> My confusion should I update the bin location or the sbin location here
>>> : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd and
>>> statestored appear to be symlinks to impalad.
>>>
>>> Regards,
>>> Pradeep Nayak
>>>
>>> http://pradeepnayak.in
>>> http://twitter.com/_prdp
>>>
>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
>>> wrote:
>>>
>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5
>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. How
>>>> about catalogd and statestored ?
>>>>
>>>> Regards,
>>>> Pradeep Nayak
>>>>
>>>> http://pradeepnayak.in
>>>> http://twitter.com/_prdp
>>>>
>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>>>>
>>>>> If you go through this process again with a build after that change
>>>>> goes in, then you'll need to update that jar.
>>>>>
>>>>> You can build it by running mvn package in Impala/ext-data-source.
>>>>>
>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <
>>>>> tarmstrong@cloudera.com> wrote:
>>>>> > It looks like impala-data-source* may not be built in all case. It
>>>>> doesn't
>>>>> > really change much if at all so I doubt you will need to update it
>>>>> anyway.
>>>>> >
>>>>> > P.S. I agree with Jim, this is exactly the kind of question this
>>>>> list is
>>>>> > for.
>>>>> >
>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <
>>>>> pradeep1288@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >> So here is what I understand:
>>>>> >>
>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>>>>> >> build/debug/service
>>>>> >>
>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>>>>> >> total 330928
>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>>>> >>
>>>>> >> and replace it with one on my test cluster for backend part.
>>>>> >>
>>>>> >>
>>>>> >> For the frontend, i need to pick the below file:
>>>>> >>
>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>>>> >>
>>>>> >>
>>>>> >> I couldn't find the impala-data-source-api*.jar file
>>>>> >>
>>>>> >>
>>>>> >> Regards,
>>>>> >> Pradeep Nayak
>>>>> >>
>>>>> >> http://pradeepnayak.in
>>>>> >> http://twitter.com/_prdp
>>>>> >>
>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>>>>> tarmstrong@cloudera.com>
>>>>> >> wrote:
>>>>> >>
>>>>> >>> I don't think this is documented anywhere. You would need to
>>>>> replace the
>>>>> >>> impalad binary and libfesupport.so under sbin-retail or sbin-debug
>>>>> for the
>>>>> >>> release or debug builds respectively. You also need to replace
>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>>>> >>>
>>>>> >>> You also need to make sure the statestored and catalogd symlinks
>>>>> point to
>>>>> >>> the impalad binary,
>>>>> >>>
>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <
>>>>> pradeep1288@gmail.com>
>>>>> >>> wrote:
>>>>> >>>
>>>>> >>>> This question may sound noob, but apologies in advance.
>>>>> >>>>
>>>>> >>>> I have changes in the impala source code (both backend and
>>>>> frontend
>>>>> >>>> components). How do I go about installing this version of Impala
>>>>> on my
>>>>> >>>> cloudera test cluster ? I know the path where I can replace the
>>>>> front end
>>>>> >>>> built jar file, but not sure of the backend parts.
>>>>> >>>>
>>>>> >>>> Regards,
>>>>> >>>> Pradeep Nayak
>>>>> >>>>
>>>>> >>>> http://pradeepnayak.in
>>>>> >>>> http://twitter.com/_prdp
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: How to install the custom built impala
Posted by Pradeep Nayak <pr...@gmail.com>.
Okay thanks Tim. I tried that.
I am seeing the below error, once I repalced the impalad and
libfesupport.so. Same errors for statestore and catalogd.
+ true
+ export IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
+ IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug
+ [[ true = '' ]]
+ '[' impalad = impalad ']'
+ exec /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad
--flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad:
error while loading shared libraries: libTestUtil.so: cannot open
shared object file: No such file or directory
Regards,
Pradeep Nayak
http://pradeepnayak.in
http://twitter.com/_prdp
On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <ta...@cloudera.com>
wrote:
> If the statestored and catalogd symlinks point to the new impalad binary
> then you're good.
>
> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
>> My confusion should I update the bin location or the sbin location here
>> : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd and
>> statestored appear to be symlinks to impalad.
>>
>> Regards,
>> Pradeep Nayak
>>
>> http://pradeepnayak.in
>> http://twitter.com/_prdp
>>
>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
>> wrote:
>>
>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5
>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. How
>>> about catalogd and statestored ?
>>>
>>> Regards,
>>> Pradeep Nayak
>>>
>>> http://pradeepnayak.in
>>> http://twitter.com/_prdp
>>>
>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
>>>
>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>>>
>>>> If you go through this process again with a build after that change
>>>> goes in, then you'll need to update that jar.
>>>>
>>>> You can build it by running mvn package in Impala/ext-data-source.
>>>>
>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <ta...@cloudera.com>
>>>> wrote:
>>>> > It looks like impala-data-source* may not be built in all case. It
>>>> doesn't
>>>> > really change much if at all so I doubt you will need to update it
>>>> anyway.
>>>> >
>>>> > P.S. I agree with Jim, this is exactly the kind of question this list
>>>> is
>>>> > for.
>>>> >
>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <pradeep1288@gmail.com
>>>> >
>>>> > wrote:
>>>> >
>>>> >> So here is what I understand:
>>>> >>
>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>>>> >> build/debug/service
>>>> >>
>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>>>> >> total 330928
>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>>> >>
>>>> >> and replace it with one on my test cluster for backend part.
>>>> >>
>>>> >>
>>>> >> For the frontend, i need to pick the below file:
>>>> >>
>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>>> >>
>>>> >>
>>>> >> I couldn't find the impala-data-source-api*.jar file
>>>> >>
>>>> >>
>>>> >> Regards,
>>>> >> Pradeep Nayak
>>>> >>
>>>> >> http://pradeepnayak.in
>>>> >> http://twitter.com/_prdp
>>>> >>
>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>>>> tarmstrong@cloudera.com>
>>>> >> wrote:
>>>> >>
>>>> >>> I don't think this is documented anywhere. You would need to
>>>> replace the
>>>> >>> impalad binary and libfesupport.so under sbin-retail or sbin-debug
>>>> for the
>>>> >>> release or debug builds respectively. You also need to replace
>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>>> >>>
>>>> >>> You also need to make sure the statestored and catalogd symlinks
>>>> point to
>>>> >>> the impalad binary,
>>>> >>>
>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <
>>>> pradeep1288@gmail.com>
>>>> >>> wrote:
>>>> >>>
>>>> >>>> This question may sound noob, but apologies in advance.
>>>> >>>>
>>>> >>>> I have changes in the impala source code (both backend and frontend
>>>> >>>> components). How do I go about installing this version of Impala
>>>> on my
>>>> >>>> cloudera test cluster ? I know the path where I can replace the
>>>> front end
>>>> >>>> built jar file, but not sure of the backend parts.
>>>> >>>>
>>>> >>>> Regards,
>>>> >>>> Pradeep Nayak
>>>> >>>>
>>>> >>>> http://pradeepnayak.in
>>>> >>>> http://twitter.com/_prdp
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>
>>>>
>>>
>>>
>>
>
Re: How to install the custom built impala
Posted by Tim Armstrong <ta...@cloudera.com>.
If the statestored and catalogd symlinks point to the new impalad binary
then you're good.
On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pr...@gmail.com>
wrote:
> My confusion should I update the bin location or the sbin location here
> : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd and
> statestored appear to be symlinks to impalad.
>
> Regards,
> Pradeep Nayak
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>
> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5
>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. How
>> about catalogd and statestored ?
>>
>> Regards,
>> Pradeep Nayak
>>
>> http://pradeepnayak.in
>> http://twitter.com/_prdp
>>
>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
>>
>>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>>
>>> If you go through this process again with a build after that change
>>> goes in, then you'll need to update that jar.
>>>
>>> You can build it by running mvn package in Impala/ext-data-source.
>>>
>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <ta...@cloudera.com>
>>> wrote:
>>> > It looks like impala-data-source* may not be built in all case. It
>>> doesn't
>>> > really change much if at all so I doubt you will need to update it
>>> anyway.
>>> >
>>> > P.S. I agree with Jim, this is exactly the kind of question this list
>>> is
>>> > for.
>>> >
>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <pr...@gmail.com>
>>> > wrote:
>>> >
>>> >> So here is what I understand:
>>> >>
>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>>> >> build/debug/service
>>> >>
>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>>> >> total 330928
>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>> >>
>>> >> and replace it with one on my test cluster for backend part.
>>> >>
>>> >>
>>> >> For the frontend, i need to pick the below file:
>>> >>
>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>> >>
>>> >>
>>> >> I couldn't find the impala-data-source-api*.jar file
>>> >>
>>> >>
>>> >> Regards,
>>> >> Pradeep Nayak
>>> >>
>>> >> http://pradeepnayak.in
>>> >> http://twitter.com/_prdp
>>> >>
>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>>> tarmstrong@cloudera.com>
>>> >> wrote:
>>> >>
>>> >>> I don't think this is documented anywhere. You would need to replace
>>> the
>>> >>> impalad binary and libfesupport.so under sbin-retail or sbin-debug
>>> for the
>>> >>> release or debug builds respectively. You also need to replace
>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>> >>>
>>> >>> You also need to make sure the statestored and catalogd symlinks
>>> point to
>>> >>> the impalad binary,
>>> >>>
>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <
>>> pradeep1288@gmail.com>
>>> >>> wrote:
>>> >>>
>>> >>>> This question may sound noob, but apologies in advance.
>>> >>>>
>>> >>>> I have changes in the impala source code (both backend and frontend
>>> >>>> components). How do I go about installing this version of Impala on
>>> my
>>> >>>> cloudera test cluster ? I know the path where I can replace the
>>> front end
>>> >>>> built jar file, but not sure of the backend parts.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Pradeep Nayak
>>> >>>>
>>> >>>> http://pradeepnayak.in
>>> >>>> http://twitter.com/_prdp
>>> >>>>
>>> >>>
>>> >>>
>>> >>
>>>
>>
>>
>
Re: How to install the custom built impala
Posted by Pradeep Nayak <pr...@gmail.com>.
My confusion should I update the bin location or the sbin location here
: /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd and
statestored appear to be symlinks to impalad.
Regards,
Pradeep Nayak
http://pradeepnayak.in
http://twitter.com/_prdp
On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pr...@gmail.com>
wrote:
> Okay, I can override impalad at /opt/cloudera/parcels/CDH-
> 5.8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. How
> about catalogd and statestored ?
>
> Regards,
> Pradeep Nayak
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>
> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
>
>> Correct, impala-data-source-api*.jar rarely changes, but it will when
>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>>
>> If you go through this process again with a build after that change
>> goes in, then you'll need to update that jar.
>>
>> You can build it by running mvn package in Impala/ext-data-source.
>>
>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <ta...@cloudera.com>
>> wrote:
>> > It looks like impala-data-source* may not be built in all case. It
>> doesn't
>> > really change much if at all so I doubt you will need to update it
>> anyway.
>> >
>> > P.S. I agree with Jim, this is exactly the kind of question this list is
>> > for.
>> >
>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <pr...@gmail.com>
>> > wrote:
>> >
>> >> So here is what I understand:
>> >>
>> >> I need to pick the files built here: /home/ubuntu/Impala/be/
>> >> build/debug/service
>> >>
>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>> >> total 330928
>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>> >>
>> >> and replace it with one on my test cluster for backend part.
>> >>
>> >>
>> >> For the frontend, i need to pick the below file:
>> >>
>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>> >>
>> >>
>> >> I couldn't find the impala-data-source-api*.jar file
>> >>
>> >>
>> >> Regards,
>> >> Pradeep Nayak
>> >>
>> >> http://pradeepnayak.in
>> >> http://twitter.com/_prdp
>> >>
>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <
>> tarmstrong@cloudera.com>
>> >> wrote:
>> >>
>> >>> I don't think this is documented anywhere. You would need to replace
>> the
>> >>> impalad binary and libfesupport.so under sbin-retail or sbin-debug
>> for the
>> >>> release or debug builds respectively. You also need to replace
>> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
>> >>>
>> >>> You also need to make sure the statestored and catalogd symlinks
>> point to
>> >>> the impalad binary,
>> >>>
>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pradeep1288@gmail.com
>> >
>> >>> wrote:
>> >>>
>> >>>> This question may sound noob, but apologies in advance.
>> >>>>
>> >>>> I have changes in the impala source code (both backend and frontend
>> >>>> components). How do I go about installing this version of Impala on
>> my
>> >>>> cloudera test cluster ? I know the path where I can replace the
>> front end
>> >>>> built jar file, but not sure of the backend parts.
>> >>>>
>> >>>> Regards,
>> >>>> Pradeep Nayak
>> >>>>
>> >>>> http://pradeepnayak.in
>> >>>> http://twitter.com/_prdp
>> >>>>
>> >>>
>> >>>
>> >>
>>
>
>
Re: How to install the custom built impala
Posted by Pradeep Nayak <pr...@gmail.com>.
Okay, I can override impalad
at /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin/impalad with the
one which I have built. How about catalogd and statestored ?
Regards,
Pradeep Nayak
http://pradeepnayak.in
http://twitter.com/_prdp
On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
> Correct, impala-data-source-api*.jar rarely changes, but it will when
> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
>
> If you go through this process again with a build after that change
> goes in, then you'll need to update that jar.
>
> You can build it by running mvn package in Impala/ext-data-source.
>
> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <ta...@cloudera.com>
> wrote:
> > It looks like impala-data-source* may not be built in all case. It
> doesn't
> > really change much if at all so I doubt you will need to update it
> anyway.
> >
> > P.S. I agree with Jim, this is exactly the kind of question this list is
> > for.
> >
> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <pr...@gmail.com>
> > wrote:
> >
> >> So here is what I understand:
> >>
> >> I need to pick the files built here: /home/ubuntu/Impala/be/
> >> build/debug/service
> >>
> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
> >> total 330928
> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
> >>
> >> and replace it with one on my test cluster for backend part.
> >>
> >>
> >> For the frontend, i need to pick the below file:
> >>
> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
> >>
> >>
> >> I couldn't find the impala-data-source-api*.jar file
> >>
> >>
> >> Regards,
> >> Pradeep Nayak
> >>
> >> http://pradeepnayak.in
> >> http://twitter.com/_prdp
> >>
> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <tarmstrong@cloudera.com
> >
> >> wrote:
> >>
> >>> I don't think this is documented anywhere. You would need to replace
> the
> >>> impalad binary and libfesupport.so under sbin-retail or sbin-debug for
> the
> >>> release or debug builds respectively. You also need to replace
> >>> impala-frontend-*.jar and impala-data-source-api*.jar.
> >>>
> >>> You also need to make sure the statestored and catalogd symlinks point
> to
> >>> the impalad binary,
> >>>
> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pr...@gmail.com>
> >>> wrote:
> >>>
> >>>> This question may sound noob, but apologies in advance.
> >>>>
> >>>> I have changes in the impala source code (both backend and frontend
> >>>> components). How do I go about installing this version of Impala on my
> >>>> cloudera test cluster ? I know the path where I can replace the front
> end
> >>>> built jar file, but not sure of the backend parts.
> >>>>
> >>>> Regards,
> >>>> Pradeep Nayak
> >>>>
> >>>> http://pradeepnayak.in
> >>>> http://twitter.com/_prdp
> >>>>
> >>>
> >>>
> >>
>
Re: How to install the custom built impala
Posted by Matthew Jacobs <mj...@cloudera.com>.
Correct, impala-data-source-api*.jar rarely changes, but it will when
we rename our java packages: https://gerrit.cloudera.org/#/c/3937/
If you go through this process again with a build after that change
goes in, then you'll need to update that jar.
You can build it by running mvn package in Impala/ext-data-source.
On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong <ta...@cloudera.com> wrote:
> It looks like impala-data-source* may not be built in all case. It doesn't
> really change much if at all so I doubt you will need to update it anyway.
>
> P.S. I agree with Jim, this is exactly the kind of question this list is
> for.
>
> On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
>> So here is what I understand:
>>
>> I need to pick the files built here: /home/ubuntu/Impala/be/
>> build/debug/service
>>
>> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
>> total 330928
>> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
>> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
>> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
>> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
>> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
>> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
>> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>>
>> and replace it with one on my test cluster for backend part.
>>
>>
>> For the frontend, i need to pick the below file:
>>
>> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>>
>>
>> I couldn't find the impala-data-source-api*.jar file
>>
>>
>> Regards,
>> Pradeep Nayak
>>
>> http://pradeepnayak.in
>> http://twitter.com/_prdp
>>
>> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
>> wrote:
>>
>>> I don't think this is documented anywhere. You would need to replace the
>>> impalad binary and libfesupport.so under sbin-retail or sbin-debug for the
>>> release or debug builds respectively. You also need to replace
>>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>>
>>> You also need to make sure the statestored and catalogd symlinks point to
>>> the impalad binary,
>>>
>>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pr...@gmail.com>
>>> wrote:
>>>
>>>> This question may sound noob, but apologies in advance.
>>>>
>>>> I have changes in the impala source code (both backend and frontend
>>>> components). How do I go about installing this version of Impala on my
>>>> cloudera test cluster ? I know the path where I can replace the front end
>>>> built jar file, but not sure of the backend parts.
>>>>
>>>> Regards,
>>>> Pradeep Nayak
>>>>
>>>> http://pradeepnayak.in
>>>> http://twitter.com/_prdp
>>>>
>>>
>>>
>>
Re: How to install the custom built impala
Posted by Tim Armstrong <ta...@cloudera.com>.
It looks like impala-data-source* may not be built in all case. It doesn't
really change much if at all so I doubt you will need to update it anyway.
P.S. I agree with Jim, this is exactly the kind of question this list is
for.
On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak <pr...@gmail.com>
wrote:
> So here is what I understand:
>
> I need to pick the files built here: /home/ubuntu/Impala/be/
> build/debug/service
>
> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
> total 330928
> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
>
> and replace it with one on my test cluster for backend part.
>
>
> For the frontend, i need to pick the below file:
>
> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
>
>
> I couldn't find the impala-data-source-api*.jar file
>
>
> Regards,
> Pradeep Nayak
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>
> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> I don't think this is documented anywhere. You would need to replace the
>> impalad binary and libfesupport.so under sbin-retail or sbin-debug for the
>> release or debug builds respectively. You also need to replace
>> impala-frontend-*.jar and impala-data-source-api*.jar.
>>
>> You also need to make sure the statestored and catalogd symlinks point to
>> the impalad binary,
>>
>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pr...@gmail.com>
>> wrote:
>>
>>> This question may sound noob, but apologies in advance.
>>>
>>> I have changes in the impala source code (both backend and frontend
>>> components). How do I go about installing this version of Impala on my
>>> cloudera test cluster ? I know the path where I can replace the front end
>>> built jar file, but not sure of the backend parts.
>>>
>>> Regards,
>>> Pradeep Nayak
>>>
>>> http://pradeepnayak.in
>>> http://twitter.com/_prdp
>>>
>>
>>
>
Re: How to install the custom built impala
Posted by Pradeep Nayak <pr...@gmail.com>.
So here is what I understand:
I need to pick the files built
here: /home/ubuntu/Impala/be/build/debug/service
ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l
total 330928
-rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test
*-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad*
*-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so*
*-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so*
-rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test
-rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 session-expiry-test
ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$
and replace it with one on my test cluster for backend part.
For the frontend, i need to pick the below file:
*/home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar *
I couldn't find the impala-data-source-api*.jar file
Regards,
Pradeep Nayak
http://pradeepnayak.in
http://twitter.com/_prdp
On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
wrote:
> I don't think this is documented anywhere. You would need to replace the
> impalad binary and libfesupport.so under sbin-retail or sbin-debug for the
> release or debug builds respectively. You also need to replace
> impala-frontend-*.jar and impala-data-source-api*.jar.
>
> You also need to make sure the statestored and catalogd symlinks point to
> the impalad binary,
>
> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pr...@gmail.com>
> wrote:
>
>> This question may sound noob, but apologies in advance.
>>
>> I have changes in the impala source code (both backend and frontend
>> components). How do I go about installing this version of Impala on my
>> cloudera test cluster ? I know the path where I can replace the front end
>> built jar file, but not sure of the backend parts.
>>
>> Regards,
>> Pradeep Nayak
>>
>> http://pradeepnayak.in
>> http://twitter.com/_prdp
>>
>
>
Re: How to install the custom built impala
Posted by Tim Armstrong <ta...@cloudera.com>.
I don't think this is documented anywhere. You would need to replace the
impalad binary and libfesupport.so under sbin-retail or sbin-debug for the
release or debug builds respectively. You also need to replace
impala-frontend-*.jar and impala-data-source-api*.jar.
You also need to make sure the statestored and catalogd symlinks point to
the impalad binary,
On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak <pr...@gmail.com>
wrote:
> This question may sound noob, but apologies in advance.
>
> I have changes in the impala source code (both backend and frontend
> components). How do I go about installing this version of Impala on my
> cloudera test cluster ? I know the path where I can replace the front end
> built jar file, but not sure of the backend parts.
>
> Regards,
> Pradeep Nayak
>
> http://pradeepnayak.in
> http://twitter.com/_prdp
>