Re: Improving Kudu Build Support

[Thomas Tauber-Marshall <tm...@cloudera.com>]

This has gone in now. You should be able to rebase, re-source
impala-config.sh, restart the minicluster (eg. with ./buildall.sh -noclean
-notests -start_minicluster), and continue working as normal.

If you rebase and fail to re-source impala-config.sh, you may see an error
like:
Traceback (most recent call last):
  File "/home/thomas/Impala/infra/python/bootstrap_virtualenv.py", line
383, in <module>
    kudu_client_dir = find_kudu_client_install_dir()
  File "/home/thomas/Impala/infra/python/bootstrap_virtualenv.py", line
316, in find_kudu_client_install_dir
    kudu_base_dir = os.environ["IMPALA_KUDU_HOME"]
  File "/usr/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'IMPALA_KUDU_HOME'

Let me know if you encounter any other issues.

On Thu, Aug 30, 2018 at 1:31 PM Thomas Tauber-Marshall <
tmarshall@cloudera.com> wrote:

> The patch for this is finally out: https://gerrit.cloudera.org/#/c/11363/
>
> I put some work into trying to make sure this will be as seamless as
> possible for everyone.
> - For people on Ubuntu 16 or the other CDH supported distros, you'll
> automatically start using the CDH Kudu.
> - For people on Ubuntu 14, Impala will automatically set
> USE_CDH_KUDU=false and fall back to the toolchain Kudu. You should see no
> change.
> - For people who want to develop on other OSes, KUDU_IS_SUPPORTED will get
> set to false, we will build a stub Kudu client, and you should be able to
> build and run as normal, except that Kudu related operations will fail. Or
> you can build Kudu locally and link Impala against it using the
> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables
>
> I also filed IMPALA-7515
> <https://issues.apache.org/jira/browse/IMPALA-7515> to track the eventual
> removal of toolchain Kudu support
>
> On Wed, Aug 22, 2018 at 11:31 AM Thomas Tauber-Marshall <
> tmarshall@cloudera.com> wrote:
>
>> So the toolchain binaries are provided for: centos5,6,7, debian7,8,
>> sles11,12, ubuntu14,16
>> The new CDH component binaries will be available for: redhat6,7, debian8,
>> sles12, ubuntu16
>> so we would be dropping easy support for building on basically centos5,
>> debian7, sles11, and ubuntu14
>>
>> For people who want to develop on other oses like Ubuntu 18.04, it would
>> still be possible by compiling Kudu locally and using the
>> KUDU_CLIENT_DIR/KUDU_BUILD_DIR variables.
>>
>> I'm also strongly leaning towards hiding this functionality behind a flag
>> for now (not many people chimed in on this thread, but I suspect there are
>> still a decent number of people developing on Ubuntu 14 who I'll get angry
>> emails from if they rebase and discover they have to upgrade their os), in
>> which case building the toolchain would remain a viable option for using
>> other oses as well, as least for now.
>>
>> On Tue, Aug 21, 2018 at 4:47 PM Tim Armstrong <ta...@cloudera.com>
>> wrote:
>>
>>> Is there a path to building a version of Kudu locally for an arbitrary
>>> linux distro?
>>>
>>> Personally I am less concerned about 14.04 support and more concerned
>>> about
>>> what the path to upgrading to 18.04. It would also be nice for it to be
>>> at
>>> least possible to develop on RedHat-derived distros even if it requires
>>> some extra effort.
>>>
>>> On Tue, Aug 21, 2018 at 6:48 AM, Laszlo Gaal <la...@cloudera.com>
>>> wrote:
>>>
>>> > +1 for simplifying Kudu updates.
>>> >
>>> > I am also still on Ubuntu 14.04, but I am all for simplifying Kudu
>>> > integration:
>>> > I agree with Thomas that Kudu snapshots should be grouped with the
>>> other
>>> > CDH components.
>>> > Given that Ubuntu 14.04 will be EOL'd next spring, upgrading the
>>> > development OS
>>> > is a reasonably small price to pay -- especially that it will soon
>>> become
>>> > necessary anyway.
>>> >
>>> > Thanks for doing this Thomas!
>>> >
>>> >   - Laszlo
>>> >
>>> > On Tue, Aug 21, 2018 at 12:34 AM Lars Volker <lv...@cloudera.com> wrote:
>>> >
>>> > > I'm in favor of not spending developer time and effort to maintain
>>> > > compatibility with 14.04. Personally I'm still developing on Ubuntu
>>> 14.04
>>> > > so I'd be happy if we can support it without much pain. On the other
>>> hand
>>> > > it EOLs in April 2019, so I might as well go to 18.04 now, should we
>>> > decide
>>> > > to drop support. Maybe not many other folks are on 14.04 after all?
>>> > >
>>> > >
>>> > >
>>> > > On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
>>> > > tmarshall@cloudera.com> wrote:
>>> > >
>>> > > > Impala community,
>>> > > >
>>> > > > For years now, Impala has utilized tarballs built by Cloudera and
>>> > > uploaded
>>> > > > to S3 for running most of the Hadoop components in the testing
>>> > > minicluster.
>>> > > > The one exception to this is Kudu, which is instead provided by the
>>> > > > toolchain.
>>> > > >
>>> > > > This was never ideal - native-toolchain makes more sense for
>>> libraries
>>> > > > where we want to build against a fairly static version, but Kudu is
>>> > under
>>> > > > active development and we'd like to always build against a
>>> relatively
>>> > > > up-to-date version. As a result, patches just bumping the version
>>> of
>>> > Kudu
>>> > > > make up a significant portion of the commit history of
>>> > native-toolchain.
>>> > > >
>>> > > > Thanks to work I'm currently doing at Cloudera, there will soon be
>>> > > snapshot
>>> > > > tarballs of Kudu getting uploaded to S3 along with the other Hadoop
>>> > > > components. I would like to propose that Impala switch to using
>>> those
>>> > > > instead of the toolchain Kudu.
>>> > > >
>>> > > > One problem here is that the new Kudu tarballs will not be getting
>>> > build
>>> > > > for Ubuntu 14.04, only 16.04, but we still officially say we
>>> support
>>> > > > development on 14.04.
>>> > > >
>>> > > > One option here would be to maintain the toolchain Kudu for now and
>>> > hide
>>> > > > downloading of the new tarballs behind a flag. We could also
>>> postpone
>>> > > some
>>> > > > of this work until 14.04 is less common. Or, given that the
>>> > > > bootstrap_development script already only supports 16.04, we might
>>> want
>>> > > to
>>> > > > just drop support for building on 14.04.
>>> > > >
>>> > > > Thoughts?
>>> > > >
>>> > >
>>> >
>>>
>>