You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Tim Armstrong <ta...@cloudera.com> on 2016/12/08 21:26:23 UTC
Overriding impala-config.sh values
Hi All,
I wanted to float an idea to improve the usability of impala-config.sh
One problem we've seen a lot is that certain config values, e.g.
IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment
variables. This is useful for testing against alternate components but
leads to confusing errors if you re-source a changed impala-config.sh and
get a mix of old and new config values. E.g. I've seen multiple people run
into confusing errors where it looks like files are missing from the
toolchain s3 bucket.
My idea is that we should remove this overriding mechanism and add an
alternative one without the problem based on additional config files.
impala-config.sh would determine the default values, which could be
overridden by additional config values per-branch or in the local dev
environment.
My initial idea is to have:
./bin/impala-config.sh
./bin/impala-config-branch.sh
./bin/impala-config-local.sh
impala-config-branch.sh would be blank by default and version-controlled,
and could be used on release/development branches to override particular
config values. This would make it simpler to merge and rebase branches.
impala-config-local.sh would be non-existent by default and added to
.gitignore. Users can then put whatever values they want for local testing.
Sourcing impala-config.sh would cause the config to be fully reset,
avoiding any staleness issues.
What do people think?
- Tim
Re: Overriding impala-config.sh values
Posted by Tim Armstrong <ta...@cloudera.com>.
I posted a patch for this here: https://gerrit.cloudera.org/#/c/5545/2
On Mon, Dec 12, 2016 at 3:02 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
> +1
>
> Obviously there's a lot of pain around the team with the toolchain
> version bumping.
>
> On Sat, Dec 10, 2016 at 8:16 AM, Lars Volker <lv...@cloudera.com> wrote:
> > +1
> >
> > On Thu, Dec 8, 2016 at 1:36 PM, Jim Apple <jb...@cloudera.com> wrote:
> >
> >> I like this idea.
> >>
> >> On Thu, Dec 8, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
> >> wrote:
> >> > Hi All,
> >> > I wanted to float an idea to improve the usability of
> impala-config.sh
> >> >
> >> > One problem we've seen a lot is that certain config values, e.g.
> >> > IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment
> >> > variables. This is useful for testing against alternate components but
> >> > leads to confusing errors if you re-source a changed impala-config.sh
> and
> >> > get a mix of old and new config values. E.g. I've seen multiple people
> >> run
> >> > into confusing errors where it looks like files are missing from the
> >> > toolchain s3 bucket.
> >> >
> >> > My idea is that we should remove this overriding mechanism and add an
> >> > alternative one without the problem based on additional config files.
> >> > impala-config.sh would determine the default values, which could be
> >> > overridden by additional config values per-branch or in the local dev
> >> > environment.
> >> >
> >> > My initial idea is to have:
> >> >
> >> > ./bin/impala-config.sh
> >> > ./bin/impala-config-branch.sh
> >> > ./bin/impala-config-local.sh
> >> >
> >> > impala-config-branch.sh would be blank by default and
> version-controlled,
> >> > and could be used on release/development branches to override
> particular
> >> > config values. This would make it simpler to merge and rebase
> branches.
> >> >
> >> > impala-config-local.sh would be non-existent by default and added to
> >> > .gitignore. Users can then put whatever values they want for local
> >> testing.
> >> >
> >> > Sourcing impala-config.sh would cause the config to be fully reset,
> >> > avoiding any staleness issues.
> >> >
> >> > What do people think?
> >> >
> >> > - Tim
> >>
>
Re: Overriding impala-config.sh values
Posted by Matthew Jacobs <mj...@cloudera.com>.
+1
Obviously there's a lot of pain around the team with the toolchain
version bumping.
On Sat, Dec 10, 2016 at 8:16 AM, Lars Volker <lv...@cloudera.com> wrote:
> +1
>
> On Thu, Dec 8, 2016 at 1:36 PM, Jim Apple <jb...@cloudera.com> wrote:
>
>> I like this idea.
>>
>> On Thu, Dec 8, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
>> wrote:
>> > Hi All,
>> > I wanted to float an idea to improve the usability of impala-config.sh
>> >
>> > One problem we've seen a lot is that certain config values, e.g.
>> > IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment
>> > variables. This is useful for testing against alternate components but
>> > leads to confusing errors if you re-source a changed impala-config.sh and
>> > get a mix of old and new config values. E.g. I've seen multiple people
>> run
>> > into confusing errors where it looks like files are missing from the
>> > toolchain s3 bucket.
>> >
>> > My idea is that we should remove this overriding mechanism and add an
>> > alternative one without the problem based on additional config files.
>> > impala-config.sh would determine the default values, which could be
>> > overridden by additional config values per-branch or in the local dev
>> > environment.
>> >
>> > My initial idea is to have:
>> >
>> > ./bin/impala-config.sh
>> > ./bin/impala-config-branch.sh
>> > ./bin/impala-config-local.sh
>> >
>> > impala-config-branch.sh would be blank by default and version-controlled,
>> > and could be used on release/development branches to override particular
>> > config values. This would make it simpler to merge and rebase branches.
>> >
>> > impala-config-local.sh would be non-existent by default and added to
>> > .gitignore. Users can then put whatever values they want for local
>> testing.
>> >
>> > Sourcing impala-config.sh would cause the config to be fully reset,
>> > avoiding any staleness issues.
>> >
>> > What do people think?
>> >
>> > - Tim
>>
Re: Overriding impala-config.sh values
Posted by Lars Volker <lv...@cloudera.com>.
+1
On Thu, Dec 8, 2016 at 1:36 PM, Jim Apple <jb...@cloudera.com> wrote:
> I like this idea.
>
> On Thu, Dec 8, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com>
> wrote:
> > Hi All,
> > I wanted to float an idea to improve the usability of impala-config.sh
> >
> > One problem we've seen a lot is that certain config values, e.g.
> > IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment
> > variables. This is useful for testing against alternate components but
> > leads to confusing errors if you re-source a changed impala-config.sh and
> > get a mix of old and new config values. E.g. I've seen multiple people
> run
> > into confusing errors where it looks like files are missing from the
> > toolchain s3 bucket.
> >
> > My idea is that we should remove this overriding mechanism and add an
> > alternative one without the problem based on additional config files.
> > impala-config.sh would determine the default values, which could be
> > overridden by additional config values per-branch or in the local dev
> > environment.
> >
> > My initial idea is to have:
> >
> > ./bin/impala-config.sh
> > ./bin/impala-config-branch.sh
> > ./bin/impala-config-local.sh
> >
> > impala-config-branch.sh would be blank by default and version-controlled,
> > and could be used on release/development branches to override particular
> > config values. This would make it simpler to merge and rebase branches.
> >
> > impala-config-local.sh would be non-existent by default and added to
> > .gitignore. Users can then put whatever values they want for local
> testing.
> >
> > Sourcing impala-config.sh would cause the config to be fully reset,
> > avoiding any staleness issues.
> >
> > What do people think?
> >
> > - Tim
>
Re: Overriding impala-config.sh values
Posted by Jim Apple <jb...@cloudera.com>.
I like this idea.
On Thu, Dec 8, 2016 at 1:26 PM, Tim Armstrong <ta...@cloudera.com> wrote:
> Hi All,
> I wanted to float an idea to improve the usability of impala-config.sh
>
> One problem we've seen a lot is that certain config values, e.g.
> IMPALA_TOOLCHAIN_BUILD_ID can be overridden by preexisting environment
> variables. This is useful for testing against alternate components but
> leads to confusing errors if you re-source a changed impala-config.sh and
> get a mix of old and new config values. E.g. I've seen multiple people run
> into confusing errors where it looks like files are missing from the
> toolchain s3 bucket.
>
> My idea is that we should remove this overriding mechanism and add an
> alternative one without the problem based on additional config files.
> impala-config.sh would determine the default values, which could be
> overridden by additional config values per-branch or in the local dev
> environment.
>
> My initial idea is to have:
>
> ./bin/impala-config.sh
> ./bin/impala-config-branch.sh
> ./bin/impala-config-local.sh
>
> impala-config-branch.sh would be blank by default and version-controlled,
> and could be used on release/development branches to override particular
> config values. This would make it simpler to merge and rebase branches.
>
> impala-config-local.sh would be non-existent by default and added to
> .gitignore. Users can then put whatever values they want for local testing.
>
> Sourcing impala-config.sh would cause the config to be fully reset,
> avoiding any staleness issues.
>
> What do people think?
>
> - Tim