You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Ed Espino <es...@apache.org> on 2017/11/13 15:25:02 UTC
Apache HAWQ (incubating) 2.3.0.0 suggestion
HAWQ,
I feel we have sufficient content to warrant a 2.3.0.0 release. It has been
4+ months since our last release (2.2.0.0). I suggest we include the
pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
chiyang10000@gmail.com) plus others (as appropriate) in the 2.3.0.0 queue in
the next release.
Are there any outstanding issues that are mandatory for the 2.3.0.0 release?
What do others think?
-=e
--
*Ed Espino*
Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
Posted by Yi JIN <ji...@gmail.com>.
Hi Radar,
According to my input, the pluggable storage framework is one desired
feature from marketing, as it is key to reenforce HAWQ performance
advantages and its code is nowunder refactor for delivery as I have seen
some activities relating to this feature, I think we just need the last
mile to have it contributed in Apache HAWQ. Therefore, I prefer to give
more patience to have one stronger release besides those amazing features
we have done, like TDE, ranger etc. Thanks.
Best,
Yi
On Thu, Nov 30, 2017 at 8:45 PM, Radar Lei <rl...@pivotal.io> wrote:
> Hi All,
>
> As Ed mentioned, we already have sufficient content(TDE, Ranger
> HA/Kerberos) to do a release, and seems Pluggable storage feature need more
> time to finish review and tests.
>
> So how about we do this release without pluggable storage? We can make
> another release soon if pluggable storage is ready for production, any
> comments? Thanks.
>
>
> Regards,
> Radar
>
> On Wed, Nov 15, 2017 at 8:44 AM, Lei Chang <ch...@gmail.com> wrote:
>
> > Cool. Looks your guys' work is quite related to the pluggable storage
> > feature (HAWQ-786).
> >
> > The feature implements a native C interface for external formats and it
> is
> > several times faster than the current JAVA interface. And the feature has
> > been validated in Oushu version for some time.
> >
> > If you guys are interested in the alignment of the feature, we can
> discuss
> > it on the JIRA.
> >
> > Cheers
> > Lei
> >
> >
> >
> >
> > On Tue, Nov 14, 2017 at 2:02 PM, 刘奎恩(局外) <ku...@alibaba-inc.com>
> > wrote:
> >
> > >
> > > Missed out DEV.
> > >
> > > ------------------------------------------------------------------
> > > 发件人：刘奎恩(局外) <ku...@alibaba-inc.com>
> > > 发送时间：2017年11月14日(星期二) 11:15
> > > 收件人：Lei Chang <ch...@gmail.com>
> > > 主 题：回复：Apache HAWQ (incubating) 2.3.0.0 suggestion
> > >
> > > Hi Dr. Chang,
> > >
> > > Good, thanks for useful suggestion. Now there is:
> > > https://issues.apache.org/jira/browse/HAWQ-1550
> > >
> > > We are trying to integrate Hawq engine onto MaxCompute (former name
> > ODPS, within
> > > Aliyun Cloud), Seahawks, it is led by Mr. Chen Xia. We develop a
> > component
> > > AXF (borrow idea from PXF) to connect data sources from Druid and
> > > MaxCompute, it works well for this 11.11 battle, but it is not
> > stand-alone
> > > program.
> > >
> > > -------------——
> > > Kuien Liu/奎恩
> > >
> > > ------------------------------------------------------------------
> > > 发件人：Lei Chang <ch...@gmail.com>
> > > 发送时间：2017年11月14日(星期二) 10:28
> > > 收件人：dev <de...@hawq.incubator.apache.org>; 刘奎恩(局外) <
> > > kuien.lke@alibaba-inc.com>
> > > 主 题：Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
> > >
> > >
> > > Kuien, we are welcoming any good contributions from the community.
> > >
> > > Looks hawq_log_master_concise is a good enhancement, I'd like suggest
> you
> > > create a JIRA and we can discuss it on the JIRA.
> > >
> > > Can you explain more about Druid Wrapper & MaxCompute Wrapper? What are
> > > the use cases here?
> > >
> > > Cheers
> > > Lei
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Nov 14, 2017 at 10:05 AM, 刘奎恩(局外) <ku...@alibaba-inc.com>
> > > wrote:
> > > Hi Ed and Hawq,
> > > I have add two GUCs (log_max_size, log_max_age) to control the logfile
> > > size on master, otherwise the query to hawq_toolkit.hawq_log_master_
> > concise
> > > will be slower and slower, and disk usage on master is hard to
> constrain.
> > > If some gys are interested to this, I can submit it to Hawq.
> > > Besides, our team has introduced many interesting features on hawq, but
> > > most of them are deep coupling with Alibaba Cloud. Some of them may be
> > > (maybe not) seperated alone, such as Druid Wrapper, MaxCompute
> Wrapper. I
> > > will discuss with teammates Mr. Chen Xia, Mr. Zhiyong Dai et al.
> > >
> > > 祝好！刘奎恩/局外
> > > ------------------------------------------------------------
> ------发件人：Ed
> > > Espino <es...@apache.org>发送时间：2017年11月13日(星期一) 23:25收件人：dev <
> > > dev@hawq.incubator.apache.org>主 题：Apache HAWQ (incubating) 2.3.0.0
> > > suggestion
> > > HAWQ,
> > >
> > > I feel we have sufficient content to warrant a 2.3.0.0
> > > release. It has been
> > > 4+ months since our last release (2.2.0.0). I suggest we include the
> > > pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
> > > chiyang10000@gmail.com) plus others (as appropriate) in
> > > the 2.3.0.0 queue in
> > > the next release.
> > >
> > > Are there any outstanding issues that are mandatory for
> > > the 2.3.0.0 release?
> > > What do others think?
> > >
> > > -=e
> > >
> > > --
> > > *Ed Espino*
> > >
> > >
> > >
> > >
> >
>
Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
Posted by Radar Lei <rl...@pivotal.io>.
Hi All,
As Ed mentioned, we already have sufficient content(TDE, Ranger
HA/Kerberos) to do a release, and seems Pluggable storage feature need more
time to finish review and tests.
So how about we do this release without pluggable storage? We can make
another release soon if pluggable storage is ready for production, any
comments? Thanks.
Regards,
Radar
On Wed, Nov 15, 2017 at 8:44 AM, Lei Chang <ch...@gmail.com> wrote:
> Cool. Looks your guys' work is quite related to the pluggable storage
> feature (HAWQ-786).
>
> The feature implements a native C interface for external formats and it is
> several times faster than the current JAVA interface. And the feature has
> been validated in Oushu version for some time.
>
> If you guys are interested in the alignment of the feature, we can discuss
> it on the JIRA.
>
> Cheers
> Lei
>
>
>
>
> On Tue, Nov 14, 2017 at 2:02 PM, 刘奎恩(局外) <ku...@alibaba-inc.com>
> wrote:
>
> >
> > Missed out DEV.
> >
> > ------------------------------------------------------------------
> > 发件人：刘奎恩(局外) <ku...@alibaba-inc.com>
> > 发送时间：2017年11月14日(星期二) 11:15
> > 收件人：Lei Chang <ch...@gmail.com>
> > 主 题：回复：Apache HAWQ (incubating) 2.3.0.0 suggestion
> >
> > Hi Dr. Chang,
> >
> > Good, thanks for useful suggestion. Now there is:
> > https://issues.apache.org/jira/browse/HAWQ-1550
> >
> > We are trying to integrate Hawq engine onto MaxCompute (former name
> ODPS, within
> > Aliyun Cloud), Seahawks, it is led by Mr. Chen Xia. We develop a
> component
> > AXF (borrow idea from PXF) to connect data sources from Druid and
> > MaxCompute, it works well for this 11.11 battle, but it is not
> stand-alone
> > program.
> >
> > -------------——
> > Kuien Liu/奎恩
> >
> > ------------------------------------------------------------------
> > 发件人：Lei Chang <ch...@gmail.com>
> > 发送时间：2017年11月14日(星期二) 10:28
> > 收件人：dev <de...@hawq.incubator.apache.org>; 刘奎恩(局外) <
> > kuien.lke@alibaba-inc.com>
> > 主 题：Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
> >
> >
> > Kuien, we are welcoming any good contributions from the community.
> >
> > Looks hawq_log_master_concise is a good enhancement, I'd like suggest you
> > create a JIRA and we can discuss it on the JIRA.
> >
> > Can you explain more about Druid Wrapper & MaxCompute Wrapper? What are
> > the use cases here?
> >
> > Cheers
> > Lei
> >
> >
> >
> >
> >
> > On Tue, Nov 14, 2017 at 10:05 AM, 刘奎恩(局外) <ku...@alibaba-inc.com>
> > wrote:
> > Hi Ed and Hawq,
> > I have add two GUCs (log_max_size, log_max_age) to control the logfile
> > size on master, otherwise the query to hawq_toolkit.hawq_log_master_
> concise
> > will be slower and slower, and disk usage on master is hard to constrain.
> > If some gys are interested to this, I can submit it to Hawq.
> > Besides, our team has introduced many interesting features on hawq, but
> > most of them are deep coupling with Alibaba Cloud. Some of them may be
> > (maybe not) seperated alone, such as Druid Wrapper, MaxCompute Wrapper. I
> > will discuss with teammates Mr. Chen Xia, Mr. Zhiyong Dai et al.
> >
> > 祝好！刘奎恩/局外
> > ------------------------------------------------------------------发件人：Ed
> > Espino <es...@apache.org>发送时间：2017年11月13日(星期一) 23:25收件人：dev <
> > dev@hawq.incubator.apache.org>主 题：Apache HAWQ (incubating) 2.3.0.0
> > suggestion
> > HAWQ,
> >
> > I feel we have sufficient content to warrant a 2.3.0.0
> > release. It has been
> > 4+ months since our last release (2.2.0.0). I suggest we include the
> > pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
> > chiyang10000@gmail.com) plus others (as appropriate) in
> > the 2.3.0.0 queue in
> > the next release.
> >
> > Are there any outstanding issues that are mandatory for
> > the 2.3.0.0 release?
> > What do others think?
> >
> > -=e
> >
> > --
> > *Ed Espino*
> >
> >
> >
> >
>
Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
Posted by Lei Chang <ch...@gmail.com>.
Cool. Looks your guys' work is quite related to the pluggable storage
feature (HAWQ-786).
The feature implements a native C interface for external formats and it is
several times faster than the current JAVA interface. And the feature has
been validated in Oushu version for some time.
If you guys are interested in the alignment of the feature, we can discuss
it on the JIRA.
Cheers
Lei
On Tue, Nov 14, 2017 at 2:02 PM, 刘奎恩(局外) <ku...@alibaba-inc.com> wrote:
>
> Missed out DEV.
>
> ------------------------------------------------------------------
> 发件人：刘奎恩(局外) <ku...@alibaba-inc.com>
> 发送时间：2017年11月14日(星期二) 11:15
> 收件人：Lei Chang <ch...@gmail.com>
> 主 题：回复：Apache HAWQ (incubating) 2.3.0.0 suggestion
>
> Hi Dr. Chang,
>
> Good, thanks for useful suggestion. Now there is:
> https://issues.apache.org/jira/browse/HAWQ-1550
>
> We are trying to integrate Hawq engine onto MaxCompute (former name ODPS, within
> Aliyun Cloud), Seahawks, it is led by Mr. Chen Xia. We develop a component
> AXF (borrow idea from PXF) to connect data sources from Druid and
> MaxCompute, it works well for this 11.11 battle, but it is not stand-alone
> program.
>
> -------------——
> Kuien Liu/奎恩
>
> ------------------------------------------------------------------
> 发件人：Lei Chang <ch...@gmail.com>
> 发送时间：2017年11月14日(星期二) 10:28
> 收件人：dev <de...@hawq.incubator.apache.org>; 刘奎恩(局外) <
> kuien.lke@alibaba-inc.com>
> 主 题：Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
>
>
> Kuien, we are welcoming any good contributions from the community.
>
> Looks hawq_log_master_concise is a good enhancement, I'd like suggest you
> create a JIRA and we can discuss it on the JIRA.
>
> Can you explain more about Druid Wrapper & MaxCompute Wrapper? What are
> the use cases here?
>
> Cheers
> Lei
>
>
>
>
>
> On Tue, Nov 14, 2017 at 10:05 AM, 刘奎恩(局外) <ku...@alibaba-inc.com>
> wrote:
> Hi Ed and Hawq,
> I have add two GUCs (log_max_size, log_max_age) to control the logfile
> size on master, otherwise the query to hawq_toolkit.hawq_log_master_concise
> will be slower and slower, and disk usage on master is hard to constrain.
> If some gys are interested to this, I can submit it to Hawq.
> Besides, our team has introduced many interesting features on hawq, but
> most of them are deep coupling with Alibaba Cloud. Some of them may be
> (maybe not) seperated alone, such as Druid Wrapper, MaxCompute Wrapper. I
> will discuss with teammates Mr. Chen Xia, Mr. Zhiyong Dai et al.
>
> 祝好！刘奎恩/局外
> ------------------------------------------------------------------发件人：Ed
> Espino <es...@apache.org>发送时间：2017年11月13日(星期一) 23:25收件人：dev <
> dev@hawq.incubator.apache.org>主 题：Apache HAWQ (incubating) 2.3.0.0
> suggestion
> HAWQ,
>
> I feel we have sufficient content to warrant a 2.3.0.0
> release. It has been
> 4+ months since our last release (2.2.0.0). I suggest we include the
> pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
> chiyang10000@gmail.com) plus others (as appropriate) in
> the 2.3.0.0 queue in
> the next release.
>
> Are there any outstanding issues that are mandatory for
> the 2.3.0.0 release?
> What do others think?
>
> -=e
>
> --
> *Ed Espino*
>
>
>
>
回复：Apache HAWQ (incubating) 2.3.0.0 suggestion
Posted by "刘奎恩(局外)" <ku...@alibaba-inc.com>.
Missed out DEV.
------------------------------------------------------------------发件人：刘奎恩(局外) <ku...@alibaba-inc.com>发送时间：2017年11月14日(星期二) 11:15收件人：Lei Chang <ch...@gmail.com>主 题：回复：Apache HAWQ (incubating) 2.3.0.0 suggestion
Hi Dr. Chang,
Good, thanks for useful suggestion. Now there is: https://issues.apache.org/jira/browse/HAWQ-1550
We are trying to integrate Hawq engine onto MaxCompute (former name ODPS, within Aliyun Cloud), Seahawks, it is led by Mr. Chen Xia. We develop a component AXF (borrow idea from PXF) to connect data sources from Druid and MaxCompute, it works well for this 11.11 battle, but it is not stand-alone program.
-------------——
Kuien Liu/奎恩
------------------------------------------------------------------发件人：Lei Chang <ch...@gmail.com>发送时间：2017年11月14日(星期二) 10:28收件人：dev <de...@hawq.incubator.apache.org>; 刘奎恩(局外) <ku...@alibaba-inc.com>主 题：Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
Kuien, we are welcoming any good contributions from the community.
Looks hawq_log_master_concise is a good enhancement, I'd like suggest you create a JIRA and we can discuss it on the JIRA.
Can you explain more about Druid Wrapper & MaxCompute Wrapper? What are the use cases here?
CheersLei
On Tue, Nov 14, 2017 at 10:05 AM, 刘奎恩(局外) <ku...@alibaba-inc.com> wrote:
Hi Ed and Hawq,
I have add two GUCs (log_max_size, log_max_age) to control the logfile size on master, otherwise the query to hawq_toolkit.hawq_log_master_concise will be slower and slower, and disk usage on master is hard to constrain. If some gys are interested to this, I can submit it to Hawq.
Besides, our team has introduced many interesting features on hawq, but most of them are deep coupling with Alibaba Cloud. Some of them may be (maybe not) seperated alone, such as Druid Wrapper, MaxCompute Wrapper. I will discuss with teammates Mr. Chen Xia, Mr. Zhiyong Dai et al.
祝好！刘奎恩/局外
------------------------------------------------------------------发件人：Ed Espino <es...@apache.org>发送时间：2017年11月13日(星期一) 23:25收件人：dev <de...@hawq.incubator.apache.org>主 题：Apache HAWQ (incubating) 2.3.0.0 suggestion
HAWQ,
I feel we have sufficient content to warrant a 2.3.0.0 release. It has been
4+ months since our last release (2.2.0.0). I suggest we include the
pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
chiyang10000@gmail.com) plus others (as appropriate) in the 2.3.0.0 queue in
the next release.
Are there any outstanding issues that are mandatory for the 2.3.0.0 release?
What do others think?
-=e
--
*Ed Espino*
Re: Apache HAWQ (incubating) 2.3.0.0 suggestion
Posted by Lei Chang <ch...@gmail.com>.
Kuien, we are welcoming any good contributions from the community.
Looks hawq_log_master_concise is a good enhancement, I'd like suggest you
create a JIRA and we can discuss it on the JIRA.
Can you explain more about Druid Wrapper & MaxCompute Wrapper? What are the
use cases here?
Cheers
Lei
On Tue, Nov 14, 2017 at 10:05 AM, 刘奎恩(局外) <ku...@alibaba-inc.com> wrote:
> Hi Ed and Hawq,
> I have add two GUCs (log_max_size, log_max_age) to control the logfile
> size on master, otherwise the query to hawq_toolkit.hawq_log_master_concise
> will be slower and slower, and disk usage on master is hard to constrain.
> If some gys are interested to this, I can submit it to Hawq.
> Besides, our team has introduced many interesting features on hawq, but
> most of them are deep coupling with Alibaba Cloud. Some of them may be
> (maybe not) seperated alone, such as Druid Wrapper, MaxCompute Wrapper. I
> will discuss with teammates Mr. Chen Xia, Mr. Zhiyong Dai et al.
>
> 祝好！刘奎恩/局外
> ------------------------------------------------------------------发件人：Ed
> Espino <es...@apache.org>发送时间：2017年11月13日(星期一) 23:25收件人：dev <
> dev@hawq.incubator.apache.org>主 题：Apache HAWQ (incubating) 2.3.0.0
> suggestion
> HAWQ,
>
> I feel we have sufficient content to warrant a 2.3.0.0
> release. It has been
> 4+ months since our last release (2.2.0.0). I suggest we include the
> pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
> chiyang10000@gmail.com) plus others (as appropriate) in
> the 2.3.0.0 queue in
> the next release.
>
> Are there any outstanding issues that are mandatory for
> the 2.3.0.0 release?
> What do others think?
>
> -=e
>
> --
> *Ed Espino*
>
回复：Apache HAWQ (incubating) 2.3.0.0 suggestion
Posted by "刘奎恩(局外)" <ku...@alibaba-inc.com>.
Hi Ed and Hawq,
I have add two GUCs (log_max_size, log_max_age) to control the logfile size on master, otherwise the query to hawq_toolkit.hawq_log_master_concise will be slower and slower, and disk usage on master is hard to constrain. If some gys are interested to this, I can submit it to Hawq.
Besides, our team has introduced many interesting features on hawq, but most of them are deep coupling with Alibaba Cloud. Some of them may be (maybe not) seperated alone, such as Druid Wrapper, MaxCompute Wrapper. I will discuss with teammates Mr. Chen Xia, Mr. Zhiyong Dai et al.
祝好！刘奎恩/局外
------------------------------------------------------------------发件人：Ed Espino <es...@apache.org>发送时间：2017年11月13日(星期一) 23:25收件人：dev <de...@hawq.incubator.apache.org>主 题：Apache HAWQ (incubating) 2.3.0.0 suggestion
HAWQ,
I feel we have sufficient content to warrant a 2.3.0.0 release. It has been
4+ months since our last release (2.2.0.0). I suggest we include the
pluggable storage feature (HAWQ-786 - assigned to Chiyang Wan -
chiyang10000@gmail.com) plus others (as appropriate) in the 2.3.0.0 queue in
the next release.
Are there any outstanding issues that are mandatory for the 2.3.0.0 release?
What do others think?
-=e
--
*Ed Espino*