Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

Hi all,

We are close to having a release ready, but there's a few things left
on the checklist before we can cut the first Apache release. I think
most things on this list are underway, but I'll put them here just for
reference/visibility. Comments and thoughts are welcomed.

- Build stability after merging YARN and Feeds seems to have seriously
declined. It's hard to get a build to go through to the end without
going OOM at all now honestly, so this is a Problem. I think it may be
related to Feeds, but even after disabling the tests
(https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
Therefore I am not precisely sure what is going on, but it only
started to happen after we merged those two features. It's not exactly
obvious to me where the memory leak is coming from. @Raman, it would
be great to get your advice/thoughts on this.

- Metadata name changes and Metadata caching consistency fixes are
underway by Ildar.

- The repackaging and license checker patches still need to be merged
in, but this should happen after the above two features are merged.
They are otherwise ready for review though.

- Now that Feeds is merged, the Apache website should be changed to
the new version that has been in draft form for a few weeks now.
Before it may have been a little premature, but now it should be
accurate. The documentation site should also be reverted to its prior
state, before it was quickly patched to serve as an interim website.


If there's anything else I am missing that should be in this list,
please feel free to add it into this thread.

Thanks,
-Ian

Re: Tasks remaining for release

[abdullah alamoudi <ba...@gmail.com>]

Yes there is.
New External data documentation and some enhancements after experiments
have been in the code review queue since March :-)

Will work on merging them ASAP.

Abdullah.

On Fri, Jul 3, 2015 at 4:42 AM, Ian Maxon <im...@uci.edu> wrote:

> Hi all,
>
> We are close to having a release ready, but there's a few things left
> on the checklist before we can cut the first Apache release. I think
> most things on this list are underway, but I'll put them here just for
> reference/visibility. Comments and thoughts are welcomed.
>
> - Build stability after merging YARN and Feeds seems to have seriously
> declined. It's hard to get a build to go through to the end without
> going OOM at all now honestly, so this is a Problem. I think it may be
> related to Feeds, but even after disabling the tests
> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> Therefore I am not precisely sure what is going on, but it only
> started to happen after we merged those two features. It's not exactly
> obvious to me where the memory leak is coming from. @Raman, it would
> be great to get your advice/thoughts on this.
>
> - Metadata name changes and Metadata caching consistency fixes are
> underway by Ildar.
>
> - The repackaging and license checker patches still need to be merged
> in, but this should happen after the above two features are merged.
> They are otherwise ready for review though.
>
> - Now that Feeds is merged, the Apache website should be changed to
> the new version that has been in draft form for a few weeks now.
> Before it may have been a little premature, but now it should be
> accurate. The documentation site should also be reverted to its prior
> state, before it was quickly patched to serve as an interim website.
>
>
> If there's anything else I am missing that should be in this list,
> please feel free to add it into this thread.
>
> Thanks,
> -Ian
>



-- 
Amoudi, Abdullah.

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Cool suggestion - I think Spark may do something similar since they have 
multiple APIs?  (Scala, Java, ...)

On 7/23/15 12:52 AM, Taewoo Kim wrote:
> Thanks, Ildar. I will fix that part (auto-generated PK). Your second idea
> looks good to me.
>
>
> Best,
> Taewoo
>
> On Wed, Jul 22, 2015 at 8:41 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> Taewoo,
>>
>> Check out the the first section “Dataverses, Datatypes and Datasets” in
>> primer.md.
>> You should look for a sentence “The only fields that must be specified a
>> priori are the primary key. (AsterixDB does not yet support auto-generated
>> keys)”.
>> Also primer-sql-like.md has the same issue.
>>
>> BTW maintaining two pages for AQL and NOSQL with the same content seems
>> like very daunting task. I have seen that Flink guys are using a neat JS
>> library called CodeTabs to make tabs on the page (see
>> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html
>> <
>> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html>
>> Java\Scala code examples). Can we incorporate it into our markdown
>> documentation as well?
>>
>>> On Jul 22, 2015, at 17:13, Taewoo Kim <wa...@gmail.com> wrote:
>>>
>>> Hi Ildar,
>>>
>>> Could you point out the place that you mentioned in "Moreover of a AQL
>>> primer page it is saying that we do not support that feature."? I am
>> fixing
>>> the documentation now. Thanks.
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com <ma...@gmail.com>> wrote:
>>>
>>>> While I was checking and fixing small issue with indexing documentation
>> I
>>>> have noticed that auto-generated ID feature documentation is completely
>>>> missing. Moreover of a AQL primer page it is saying that we do not
>> support
>>>> that feature.
>>>> It is considered to be done, or there is some reason to keep it below
>> the
>>>> radar?
>>>>
>>>> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
>> dtabass@gmail.com> <mailto:
>>>> dtabass@gmail.com <ma...@gmail.com>>>:
>>>> Cool.  And feed testing/stability?
>>>> We also need to have docs for the new stuff (feeds 2.0, external
>>>> data/indexing 2.0, nested/open indexing 1.0, semistructured functions
>> 0.8,
>>>> ...).
>>>>
>>>> On 7/19/15 2:12 PM, Ian Maxon wrote:
>>>> I think we basically have everything that was on this checklist merged
>>>> or ready to be merged. After the Metadata stuff is merged, all that's
>>>> left is the package change.
>>>>
>>>> -Ian
>>>>
>>>> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
>> dtabass@gmail.com> <mailto:
>>>> dtabass@gmail.com>> wrote:
>>>> Q: Any progress (e.g., @Raman) on feed/build stability?
>>>> What's still on our short list to do for the first release, in terms of
>>>> being code complete?
>>>> (I know I owe reviews of Ildar's user-visible changes.)
>>>> We'll also need to run the old traditional "Pouria torture tests" for an
>>>> extended period of time.
>>>>
>>>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>>> Hi all,
>>>>
>>>> We are close to having a release ready, but there's a few things left
>>>> on the checklist before we can cut the first Apache release. I think
>>>> most things on this list are underway, but I'll put them here just for
>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>
>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>> declined. It's hard to get a build to go through to the end without
>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>> related to Feeds, but even after disabling the tests
>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
>> https://asterix-gerrit.ics.uci.edu/#/c/312/> <
>>>> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
>>>> Therefore I am not precisely sure what is going on, but it only
>>>> started to happen after we merged those two features. It's not exactly
>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>> be great to get your advice/thoughts on this.
>>>>
>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>> underway by Ildar.
>>>>
>>>> - The repackaging and license checker patches still need to be merged
>>>> in, but this should happen after the above two features are merged.
>>>> They are otherwise ready for review though.
>>>>
>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>> the new version that has been in draft form for a few weeks now.
>>>> Before it may have been a little premature, but now it should be
>>>> accurate. The documentation site should also be reverted to its prior
>>>> state, before it was quickly patched to serve as an interim website.
>>>>
>>>>
>>>> If there's anything else I am missing that should be in this list,
>>>> please feel free to add it into this thread.
>>>>
>>>> Thanks,
>>>> -Ian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Ildar
>> Best regards,
>> Ildar
>>
>>

Re: Tasks remaining for release

[abdullah alamoudi <ba...@gmail.com>]

My external data changes have been merged. No release blocker for me.

Cheers,
Abdullah.

On Thu, Jul 23, 2015 at 10:52 AM, Taewoo Kim <wa...@gmail.com> wrote:

> Thanks, Ildar. I will fix that part (auto-generated PK). Your second idea
> looks good to me.
>
>
> Best,
> Taewoo
>
> On Wed, Jul 22, 2015 at 8:41 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
> > Taewoo,
> >
> > Check out the the first section “Dataverses, Datatypes and Datasets” in
> > primer.md.
> > You should look for a sentence “The only fields that must be specified a
> > priori are the primary key. (AsterixDB does not yet support
> auto-generated
> > keys)”.
> > Also primer-sql-like.md has the same issue.
> >
> > BTW maintaining two pages for AQL and NOSQL with the same content seems
> > like very daunting task. I have seen that Flink guys are using a neat JS
> > library called CodeTabs to make tabs on the page (see
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html
> > <
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html
> >
> > Java\Scala code examples). Can we incorporate it into our markdown
> > documentation as well?
> >
> > > On Jul 22, 2015, at 17:13, Taewoo Kim <wa...@gmail.com> wrote:
> > >
> > > Hi Ildar,
> > >
> > > Could you point out the place that you mentioned in "Moreover of a AQL
> > > primer page it is saying that we do not support that feature."? I am
> > fixing
> > > the documentation now. Thanks.
> > >
> > > Best,
> > > Taewoo
> > >
> > > On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
> > > ildar.absalyamov@gmail.com <ma...@gmail.com>> wrote:
> > >
> > >> While I was checking and fixing small issue with indexing
> documentation
> > I
> > >> have noticed that auto-generated ID feature documentation is
> completely
> > >> missing. Moreover of a AQL primer page it is saying that we do not
> > support
> > >> that feature.
> > >> It is considered to be done, or there is some reason to keep it below
> > the
> > >> radar?
> > >>
> > >> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
> > dtabass@gmail.com> <mailto:
> > >> dtabass@gmail.com <ma...@gmail.com>>>:
> > >> Cool.  And feed testing/stability?
> > >> We also need to have docs for the new stuff (feeds 2.0, external
> > >> data/indexing 2.0, nested/open indexing 1.0, semistructured functions
> > 0.8,
> > >> ...).
> > >>
> > >> On 7/19/15 2:12 PM, Ian Maxon wrote:
> > >> I think we basically have everything that was on this checklist merged
> > >> or ready to be merged. After the Metadata stuff is merged, all that's
> > >> left is the package change.
> > >>
> > >> -Ian
> > >>
> > >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com
> <mailto:
> > dtabass@gmail.com> <mailto:
> > >> dtabass@gmail.com>> wrote:
> > >> Q: Any progress (e.g., @Raman) on feed/build stability?
> > >> What's still on our short list to do for the first release, in terms
> of
> > >> being code complete?
> > >> (I know I owe reviews of Ildar's user-visible changes.)
> > >> We'll also need to run the old traditional "Pouria torture tests" for
> an
> > >> extended period of time.
> > >>
> > >> On 7/2/15 6:42 PM, Ian Maxon wrote:
> > >> Hi all,
> > >>
> > >> We are close to having a release ready, but there's a few things left
> > >> on the checklist before we can cut the first Apache release. I think
> > >> most things on this list are underway, but I'll put them here just for
> > >> reference/visibility. Comments and thoughts are welcomed.
> > >>
> > >> - Build stability after merging YARN and Feeds seems to have seriously
> > >> declined. It's hard to get a build to go through to the end without
> > >> going OOM at all now honestly, so this is a Problem. I think it may be
> > >> related to Feeds, but even after disabling the tests
> > >> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
> > https://asterix-gerrit.ics.uci.edu/#/c/312/> <
> > >> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
> > >> Therefore I am not precisely sure what is going on, but it only
> > >> started to happen after we merged those two features. It's not exactly
> > >> obvious to me where the memory leak is coming from. @Raman, it would
> > >> be great to get your advice/thoughts on this.
> > >>
> > >> - Metadata name changes and Metadata caching consistency fixes are
> > >> underway by Ildar.
> > >>
> > >> - The repackaging and license checker patches still need to be merged
> > >> in, but this should happen after the above two features are merged.
> > >> They are otherwise ready for review though.
> > >>
> > >> - Now that Feeds is merged, the Apache website should be changed to
> > >> the new version that has been in draft form for a few weeks now.
> > >> Before it may have been a little premature, but now it should be
> > >> accurate. The documentation site should also be reverted to its prior
> > >> state, before it was quickly patched to serve as an interim website.
> > >>
> > >>
> > >> If there's anything else I am missing that should be in this list,
> > >> please feel free to add it into this thread.
> > >>
> > >> Thanks,
> > >> -Ian
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >> Ildar
> >
> > Best regards,
> > Ildar
> >
> >
>



-- 
Amoudi, Abdullah.

Re: Tasks remaining for release

[Taewoo Kim <wa...@gmail.com>]

Thanks, Ildar. I will fix that part (auto-generated PK). Your second idea
looks good to me.


Best,
Taewoo

On Wed, Jul 22, 2015 at 8:41 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:

> Taewoo,
>
> Check out the the first section “Dataverses, Datatypes and Datasets” in
> primer.md.
> You should look for a sentence “The only fields that must be specified a
> priori are the primary key. (AsterixDB does not yet support auto-generated
> keys)”.
> Also primer-sql-like.md has the same issue.
>
> BTW maintaining two pages for AQL and NOSQL with the same content seems
> like very daunting task. I have seen that Flink guys are using a neat JS
> library called CodeTabs to make tabs on the page (see
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html
> <
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html>
> Java\Scala code examples). Can we incorporate it into our markdown
> documentation as well?
>
> > On Jul 22, 2015, at 17:13, Taewoo Kim <wa...@gmail.com> wrote:
> >
> > Hi Ildar,
> >
> > Could you point out the place that you mentioned in "Moreover of a AQL
> > primer page it is saying that we do not support that feature."? I am
> fixing
> > the documentation now. Thanks.
> >
> > Best,
> > Taewoo
> >
> > On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
> > ildar.absalyamov@gmail.com <ma...@gmail.com>> wrote:
> >
> >> While I was checking and fixing small issue with indexing documentation
> I
> >> have noticed that auto-generated ID feature documentation is completely
> >> missing. Moreover of a AQL primer page it is saying that we do not
> support
> >> that feature.
> >> It is considered to be done, or there is some reason to keep it below
> the
> >> radar?
> >>
> >> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
> dtabass@gmail.com> <mailto:
> >> dtabass@gmail.com <ma...@gmail.com>>>:
> >> Cool.  And feed testing/stability?
> >> We also need to have docs for the new stuff (feeds 2.0, external
> >> data/indexing 2.0, nested/open indexing 1.0, semistructured functions
> 0.8,
> >> ...).
> >>
> >> On 7/19/15 2:12 PM, Ian Maxon wrote:
> >> I think we basically have everything that was on this checklist merged
> >> or ready to be merged. After the Metadata stuff is merged, all that's
> >> left is the package change.
> >>
> >> -Ian
> >>
> >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
> dtabass@gmail.com> <mailto:
> >> dtabass@gmail.com>> wrote:
> >> Q: Any progress (e.g., @Raman) on feed/build stability?
> >> What's still on our short list to do for the first release, in terms of
> >> being code complete?
> >> (I know I owe reviews of Ildar's user-visible changes.)
> >> We'll also need to run the old traditional "Pouria torture tests" for an
> >> extended period of time.
> >>
> >> On 7/2/15 6:42 PM, Ian Maxon wrote:
> >> Hi all,
> >>
> >> We are close to having a release ready, but there's a few things left
> >> on the checklist before we can cut the first Apache release. I think
> >> most things on this list are underway, but I'll put them here just for
> >> reference/visibility. Comments and thoughts are welcomed.
> >>
> >> - Build stability after merging YARN and Feeds seems to have seriously
> >> declined. It's hard to get a build to go through to the end without
> >> going OOM at all now honestly, so this is a Problem. I think it may be
> >> related to Feeds, but even after disabling the tests
> >> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
> https://asterix-gerrit.ics.uci.edu/#/c/312/> <
> >> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
> >> Therefore I am not precisely sure what is going on, but it only
> >> started to happen after we merged those two features. It's not exactly
> >> obvious to me where the memory leak is coming from. @Raman, it would
> >> be great to get your advice/thoughts on this.
> >>
> >> - Metadata name changes and Metadata caching consistency fixes are
> >> underway by Ildar.
> >>
> >> - The repackaging and license checker patches still need to be merged
> >> in, but this should happen after the above two features are merged.
> >> They are otherwise ready for review though.
> >>
> >> - Now that Feeds is merged, the Apache website should be changed to
> >> the new version that has been in draft form for a few weeks now.
> >> Before it may have been a little premature, but now it should be
> >> accurate. The documentation site should also be reverted to its prior
> >> state, before it was quickly patched to serve as an interim website.
> >>
> >>
> >> If there's anything else I am missing that should be in this list,
> >> please feel free to add it into this thread.
> >>
> >> Thanks,
> >> -Ian
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Ildar
>
> Best regards,
> Ildar
>
>

Re: Tasks remaining for release

[Ildar Absalyamov <il...@gmail.com>]

Taewoo,

Check out the the first section “Dataverses, Datatypes and Datasets” in primer.md.
You should look for a sentence “The only fields that must be specified a priori are the primary key. (AsterixDB does not yet support auto-generated keys)”.
Also primer-sql-like.md has the same issue.

BTW maintaining two pages for AQL and NOSQL with the same content seems like very daunting task. I have seen that Flink guys are using a neat JS library called CodeTabs to make tabs on the page (see https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html <https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html> Java\Scala code examples). Can we incorporate it into our markdown documentation as well? 

> On Jul 22, 2015, at 17:13, Taewoo Kim <wa...@gmail.com> wrote:
> 
> Hi Ildar,
> 
> Could you point out the place that you mentioned in "Moreover of a AQL
> primer page it is saying that we do not support that feature."? I am fixing
> the documentation now. Thanks.
> 
> Best,
> Taewoo
> 
> On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com <ma...@gmail.com>> wrote:
> 
>> While I was checking and fixing small issue with indexing documentation I
>> have noticed that auto-generated ID feature documentation is completely
>> missing. Moreover of a AQL primer page it is saying that we do not support
>> that feature.
>> It is considered to be done, or there is some reason to keep it below the
>> radar?
>> 
>> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <ma...@gmail.com> <mailto:
>> dtabass@gmail.com <ma...@gmail.com>>>:
>> Cool.  And feed testing/stability?
>> We also need to have docs for the new stuff (feeds 2.0, external
>> data/indexing 2.0, nested/open indexing 1.0, semistructured functions 0.8,
>> ...).
>> 
>> On 7/19/15 2:12 PM, Ian Maxon wrote:
>> I think we basically have everything that was on this checklist merged
>> or ready to be merged. After the Metadata stuff is merged, all that's
>> left is the package change.
>> 
>> -Ian
>> 
>> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <ma...@gmail.com> <mailto:
>> dtabass@gmail.com>> wrote:
>> Q: Any progress (e.g., @Raman) on feed/build stability?
>> What's still on our short list to do for the first release, in terms of
>> being code complete?
>> (I know I owe reviews of Ildar's user-visible changes.)
>> We'll also need to run the old traditional "Pouria torture tests" for an
>> extended period of time.
>> 
>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>> Hi all,
>> 
>> We are close to having a release ready, but there's a few things left
>> on the checklist before we can cut the first Apache release. I think
>> most things on this list are underway, but I'll put them here just for
>> reference/visibility. Comments and thoughts are welcomed.
>> 
>> - Build stability after merging YARN and Feeds seems to have seriously
>> declined. It's hard to get a build to go through to the end without
>> going OOM at all now honestly, so this is a Problem. I think it may be
>> related to Feeds, but even after disabling the tests
>> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <https://asterix-gerrit.ics.uci.edu/#/c/312/> <
>> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
>> Therefore I am not precisely sure what is going on, but it only
>> started to happen after we merged those two features. It's not exactly
>> obvious to me where the memory leak is coming from. @Raman, it would
>> be great to get your advice/thoughts on this.
>> 
>> - Metadata name changes and Metadata caching consistency fixes are
>> underway by Ildar.
>> 
>> - The repackaging and license checker patches still need to be merged
>> in, but this should happen after the above two features are merged.
>> They are otherwise ready for review though.
>> 
>> - Now that Feeds is merged, the Apache website should be changed to
>> the new version that has been in draft form for a few weeks now.
>> Before it may have been a little premature, but now it should be
>> accurate. The documentation site should also be reverted to its prior
>> state, before it was quickly patched to serve as an interim website.
>> 
>> 
>> If there's anything else I am missing that should be in this list,
>> please feel free to add it into this thread.
>> 
>> Thanks,
>> -Ian
>> 
>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> Ildar

Best regards,
Ildar

Re: Tasks remaining for release

[Taewoo Kim <wa...@gmail.com>]

Hi Ildar,

Could you point out the place that you mentioned in "Moreover of a AQL
primer page it is saying that we do not support that feature."? I am fixing
the documentation now. Thanks.

Best,
Taewoo

On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:

> While I was checking and fixing small issue with indexing documentation I
> have noticed that auto-generated ID feature documentation is completely
> missing. Moreover of a AQL primer page it is saying that we do not support
> that feature.
> It is considered to be done, or there is some reason to keep it below the
> radar?
>
> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
> dtabass@gmail.com>>:
> Cool.  And feed testing/stability?
> We also need to have docs for the new stuff (feeds 2.0, external
> data/indexing 2.0, nested/open indexing 1.0, semistructured functions 0.8,
> ...).
>
> On 7/19/15 2:12 PM, Ian Maxon wrote:
> I think we basically have everything that was on this checklist merged
> or ready to be merged. After the Metadata stuff is merged, all that's
> left is the package change.
>
> -Ian
>
> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
> dtabass@gmail.com>> wrote:
> Q: Any progress (e.g., @Raman) on feed/build stability?
> What's still on our short list to do for the first release, in terms of
> being code complete?
> (I know I owe reviews of Ildar's user-visible changes.)
> We'll also need to run the old traditional "Pouria torture tests" for an
> extended period of time.
>
> On 7/2/15 6:42 PM, Ian Maxon wrote:
> Hi all,
>
> We are close to having a release ready, but there's a few things left
> on the checklist before we can cut the first Apache release. I think
> most things on this list are underway, but I'll put them here just for
> reference/visibility. Comments and thoughts are welcomed.
>
> - Build stability after merging YARN and Feeds seems to have seriously
> declined. It's hard to get a build to go through to the end without
> going OOM at all now honestly, so this is a Problem. I think it may be
> related to Feeds, but even after disabling the tests
> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
> Therefore I am not precisely sure what is going on, but it only
> started to happen after we merged those two features. It's not exactly
> obvious to me where the memory leak is coming from. @Raman, it would
> be great to get your advice/thoughts on this.
>
> - Metadata name changes and Metadata caching consistency fixes are
> underway by Ildar.
>
> - The repackaging and license checker patches still need to be merged
> in, but this should happen after the above two features are merged.
> They are otherwise ready for review though.
>
> - Now that Feeds is merged, the Apache website should be changed to
> the new version that has been in draft form for a few weeks now.
> Before it may have been a little premature, but now it should be
> accurate. The documentation site should also be reverted to its prior
> state, before it was quickly patched to serve as an interim website.
>
>
> If there's anything else I am missing that should be in this list,
> please feel free to add it into this thread.
>
> Thanks,
> -Ian
>
>
>
>
>
> --
> Best regards,
> Ildar

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Just an update on this: Taewoo and I discussed this AM what'll need to
be done doc-wise, and found some bugs in the process (system-wise :-)).
Right now the uuid data type is not working entirely as it should, and 
Taewoo
is fixing that.  We identified 4-5 places where some doc changes will be 
needed
after the bugs are fixed, and we also decided this will be an 
interesting enough
bit of new functionality that it probably warrants having a little 
sidebar manual
of its own (length-comparable to the one on similarity query support).

Cheers,
Mike


On 7/21/15 8:08 AM, Ian Maxon wrote:
> Speaking of docs bugs, the feeds manual also isn't visible from the
> sidebar index. You just have to know it's there :) Its markdown could
> also use a bit of cleanup.
>
> On Mon, Jul 20, 2015 at 10:53 PM, Mike Carey <dt...@gmail.com> wrote:
>> Just never documented!
>> On Jul 20, 2015 10:39 PM, "Taewoo Kim" <wa...@gmail.com> wrote:
>>
>>> When I added the auto-generated PK feature for the bulk-loading, the
>>> feature for INSERT statement was already there. A few months later,
>>> Abdullah fixed my implementation. However, it would not change the
>>> user-level syntax. I guess we can add a simple documentation regarding this
>>> feature if there is no reason to put it below the radar.
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com> wrote:
>>>
>>>> While I was checking and fixing small issue with indexing documentation I
>>>> have noticed that auto-generated ID feature documentation is completely
>>>> missing. Moreover of a AQL primer page it is saying that we do not
>>> support
>>>> that feature.
>>>> It is considered to be done, or there is some reason to keep it below the
>>>> radar?
>>>>
>>>> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
>>>> dtabass@gmail.com>>:
>>>> Cool.  And feed testing/stability?
>>>> We also need to have docs for the new stuff (feeds 2.0, external
>>>> data/indexing 2.0, nested/open indexing 1.0, semistructured functions
>>> 0.8,
>>>> ...).
>>>>
>>>> On 7/19/15 2:12 PM, Ian Maxon wrote:
>>>> I think we basically have everything that was on this checklist merged
>>>> or ready to be merged. After the Metadata stuff is merged, all that's
>>>> left is the package change.
>>>>
>>>> -Ian
>>>>
>>>> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
>>>> dtabass@gmail.com>> wrote:
>>>> Q: Any progress (e.g., @Raman) on feed/build stability?
>>>> What's still on our short list to do for the first release, in terms of
>>>> being code complete?
>>>> (I know I owe reviews of Ildar's user-visible changes.)
>>>> We'll also need to run the old traditional "Pouria torture tests" for an
>>>> extended period of time.
>>>>
>>>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>>> Hi all,
>>>>
>>>> We are close to having a release ready, but there's a few things left
>>>> on the checklist before we can cut the first Apache release. I think
>>>> most things on this list are underway, but I'll put them here just for
>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>
>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>> declined. It's hard to get a build to go through to the end without
>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>> related to Feeds, but even after disabling the tests
>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
>>>> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
>>>> Therefore I am not precisely sure what is going on, but it only
>>>> started to happen after we merged those two features. It's not exactly
>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>> be great to get your advice/thoughts on this.
>>>>
>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>> underway by Ildar.
>>>>
>>>> - The repackaging and license checker patches still need to be merged
>>>> in, but this should happen after the above two features are merged.
>>>> They are otherwise ready for review though.
>>>>
>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>> the new version that has been in draft form for a few weeks now.
>>>> Before it may have been a little premature, but now it should be
>>>> accurate. The documentation site should also be reverted to its prior
>>>> state, before it was quickly patched to serve as an interim website.
>>>>
>>>>
>>>> If there's anything else I am missing that should be in this list,
>>>> please feel free to add it into this thread.
>>>>
>>>> Thanks,
>>>> -Ian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Ildar

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

Speaking of docs bugs, the feeds manual also isn't visible from the
sidebar index. You just have to know it's there :) Its markdown could
also use a bit of cleanup.

On Mon, Jul 20, 2015 at 10:53 PM, Mike Carey <dt...@gmail.com> wrote:
> Just never documented!
> On Jul 20, 2015 10:39 PM, "Taewoo Kim" <wa...@gmail.com> wrote:
>
>> When I added the auto-generated PK feature for the bulk-loading, the
>> feature for INSERT statement was already there. A few months later,
>> Abdullah fixed my implementation. However, it would not change the
>> user-level syntax. I guess we can add a simple documentation regarding this
>> feature if there is no reason to put it below the radar.
>>
>> Best,
>> Taewoo
>>
>> On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
>> ildar.absalyamov@gmail.com> wrote:
>>
>> > While I was checking and fixing small issue with indexing documentation I
>> > have noticed that auto-generated ID feature documentation is completely
>> > missing. Moreover of a AQL primer page it is saying that we do not
>> support
>> > that feature.
>> > It is considered to be done, or there is some reason to keep it below the
>> > radar?
>> >
>> > 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
>> > dtabass@gmail.com>>:
>> > Cool.  And feed testing/stability?
>> > We also need to have docs for the new stuff (feeds 2.0, external
>> > data/indexing 2.0, nested/open indexing 1.0, semistructured functions
>> 0.8,
>> > ...).
>> >
>> > On 7/19/15 2:12 PM, Ian Maxon wrote:
>> > I think we basically have everything that was on this checklist merged
>> > or ready to be merged. After the Metadata stuff is merged, all that's
>> > left is the package change.
>> >
>> > -Ian
>> >
>> > On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
>> > dtabass@gmail.com>> wrote:
>> > Q: Any progress (e.g., @Raman) on feed/build stability?
>> > What's still on our short list to do for the first release, in terms of
>> > being code complete?
>> > (I know I owe reviews of Ildar's user-visible changes.)
>> > We'll also need to run the old traditional "Pouria torture tests" for an
>> > extended period of time.
>> >
>> > On 7/2/15 6:42 PM, Ian Maxon wrote:
>> > Hi all,
>> >
>> > We are close to having a release ready, but there's a few things left
>> > on the checklist before we can cut the first Apache release. I think
>> > most things on this list are underway, but I'll put them here just for
>> > reference/visibility. Comments and thoughts are welcomed.
>> >
>> > - Build stability after merging YARN and Feeds seems to have seriously
>> > declined. It's hard to get a build to go through to the end without
>> > going OOM at all now honestly, so this is a Problem. I think it may be
>> > related to Feeds, but even after disabling the tests
>> > (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
>> > https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
>> > Therefore I am not precisely sure what is going on, but it only
>> > started to happen after we merged those two features. It's not exactly
>> > obvious to me where the memory leak is coming from. @Raman, it would
>> > be great to get your advice/thoughts on this.
>> >
>> > - Metadata name changes and Metadata caching consistency fixes are
>> > underway by Ildar.
>> >
>> > - The repackaging and license checker patches still need to be merged
>> > in, but this should happen after the above two features are merged.
>> > They are otherwise ready for review though.
>> >
>> > - Now that Feeds is merged, the Apache website should be changed to
>> > the new version that has been in draft form for a few weeks now.
>> > Before it may have been a little premature, but now it should be
>> > accurate. The documentation site should also be reverted to its prior
>> > state, before it was quickly patched to serve as an interim website.
>> >
>> >
>> > If there's anything else I am missing that should be in this list,
>> > please feel free to add it into this thread.
>> >
>> > Thanks,
>> > -Ian
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Ildar
>>

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Just never documented!
On Jul 20, 2015 10:39 PM, "Taewoo Kim" <wa...@gmail.com> wrote:

> When I added the auto-generated PK feature for the bulk-loading, the
> feature for INSERT statement was already there. A few months later,
> Abdullah fixed my implementation. However, it would not change the
> user-level syntax. I guess we can add a simple documentation regarding this
> feature if there is no reason to put it below the radar.
>
> Best,
> Taewoo
>
> On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
> > While I was checking and fixing small issue with indexing documentation I
> > have noticed that auto-generated ID feature documentation is completely
> > missing. Moreover of a AQL primer page it is saying that we do not
> support
> > that feature.
> > It is considered to be done, or there is some reason to keep it below the
> > radar?
> >
> > 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
> > dtabass@gmail.com>>:
> > Cool.  And feed testing/stability?
> > We also need to have docs for the new stuff (feeds 2.0, external
> > data/indexing 2.0, nested/open indexing 1.0, semistructured functions
> 0.8,
> > ...).
> >
> > On 7/19/15 2:12 PM, Ian Maxon wrote:
> > I think we basically have everything that was on this checklist merged
> > or ready to be merged. After the Metadata stuff is merged, all that's
> > left is the package change.
> >
> > -Ian
> >
> > On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
> > dtabass@gmail.com>> wrote:
> > Q: Any progress (e.g., @Raman) on feed/build stability?
> > What's still on our short list to do for the first release, in terms of
> > being code complete?
> > (I know I owe reviews of Ildar's user-visible changes.)
> > We'll also need to run the old traditional "Pouria torture tests" for an
> > extended period of time.
> >
> > On 7/2/15 6:42 PM, Ian Maxon wrote:
> > Hi all,
> >
> > We are close to having a release ready, but there's a few things left
> > on the checklist before we can cut the first Apache release. I think
> > most things on this list are underway, but I'll put them here just for
> > reference/visibility. Comments and thoughts are welcomed.
> >
> > - Build stability after merging YARN and Feeds seems to have seriously
> > declined. It's hard to get a build to go through to the end without
> > going OOM at all now honestly, so this is a Problem. I think it may be
> > related to Feeds, but even after disabling the tests
> > (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
> > https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
> > Therefore I am not precisely sure what is going on, but it only
> > started to happen after we merged those two features. It's not exactly
> > obvious to me where the memory leak is coming from. @Raman, it would
> > be great to get your advice/thoughts on this.
> >
> > - Metadata name changes and Metadata caching consistency fixes are
> > underway by Ildar.
> >
> > - The repackaging and license checker patches still need to be merged
> > in, but this should happen after the above two features are merged.
> > They are otherwise ready for review though.
> >
> > - Now that Feeds is merged, the Apache website should be changed to
> > the new version that has been in draft form for a few weeks now.
> > Before it may have been a little premature, but now it should be
> > accurate. The documentation site should also be reverted to its prior
> > state, before it was quickly patched to serve as an interim website.
> >
> >
> > If there's anything else I am missing that should be in this list,
> > please feel free to add it into this thread.
> >
> > Thanks,
> > -Ian
> >
> >
> >
> >
> >
> > --
> > Best regards,
> > Ildar
>

Re: Tasks remaining for release

[Taewoo Kim <wa...@gmail.com>]

When I added the auto-generated PK feature for the bulk-loading, the
feature for INSERT statement was already there. A few months later,
Abdullah fixed my implementation. However, it would not change the
user-level syntax. I guess we can add a simple documentation regarding this
feature if there is no reason to put it below the radar.

Best,
Taewoo

On Mon, Jul 20, 2015 at 10:26 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:

> While I was checking and fixing small issue with indexing documentation I
> have noticed that auto-generated ID feature documentation is completely
> missing. Moreover of a AQL primer page it is saying that we do not support
> that feature.
> It is considered to be done, or there is some reason to keep it below the
> radar?
>
> 2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <mailto:
> dtabass@gmail.com>>:
> Cool.  And feed testing/stability?
> We also need to have docs for the new stuff (feeds 2.0, external
> data/indexing 2.0, nested/open indexing 1.0, semistructured functions 0.8,
> ...).
>
> On 7/19/15 2:12 PM, Ian Maxon wrote:
> I think we basically have everything that was on this checklist merged
> or ready to be merged. After the Metadata stuff is merged, all that's
> left is the package change.
>
> -Ian
>
> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <mailto:
> dtabass@gmail.com>> wrote:
> Q: Any progress (e.g., @Raman) on feed/build stability?
> What's still on our short list to do for the first release, in terms of
> being code complete?
> (I know I owe reviews of Ildar's user-visible changes.)
> We'll also need to run the old traditional "Pouria torture tests" for an
> extended period of time.
>
> On 7/2/15 6:42 PM, Ian Maxon wrote:
> Hi all,
>
> We are close to having a release ready, but there's a few things left
> on the checklist before we can cut the first Apache release. I think
> most things on this list are underway, but I'll put them here just for
> reference/visibility. Comments and thoughts are welcomed.
>
> - Build stability after merging YARN and Feeds seems to have seriously
> declined. It's hard to get a build to go through to the end without
> going OOM at all now honestly, so this is a Problem. I think it may be
> related to Feeds, but even after disabling the tests
> (https://asterix-gerrit.ics.uci.edu/#/c/312/ <
> https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
> Therefore I am not precisely sure what is going on, but it only
> started to happen after we merged those two features. It's not exactly
> obvious to me where the memory leak is coming from. @Raman, it would
> be great to get your advice/thoughts on this.
>
> - Metadata name changes and Metadata caching consistency fixes are
> underway by Ildar.
>
> - The repackaging and license checker patches still need to be merged
> in, but this should happen after the above two features are merged.
> They are otherwise ready for review though.
>
> - Now that Feeds is merged, the Apache website should be changed to
> the new version that has been in draft form for a few weeks now.
> Before it may have been a little premature, but now it should be
> accurate. The documentation site should also be reverted to its prior
> state, before it was quickly patched to serve as an interim website.
>
>
> If there's anything else I am missing that should be in this list,
> please feel free to add it into this thread.
>
> Thanks,
> -Ian
>
>
>
>
>
> --
> Best regards,
> Ildar

Re: Tasks remaining for release

[Ildar Absalyamov <il...@gmail.com>]

While I was checking and fixing small issue with indexing documentation I have noticed that auto-generated ID feature documentation is completely missing. Moreover of a AQL primer page it is saying that we do not support that feature.
It is considered to be done, or there is some reason to keep it below the radar?

2015-07-20 14:55 GMT-07:00 Mike Carey <dtabass@gmail.com <ma...@gmail.com>>:
Cool.  And feed testing/stability?
We also need to have docs for the new stuff (feeds 2.0, external data/indexing 2.0, nested/open indexing 1.0, semistructured functions 0.8, ...).

On 7/19/15 2:12 PM, Ian Maxon wrote:
I think we basically have everything that was on this checklist merged
or ready to be merged. After the Metadata stuff is merged, all that's
left is the package change.

-Ian

On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dtabass@gmail.com <ma...@gmail.com>> wrote:
Q: Any progress (e.g., @Raman) on feed/build stability?
What's still on our short list to do for the first release, in terms of
being code complete?
(I know I owe reviews of Ildar's user-visible changes.)
We'll also need to run the old traditional "Pouria torture tests" for an
extended period of time.

On 7/2/15 6:42 PM, Ian Maxon wrote:
Hi all,

We are close to having a release ready, but there's a few things left
on the checklist before we can cut the first Apache release. I think
most things on this list are underway, but I'll put them here just for
reference/visibility. Comments and thoughts are welcomed.

- Build stability after merging YARN and Feeds seems to have seriously
declined. It's hard to get a build to go through to the end without
going OOM at all now honestly, so this is a Problem. I think it may be
related to Feeds, but even after disabling the tests
(https://asterix-gerrit.ics.uci.edu/#/c/312/ <https://asterix-gerrit.ics.uci.edu/#/c/312/>), I still see it.
Therefore I am not precisely sure what is going on, but it only
started to happen after we merged those two features. It's not exactly
obvious to me where the memory leak is coming from. @Raman, it would
be great to get your advice/thoughts on this.

- Metadata name changes and Metadata caching consistency fixes are
underway by Ildar.

- The repackaging and license checker patches still need to be merged
in, but this should happen after the above two features are merged.
They are otherwise ready for review though.

- Now that Feeds is merged, the Apache website should be changed to
the new version that has been in draft form for a few weeks now.
Before it may have been a little premature, but now it should be
accurate. The documentation site should also be reverted to its prior
state, before it was quickly patched to serve as an interim website.


If there's anything else I am missing that should be in this list,
please feel free to add it into this thread.

Thanks,
-Ian





-- 
Best regards,
Ildar

Re: Tasks remaining for release

[Heri Ramampiaro <he...@gmail.com>]

Should the changes I proposed be pushed too?

-heri

Sent from my iPhone

On Jul 21, 2015, at 6:02 PM, Ian Maxon <im...@uci.edu> wrote:

>> Cool.  And feed testing/stability?
> 
> As far as the build is concerned, with the tests disabled it is not a
> "fire" like it was. There's still likely resource leaks, but it's not
> clear why and how severe they are (and whether or not they have
> anything to do with feeds, or that's just what provoked them). I don't
> know if Raman is actively investigating that or not. Heri was able to
> use it, so it is hopefully stable in that sense. It would probably be
> good to get his patches regarding Feeds merged in before release,
> assuming there's no issues with them.
> 
> - Ian
> 
>> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com> wrote:
>> Cool.  And feed testing/stability?
>> We also need to have docs for the new stuff (feeds 2.0, external
>> data/indexing 2.0, nested/open indexing 1.0, semistructured functions 0.8,
>> ...).
>> 
>>> On 7/19/15 2:12 PM, Ian Maxon wrote:
>>> 
>>> I think we basically have everything that was on this checklist merged
>>> or ready to be merged. After the Metadata stuff is merged, all that's
>>> left is the package change.
>>> 
>>> -Ian
>>> 
>>>> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
>>>> 
>>>> Q: Any progress (e.g., @Raman) on feed/build stability?
>>>> What's still on our short list to do for the first release, in terms of
>>>> being code complete?
>>>> (I know I owe reviews of Ildar's user-visible changes.)
>>>> We'll also need to run the old traditional "Pouria torture tests" for an
>>>> extended period of time.
>>>> 
>>>>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> We are close to having a release ready, but there's a few things left
>>>>> on the checklist before we can cut the first Apache release. I think
>>>>> most things on this list are underway, but I'll put them here just for
>>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>> 
>>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>>> declined. It's hard to get a build to go through to the end without
>>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>>> related to Feeds, but even after disabling the tests
>>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>>>>> Therefore I am not precisely sure what is going on, but it only
>>>>> started to happen after we merged those two features. It's not exactly
>>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>>> be great to get your advice/thoughts on this.
>>>>> 
>>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>>> underway by Ildar.
>>>>> 
>>>>> - The repackaging and license checker patches still need to be merged
>>>>> in, but this should happen after the above two features are merged.
>>>>> They are otherwise ready for review though.
>>>>> 
>>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>>> the new version that has been in draft form for a few weeks now.
>>>>> Before it may have been a little premature, but now it should be
>>>>> accurate. The documentation site should also be reverted to its prior
>>>>> state, before it was quickly patched to serve as an interim website.
>>>>> 
>>>>> 
>>>>> If there's anything else I am missing that should be in this list,
>>>>> please feel free to add it into this thread.
>>>>> 
>>>>> Thanks,
>>>>> -Ian
>> 

Re: Tasks remaining for release

[abdullah alamoudi <ba...@gmail.com>]

This turned out to be a problem with my internet connection which is now
fixed.

Cheers,
Abdullah.

On Wed, Jul 22, 2015 at 10:20 AM, abdullah alamoudi <ba...@gmail.com>
wrote:

> Got it.
>
> On Wed, Jul 22, 2015 at 10:19 AM, Ian Maxon <im...@uci.edu> wrote:
>
>> Gerrit actually gets backed up around midnight PST so it goes down for a
>> minute or two. Theres no way to do it live AFAIK :)
>> On Jul 22, 2015 12:07 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:
>>
>> > I am having a problem pushing the last patch. I keep getting the
>> following
>> > output:
>> >
>> > ssh: connect to host fulliautomatix.ics.uci.edu port 29418: Operation
>> > timed
>> > out
>> > fatal: Could not read from remote repository.
>> >
>> > Does anyone else see similar output?
>> >
>> >
>> > On Wed, Jul 22, 2015 at 9:29 AM, abdullah alamoudi <ba...@gmail.com>
>> > wrote:
>> >
>> > > Hold on for my last change. Raman has already reviewed it and I have
>> > > already addressed his comments.
>> > >
>> > > Cheers,
>> > > Abdullah.
>> > >
>> > > On Wed, Jul 22, 2015 at 4:02 AM, Ian Maxon <im...@uci.edu> wrote:
>> > >
>> > >> >Cool.  And feed testing/stability?
>> > >>
>> > >> As far as the build is concerned, with the tests disabled it is not a
>> > >> "fire" like it was. There's still likely resource leaks, but it's not
>> > >> clear why and how severe they are (and whether or not they have
>> > >> anything to do with feeds, or that's just what provoked them). I
>> don't
>> > >> know if Raman is actively investigating that or not. Heri was able to
>> > >> use it, so it is hopefully stable in that sense. It would probably be
>> > >> good to get his patches regarding Feeds merged in before release,
>> > >> assuming there's no issues with them.
>> > >>
>> > >> - Ian
>> > >>
>> > >> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com>
>> wrote:
>> > >> > Cool.  And feed testing/stability?
>> > >> > We also need to have docs for the new stuff (feeds 2.0, external
>> > >> > data/indexing 2.0, nested/open indexing 1.0, semistructured
>> functions
>> > >> 0.8,
>> > >> > ...).
>> > >> >
>> > >> > On 7/19/15 2:12 PM, Ian Maxon wrote:
>> > >> >>
>> > >> >> I think we basically have everything that was on this checklist
>> > merged
>> > >> >> or ready to be merged. After the Metadata stuff is merged, all
>> that's
>> > >> >> left is the package change.
>> > >> >>
>> > >> >> -Ian
>> > >> >>
>> > >> >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com>
>> > wrote:
>> > >> >>>
>> > >> >>> Q: Any progress (e.g., @Raman) on feed/build stability?
>> > >> >>> What's still on our short list to do for the first release, in
>> terms
>> > >> of
>> > >> >>> being code complete?
>> > >> >>> (I know I owe reviews of Ildar's user-visible changes.)
>> > >> >>> We'll also need to run the old traditional "Pouria torture tests"
>> > for
>> > >> an
>> > >> >>> extended period of time.
>> > >> >>>
>> > >> >>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>> > >> >>>>
>> > >> >>>> Hi all,
>> > >> >>>>
>> > >> >>>> We are close to having a release ready, but there's a few things
>> > left
>> > >> >>>> on the checklist before we can cut the first Apache release. I
>> > think
>> > >> >>>> most things on this list are underway, but I'll put them here
>> just
>> > >> for
>> > >> >>>> reference/visibility. Comments and thoughts are welcomed.
>> > >> >>>>
>> > >> >>>> - Build stability after merging YARN and Feeds seems to have
>> > >> seriously
>> > >> >>>> declined. It's hard to get a build to go through to the end
>> without
>> > >> >>>> going OOM at all now honestly, so this is a Problem. I think it
>> may
>> > >> be
>> > >> >>>> related to Feeds, but even after disabling the tests
>> > >> >>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>> > >> >>>> Therefore I am not precisely sure what is going on, but it only
>> > >> >>>> started to happen after we merged those two features. It's not
>> > >> exactly
>> > >> >>>> obvious to me where the memory leak is coming from. @Raman, it
>> > would
>> > >> >>>> be great to get your advice/thoughts on this.
>> > >> >>>>
>> > >> >>>> - Metadata name changes and Metadata caching consistency fixes
>> are
>> > >> >>>> underway by Ildar.
>> > >> >>>>
>> > >> >>>> - The repackaging and license checker patches still need to be
>> > merged
>> > >> >>>> in, but this should happen after the above two features are
>> merged.
>> > >> >>>> They are otherwise ready for review though.
>> > >> >>>>
>> > >> >>>> - Now that Feeds is merged, the Apache website should be
>> changed to
>> > >> >>>> the new version that has been in draft form for a few weeks now.
>> > >> >>>> Before it may have been a little premature, but now it should be
>> > >> >>>> accurate. The documentation site should also be reverted to its
>> > prior
>> > >> >>>> state, before it was quickly patched to serve as an interim
>> > website.
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> If there's anything else I am missing that should be in this
>> list,
>> > >> >>>> please feel free to add it into this thread.
>> > >> >>>>
>> > >> >>>> Thanks,
>> > >> >>>> -Ian
>> > >> >>>
>> > >> >>>
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Amoudi, Abdullah.
>> > >
>> >
>> >
>> >
>> > --
>> > Amoudi, Abdullah.
>> >
>>
>
>
>
> --
> Amoudi, Abdullah.
>



-- 
Amoudi, Abdullah.

Re: Tasks remaining for release

[abdullah alamoudi <ba...@gmail.com>]

Got it.

On Wed, Jul 22, 2015 at 10:19 AM, Ian Maxon <im...@uci.edu> wrote:

> Gerrit actually gets backed up around midnight PST so it goes down for a
> minute or two. Theres no way to do it live AFAIK :)
> On Jul 22, 2015 12:07 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:
>
> > I am having a problem pushing the last patch. I keep getting the
> following
> > output:
> >
> > ssh: connect to host fulliautomatix.ics.uci.edu port 29418: Operation
> > timed
> > out
> > fatal: Could not read from remote repository.
> >
> > Does anyone else see similar output?
> >
> >
> > On Wed, Jul 22, 2015 at 9:29 AM, abdullah alamoudi <ba...@gmail.com>
> > wrote:
> >
> > > Hold on for my last change. Raman has already reviewed it and I have
> > > already addressed his comments.
> > >
> > > Cheers,
> > > Abdullah.
> > >
> > > On Wed, Jul 22, 2015 at 4:02 AM, Ian Maxon <im...@uci.edu> wrote:
> > >
> > >> >Cool.  And feed testing/stability?
> > >>
> > >> As far as the build is concerned, with the tests disabled it is not a
> > >> "fire" like it was. There's still likely resource leaks, but it's not
> > >> clear why and how severe they are (and whether or not they have
> > >> anything to do with feeds, or that's just what provoked them). I don't
> > >> know if Raman is actively investigating that or not. Heri was able to
> > >> use it, so it is hopefully stable in that sense. It would probably be
> > >> good to get his patches regarding Feeds merged in before release,
> > >> assuming there's no issues with them.
> > >>
> > >> - Ian
> > >>
> > >> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com>
> wrote:
> > >> > Cool.  And feed testing/stability?
> > >> > We also need to have docs for the new stuff (feeds 2.0, external
> > >> > data/indexing 2.0, nested/open indexing 1.0, semistructured
> functions
> > >> 0.8,
> > >> > ...).
> > >> >
> > >> > On 7/19/15 2:12 PM, Ian Maxon wrote:
> > >> >>
> > >> >> I think we basically have everything that was on this checklist
> > merged
> > >> >> or ready to be merged. After the Metadata stuff is merged, all
> that's
> > >> >> left is the package change.
> > >> >>
> > >> >> -Ian
> > >> >>
> > >> >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com>
> > wrote:
> > >> >>>
> > >> >>> Q: Any progress (e.g., @Raman) on feed/build stability?
> > >> >>> What's still on our short list to do for the first release, in
> terms
> > >> of
> > >> >>> being code complete?
> > >> >>> (I know I owe reviews of Ildar's user-visible changes.)
> > >> >>> We'll also need to run the old traditional "Pouria torture tests"
> > for
> > >> an
> > >> >>> extended period of time.
> > >> >>>
> > >> >>> On 7/2/15 6:42 PM, Ian Maxon wrote:
> > >> >>>>
> > >> >>>> Hi all,
> > >> >>>>
> > >> >>>> We are close to having a release ready, but there's a few things
> > left
> > >> >>>> on the checklist before we can cut the first Apache release. I
> > think
> > >> >>>> most things on this list are underway, but I'll put them here
> just
> > >> for
> > >> >>>> reference/visibility. Comments and thoughts are welcomed.
> > >> >>>>
> > >> >>>> - Build stability after merging YARN and Feeds seems to have
> > >> seriously
> > >> >>>> declined. It's hard to get a build to go through to the end
> without
> > >> >>>> going OOM at all now honestly, so this is a Problem. I think it
> may
> > >> be
> > >> >>>> related to Feeds, but even after disabling the tests
> > >> >>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> > >> >>>> Therefore I am not precisely sure what is going on, but it only
> > >> >>>> started to happen after we merged those two features. It's not
> > >> exactly
> > >> >>>> obvious to me where the memory leak is coming from. @Raman, it
> > would
> > >> >>>> be great to get your advice/thoughts on this.
> > >> >>>>
> > >> >>>> - Metadata name changes and Metadata caching consistency fixes
> are
> > >> >>>> underway by Ildar.
> > >> >>>>
> > >> >>>> - The repackaging and license checker patches still need to be
> > merged
> > >> >>>> in, but this should happen after the above two features are
> merged.
> > >> >>>> They are otherwise ready for review though.
> > >> >>>>
> > >> >>>> - Now that Feeds is merged, the Apache website should be changed
> to
> > >> >>>> the new version that has been in draft form for a few weeks now.
> > >> >>>> Before it may have been a little premature, but now it should be
> > >> >>>> accurate. The documentation site should also be reverted to its
> > prior
> > >> >>>> state, before it was quickly patched to serve as an interim
> > website.
> > >> >>>>
> > >> >>>>
> > >> >>>> If there's anything else I am missing that should be in this
> list,
> > >> >>>> please feel free to add it into this thread.
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> -Ian
> > >> >>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Amoudi, Abdullah.
> > >
> >
> >
> >
> > --
> > Amoudi, Abdullah.
> >
>



-- 
Amoudi, Abdullah.

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

Gerrit actually gets backed up around midnight PST so it goes down for a
minute or two. Theres no way to do it live AFAIK :)
On Jul 22, 2015 12:07 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:

> I am having a problem pushing the last patch. I keep getting the following
> output:
>
> ssh: connect to host fulliautomatix.ics.uci.edu port 29418: Operation
> timed
> out
> fatal: Could not read from remote repository.
>
> Does anyone else see similar output?
>
>
> On Wed, Jul 22, 2015 at 9:29 AM, abdullah alamoudi <ba...@gmail.com>
> wrote:
>
> > Hold on for my last change. Raman has already reviewed it and I have
> > already addressed his comments.
> >
> > Cheers,
> > Abdullah.
> >
> > On Wed, Jul 22, 2015 at 4:02 AM, Ian Maxon <im...@uci.edu> wrote:
> >
> >> >Cool.  And feed testing/stability?
> >>
> >> As far as the build is concerned, with the tests disabled it is not a
> >> "fire" like it was. There's still likely resource leaks, but it's not
> >> clear why and how severe they are (and whether or not they have
> >> anything to do with feeds, or that's just what provoked them). I don't
> >> know if Raman is actively investigating that or not. Heri was able to
> >> use it, so it is hopefully stable in that sense. It would probably be
> >> good to get his patches regarding Feeds merged in before release,
> >> assuming there's no issues with them.
> >>
> >> - Ian
> >>
> >> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com> wrote:
> >> > Cool.  And feed testing/stability?
> >> > We also need to have docs for the new stuff (feeds 2.0, external
> >> > data/indexing 2.0, nested/open indexing 1.0, semistructured functions
> >> 0.8,
> >> > ...).
> >> >
> >> > On 7/19/15 2:12 PM, Ian Maxon wrote:
> >> >>
> >> >> I think we basically have everything that was on this checklist
> merged
> >> >> or ready to be merged. After the Metadata stuff is merged, all that's
> >> >> left is the package change.
> >> >>
> >> >> -Ian
> >> >>
> >> >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com>
> wrote:
> >> >>>
> >> >>> Q: Any progress (e.g., @Raman) on feed/build stability?
> >> >>> What's still on our short list to do for the first release, in terms
> >> of
> >> >>> being code complete?
> >> >>> (I know I owe reviews of Ildar's user-visible changes.)
> >> >>> We'll also need to run the old traditional "Pouria torture tests"
> for
> >> an
> >> >>> extended period of time.
> >> >>>
> >> >>> On 7/2/15 6:42 PM, Ian Maxon wrote:
> >> >>>>
> >> >>>> Hi all,
> >> >>>>
> >> >>>> We are close to having a release ready, but there's a few things
> left
> >> >>>> on the checklist before we can cut the first Apache release. I
> think
> >> >>>> most things on this list are underway, but I'll put them here just
> >> for
> >> >>>> reference/visibility. Comments and thoughts are welcomed.
> >> >>>>
> >> >>>> - Build stability after merging YARN and Feeds seems to have
> >> seriously
> >> >>>> declined. It's hard to get a build to go through to the end without
> >> >>>> going OOM at all now honestly, so this is a Problem. I think it may
> >> be
> >> >>>> related to Feeds, but even after disabling the tests
> >> >>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> >> >>>> Therefore I am not precisely sure what is going on, but it only
> >> >>>> started to happen after we merged those two features. It's not
> >> exactly
> >> >>>> obvious to me where the memory leak is coming from. @Raman, it
> would
> >> >>>> be great to get your advice/thoughts on this.
> >> >>>>
> >> >>>> - Metadata name changes and Metadata caching consistency fixes are
> >> >>>> underway by Ildar.
> >> >>>>
> >> >>>> - The repackaging and license checker patches still need to be
> merged
> >> >>>> in, but this should happen after the above two features are merged.
> >> >>>> They are otherwise ready for review though.
> >> >>>>
> >> >>>> - Now that Feeds is merged, the Apache website should be changed to
> >> >>>> the new version that has been in draft form for a few weeks now.
> >> >>>> Before it may have been a little premature, but now it should be
> >> >>>> accurate. The documentation site should also be reverted to its
> prior
> >> >>>> state, before it was quickly patched to serve as an interim
> website.
> >> >>>>
> >> >>>>
> >> >>>> If there's anything else I am missing that should be in this list,
> >> >>>> please feel free to add it into this thread.
> >> >>>>
> >> >>>> Thanks,
> >> >>>> -Ian
> >> >>>
> >> >>>
> >> >
> >>
> >
> >
> >
> > --
> > Amoudi, Abdullah.
> >
>
>
>
> --
> Amoudi, Abdullah.
>

Re: Tasks remaining for release

[abdullah alamoudi <ba...@gmail.com>]

I am having a problem pushing the last patch. I keep getting the following
output:

ssh: connect to host fulliautomatix.ics.uci.edu port 29418: Operation timed
out
fatal: Could not read from remote repository.

Does anyone else see similar output?


On Wed, Jul 22, 2015 at 9:29 AM, abdullah alamoudi <ba...@gmail.com>
wrote:

> Hold on for my last change. Raman has already reviewed it and I have
> already addressed his comments.
>
> Cheers,
> Abdullah.
>
> On Wed, Jul 22, 2015 at 4:02 AM, Ian Maxon <im...@uci.edu> wrote:
>
>> >Cool.  And feed testing/stability?
>>
>> As far as the build is concerned, with the tests disabled it is not a
>> "fire" like it was. There's still likely resource leaks, but it's not
>> clear why and how severe they are (and whether or not they have
>> anything to do with feeds, or that's just what provoked them). I don't
>> know if Raman is actively investigating that or not. Heri was able to
>> use it, so it is hopefully stable in that sense. It would probably be
>> good to get his patches regarding Feeds merged in before release,
>> assuming there's no issues with them.
>>
>> - Ian
>>
>> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com> wrote:
>> > Cool.  And feed testing/stability?
>> > We also need to have docs for the new stuff (feeds 2.0, external
>> > data/indexing 2.0, nested/open indexing 1.0, semistructured functions
>> 0.8,
>> > ...).
>> >
>> > On 7/19/15 2:12 PM, Ian Maxon wrote:
>> >>
>> >> I think we basically have everything that was on this checklist merged
>> >> or ready to be merged. After the Metadata stuff is merged, all that's
>> >> left is the package change.
>> >>
>> >> -Ian
>> >>
>> >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
>> >>>
>> >>> Q: Any progress (e.g., @Raman) on feed/build stability?
>> >>> What's still on our short list to do for the first release, in terms
>> of
>> >>> being code complete?
>> >>> (I know I owe reviews of Ildar's user-visible changes.)
>> >>> We'll also need to run the old traditional "Pouria torture tests" for
>> an
>> >>> extended period of time.
>> >>>
>> >>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>> >>>>
>> >>>> Hi all,
>> >>>>
>> >>>> We are close to having a release ready, but there's a few things left
>> >>>> on the checklist before we can cut the first Apache release. I think
>> >>>> most things on this list are underway, but I'll put them here just
>> for
>> >>>> reference/visibility. Comments and thoughts are welcomed.
>> >>>>
>> >>>> - Build stability after merging YARN and Feeds seems to have
>> seriously
>> >>>> declined. It's hard to get a build to go through to the end without
>> >>>> going OOM at all now honestly, so this is a Problem. I think it may
>> be
>> >>>> related to Feeds, but even after disabling the tests
>> >>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>> >>>> Therefore I am not precisely sure what is going on, but it only
>> >>>> started to happen after we merged those two features. It's not
>> exactly
>> >>>> obvious to me where the memory leak is coming from. @Raman, it would
>> >>>> be great to get your advice/thoughts on this.
>> >>>>
>> >>>> - Metadata name changes and Metadata caching consistency fixes are
>> >>>> underway by Ildar.
>> >>>>
>> >>>> - The repackaging and license checker patches still need to be merged
>> >>>> in, but this should happen after the above two features are merged.
>> >>>> They are otherwise ready for review though.
>> >>>>
>> >>>> - Now that Feeds is merged, the Apache website should be changed to
>> >>>> the new version that has been in draft form for a few weeks now.
>> >>>> Before it may have been a little premature, but now it should be
>> >>>> accurate. The documentation site should also be reverted to its prior
>> >>>> state, before it was quickly patched to serve as an interim website.
>> >>>>
>> >>>>
>> >>>> If there's anything else I am missing that should be in this list,
>> >>>> please feel free to add it into this thread.
>> >>>>
>> >>>> Thanks,
>> >>>> -Ian
>> >>>
>> >>>
>> >
>>
>
>
>
> --
> Amoudi, Abdullah.
>



-- 
Amoudi, Abdullah.

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Okay!  And we should ask Pouria to do the torture testing he's done in 
past (long ago now!) releases.

On 7/21/15 11:29 PM, abdullah alamoudi wrote:
> Hold on for my last change. Raman has already reviewed it and I have
> already addressed his comments.
>
> Cheers,
> Abdullah.
>
> On Wed, Jul 22, 2015 at 4:02 AM, Ian Maxon <im...@uci.edu> wrote:
>
>>> Cool.  And feed testing/stability?
>> As far as the build is concerned, with the tests disabled it is not a
>> "fire" like it was. There's still likely resource leaks, but it's not
>> clear why and how severe they are (and whether or not they have
>> anything to do with feeds, or that's just what provoked them). I don't
>> know if Raman is actively investigating that or not. Heri was able to
>> use it, so it is hopefully stable in that sense. It would probably be
>> good to get his patches regarding Feeds merged in before release,
>> assuming there's no issues with them.
>>
>> - Ian
>>
>> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com> wrote:
>>> Cool.  And feed testing/stability?
>>> We also need to have docs for the new stuff (feeds 2.0, external
>>> data/indexing 2.0, nested/open indexing 1.0, semistructured functions
>> 0.8,
>>> ...).
>>>
>>> On 7/19/15 2:12 PM, Ian Maxon wrote:
>>>> I think we basically have everything that was on this checklist merged
>>>> or ready to be merged. After the Metadata stuff is merged, all that's
>>>> left is the package change.
>>>>
>>>> -Ian
>>>>
>>>> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
>>>>> Q: Any progress (e.g., @Raman) on feed/build stability?
>>>>> What's still on our short list to do for the first release, in terms of
>>>>> being code complete?
>>>>> (I know I owe reviews of Ildar's user-visible changes.)
>>>>> We'll also need to run the old traditional "Pouria torture tests" for
>> an
>>>>> extended period of time.
>>>>>
>>>>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We are close to having a release ready, but there's a few things left
>>>>>> on the checklist before we can cut the first Apache release. I think
>>>>>> most things on this list are underway, but I'll put them here just for
>>>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>>>
>>>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>>>> declined. It's hard to get a build to go through to the end without
>>>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>>>> related to Feeds, but even after disabling the tests
>>>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>>>>>> Therefore I am not precisely sure what is going on, but it only
>>>>>> started to happen after we merged those two features. It's not exactly
>>>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>>>> be great to get your advice/thoughts on this.
>>>>>>
>>>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>>>> underway by Ildar.
>>>>>>
>>>>>> - The repackaging and license checker patches still need to be merged
>>>>>> in, but this should happen after the above two features are merged.
>>>>>> They are otherwise ready for review though.
>>>>>>
>>>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>>>> the new version that has been in draft form for a few weeks now.
>>>>>> Before it may have been a little premature, but now it should be
>>>>>> accurate. The documentation site should also be reverted to its prior
>>>>>> state, before it was quickly patched to serve as an interim website.
>>>>>>
>>>>>>
>>>>>> If there's anything else I am missing that should be in this list,
>>>>>> please feel free to add it into this thread.
>>>>>>
>>>>>> Thanks,
>>>>>> -Ian
>>>>>
>
>

Re: Tasks remaining for release

[abdullah alamoudi <ba...@gmail.com>]

Hold on for my last change. Raman has already reviewed it and I have
already addressed his comments.

Cheers,
Abdullah.

On Wed, Jul 22, 2015 at 4:02 AM, Ian Maxon <im...@uci.edu> wrote:

> >Cool.  And feed testing/stability?
>
> As far as the build is concerned, with the tests disabled it is not a
> "fire" like it was. There's still likely resource leaks, but it's not
> clear why and how severe they are (and whether or not they have
> anything to do with feeds, or that's just what provoked them). I don't
> know if Raman is actively investigating that or not. Heri was able to
> use it, so it is hopefully stable in that sense. It would probably be
> good to get his patches regarding Feeds merged in before release,
> assuming there's no issues with them.
>
> - Ian
>
> On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com> wrote:
> > Cool.  And feed testing/stability?
> > We also need to have docs for the new stuff (feeds 2.0, external
> > data/indexing 2.0, nested/open indexing 1.0, semistructured functions
> 0.8,
> > ...).
> >
> > On 7/19/15 2:12 PM, Ian Maxon wrote:
> >>
> >> I think we basically have everything that was on this checklist merged
> >> or ready to be merged. After the Metadata stuff is merged, all that's
> >> left is the package change.
> >>
> >> -Ian
> >>
> >> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
> >>>
> >>> Q: Any progress (e.g., @Raman) on feed/build stability?
> >>> What's still on our short list to do for the first release, in terms of
> >>> being code complete?
> >>> (I know I owe reviews of Ildar's user-visible changes.)
> >>> We'll also need to run the old traditional "Pouria torture tests" for
> an
> >>> extended period of time.
> >>>
> >>> On 7/2/15 6:42 PM, Ian Maxon wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> We are close to having a release ready, but there's a few things left
> >>>> on the checklist before we can cut the first Apache release. I think
> >>>> most things on this list are underway, but I'll put them here just for
> >>>> reference/visibility. Comments and thoughts are welcomed.
> >>>>
> >>>> - Build stability after merging YARN and Feeds seems to have seriously
> >>>> declined. It's hard to get a build to go through to the end without
> >>>> going OOM at all now honestly, so this is a Problem. I think it may be
> >>>> related to Feeds, but even after disabling the tests
> >>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> >>>> Therefore I am not precisely sure what is going on, but it only
> >>>> started to happen after we merged those two features. It's not exactly
> >>>> obvious to me where the memory leak is coming from. @Raman, it would
> >>>> be great to get your advice/thoughts on this.
> >>>>
> >>>> - Metadata name changes and Metadata caching consistency fixes are
> >>>> underway by Ildar.
> >>>>
> >>>> - The repackaging and license checker patches still need to be merged
> >>>> in, but this should happen after the above two features are merged.
> >>>> They are otherwise ready for review though.
> >>>>
> >>>> - Now that Feeds is merged, the Apache website should be changed to
> >>>> the new version that has been in draft form for a few weeks now.
> >>>> Before it may have been a little premature, but now it should be
> >>>> accurate. The documentation site should also be reverted to its prior
> >>>> state, before it was quickly patched to serve as an interim website.
> >>>>
> >>>>
> >>>> If there's anything else I am missing that should be in this list,
> >>>> please feel free to add it into this thread.
> >>>>
> >>>> Thanks,
> >>>> -Ian
> >>>
> >>>
> >
>



-- 
Amoudi, Abdullah.

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

>Cool.  And feed testing/stability?

As far as the build is concerned, with the tests disabled it is not a
"fire" like it was. There's still likely resource leaks, but it's not
clear why and how severe they are (and whether or not they have
anything to do with feeds, or that's just what provoked them). I don't
know if Raman is actively investigating that or not. Heri was able to
use it, so it is hopefully stable in that sense. It would probably be
good to get his patches regarding Feeds merged in before release,
assuming there's no issues with them.

- Ian

On Mon, Jul 20, 2015 at 2:55 PM, Mike Carey <dt...@gmail.com> wrote:
> Cool.  And feed testing/stability?
> We also need to have docs for the new stuff (feeds 2.0, external
> data/indexing 2.0, nested/open indexing 1.0, semistructured functions 0.8,
> ...).
>
> On 7/19/15 2:12 PM, Ian Maxon wrote:
>>
>> I think we basically have everything that was on this checklist merged
>> or ready to be merged. After the Metadata stuff is merged, all that's
>> left is the package change.
>>
>> -Ian
>>
>> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
>>>
>>> Q: Any progress (e.g., @Raman) on feed/build stability?
>>> What's still on our short list to do for the first release, in terms of
>>> being code complete?
>>> (I know I owe reviews of Ildar's user-visible changes.)
>>> We'll also need to run the old traditional "Pouria torture tests" for an
>>> extended period of time.
>>>
>>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>>>
>>>> Hi all,
>>>>
>>>> We are close to having a release ready, but there's a few things left
>>>> on the checklist before we can cut the first Apache release. I think
>>>> most things on this list are underway, but I'll put them here just for
>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>
>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>> declined. It's hard to get a build to go through to the end without
>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>> related to Feeds, but even after disabling the tests
>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>>>> Therefore I am not precisely sure what is going on, but it only
>>>> started to happen after we merged those two features. It's not exactly
>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>> be great to get your advice/thoughts on this.
>>>>
>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>> underway by Ildar.
>>>>
>>>> - The repackaging and license checker patches still need to be merged
>>>> in, but this should happen after the above two features are merged.
>>>> They are otherwise ready for review though.
>>>>
>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>> the new version that has been in draft form for a few weeks now.
>>>> Before it may have been a little premature, but now it should be
>>>> accurate. The documentation site should also be reverted to its prior
>>>> state, before it was quickly patched to serve as an interim website.
>>>>
>>>>
>>>> If there's anything else I am missing that should be in this list,
>>>> please feel free to add it into this thread.
>>>>
>>>> Thanks,
>>>> -Ian
>>>
>>>
>

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Cool.  And feed testing/stability?
We also need to have docs for the new stuff (feeds 2.0, external 
data/indexing 2.0, nested/open indexing 1.0, semistructured functions 
0.8, ...).

On 7/19/15 2:12 PM, Ian Maxon wrote:
> I think we basically have everything that was on this checklist merged
> or ready to be merged. After the Metadata stuff is merged, all that's
> left is the package change.
>
> -Ian
>
> On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
>> Q: Any progress (e.g., @Raman) on feed/build stability?
>> What's still on our short list to do for the first release, in terms of
>> being code complete?
>> (I know I owe reviews of Ildar's user-visible changes.)
>> We'll also need to run the old traditional "Pouria torture tests" for an
>> extended period of time.
>>
>> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>> Hi all,
>>>
>>> We are close to having a release ready, but there's a few things left
>>> on the checklist before we can cut the first Apache release. I think
>>> most things on this list are underway, but I'll put them here just for
>>> reference/visibility. Comments and thoughts are welcomed.
>>>
>>> - Build stability after merging YARN and Feeds seems to have seriously
>>> declined. It's hard to get a build to go through to the end without
>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>> related to Feeds, but even after disabling the tests
>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>>> Therefore I am not precisely sure what is going on, but it only
>>> started to happen after we merged those two features. It's not exactly
>>> obvious to me where the memory leak is coming from. @Raman, it would
>>> be great to get your advice/thoughts on this.
>>>
>>> - Metadata name changes and Metadata caching consistency fixes are
>>> underway by Ildar.
>>>
>>> - The repackaging and license checker patches still need to be merged
>>> in, but this should happen after the above two features are merged.
>>> They are otherwise ready for review though.
>>>
>>> - Now that Feeds is merged, the Apache website should be changed to
>>> the new version that has been in draft form for a few weeks now.
>>> Before it may have been a little premature, but now it should be
>>> accurate. The documentation site should also be reverted to its prior
>>> state, before it was quickly patched to serve as an interim website.
>>>
>>>
>>> If there's anything else I am missing that should be in this list,
>>> please feel free to add it into this thread.
>>>
>>> Thanks,
>>> -Ian
>>

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

I think we basically have everything that was on this checklist merged
or ready to be merged. After the Metadata stuff is merged, all that's
left is the package change.

-Ian

On Sun, Jul 19, 2015 at 7:34 AM, Mike Carey <dt...@gmail.com> wrote:
> Q: Any progress (e.g., @Raman) on feed/build stability?
> What's still on our short list to do for the first release, in terms of
> being code complete?
> (I know I owe reviews of Ildar's user-visible changes.)
> We'll also need to run the old traditional "Pouria torture tests" for an
> extended period of time.
>
> On 7/2/15 6:42 PM, Ian Maxon wrote:
>>
>> Hi all,
>>
>> We are close to having a release ready, but there's a few things left
>> on the checklist before we can cut the first Apache release. I think
>> most things on this list are underway, but I'll put them here just for
>> reference/visibility. Comments and thoughts are welcomed.
>>
>> - Build stability after merging YARN and Feeds seems to have seriously
>> declined. It's hard to get a build to go through to the end without
>> going OOM at all now honestly, so this is a Problem. I think it may be
>> related to Feeds, but even after disabling the tests
>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>> Therefore I am not precisely sure what is going on, but it only
>> started to happen after we merged those two features. It's not exactly
>> obvious to me where the memory leak is coming from. @Raman, it would
>> be great to get your advice/thoughts on this.
>>
>> - Metadata name changes and Metadata caching consistency fixes are
>> underway by Ildar.
>>
>> - The repackaging and license checker patches still need to be merged
>> in, but this should happen after the above two features are merged.
>> They are otherwise ready for review though.
>>
>> - Now that Feeds is merged, the Apache website should be changed to
>> the new version that has been in draft form for a few weeks now.
>> Before it may have been a little premature, but now it should be
>> accurate. The documentation site should also be reverted to its prior
>> state, before it was quickly patched to serve as an interim website.
>>
>>
>> If there's anything else I am missing that should be in this list,
>> please feel free to add it into this thread.
>>
>> Thanks,
>> -Ian
>
>

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Q: Any progress (e.g., @Raman) on feed/build stability?
What's still on our short list to do for the first release, in terms of 
being code complete?
(I know I owe reviews of Ildar's user-visible changes.)
We'll also need to run the old traditional "Pouria torture tests" for an 
extended period of time.

On 7/2/15 6:42 PM, Ian Maxon wrote:
> Hi all,
>
> We are close to having a release ready, but there's a few things left
> on the checklist before we can cut the first Apache release. I think
> most things on this list are underway, but I'll put them here just for
> reference/visibility. Comments and thoughts are welcomed.
>
> - Build stability after merging YARN and Feeds seems to have seriously
> declined. It's hard to get a build to go through to the end without
> going OOM at all now honestly, so this is a Problem. I think it may be
> related to Feeds, but even after disabling the tests
> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> Therefore I am not precisely sure what is going on, but it only
> started to happen after we merged those two features. It's not exactly
> obvious to me where the memory leak is coming from. @Raman, it would
> be great to get your advice/thoughts on this.
>
> - Metadata name changes and Metadata caching consistency fixes are
> underway by Ildar.
>
> - The repackaging and license checker patches still need to be merged
> in, but this should happen after the above two features are merged.
> They are otherwise ready for review though.
>
> - Now that Feeds is merged, the Apache website should be changed to
> the new version that has been in draft form for a few weeks now.
> Before it may have been a little premature, but now it should be
> accurate. The documentation site should also be reverted to its prior
> state, before it was quickly patched to serve as an interim website.
>
>
> If there's anything else I am missing that should be in this list,
> please feel free to add it into this thread.
>
> Thanks,
> -Ian

Re: Tasks remaining for release

[Jochen Wiedmann <jo...@gmail.com>]

On Mon, Jul 6, 2015 at 8:36 AM, Till Westmann <ti...@apache.org> wrote:

> Yes, please. I think that build stability in general are a big problem.
> And OOM issues specifically seem to be against AsterixDBs goal of not falling over when memory gets tight.
> So we probably should not release, if that’s something that everybody has a high chance of running into on first try.

There's nothing that prevents a release. If you expect issues with
that release, make users aware, recommend to stay with a previous
release, fix those problems ASAP, and cut another release.

And, btw, if you want AsterixDB to deal with OOM problems
automatically, or semiautomatically: Here's your chance to work on
that.

Jochen

Re: Tasks remaining for release

[Jochen Wiedmann <jo...@gmail.com>]

On Mon, Jul 6, 2015 at 8:36 AM, Till Westmann <ti...@apache.org> wrote:
>
>> On Jul 3, 2015, at 3:42 AM, Ian Maxon <im...@uci.edu> wrote:
>>
>> We are close to having a release ready, but there's a few things left
>> on the checklist before we can cut the first Apache release. I think
>> most things on this list are underway, but I'll put them here just for
>> reference/visibility. Comments and thoughts are welcomed.
>>
>> - Build stability after merging YARN and Feeds seems to have seriously
>> declined. It's hard to get a build to go through to the end without
>> going OOM at all now honestly, so this is a Problem. I think it may be
>> related to Feeds, but even after disabling the tests
>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>> Therefore I am not precisely sure what is going on, but it only
>> started to happen after we merged those two features. It's not exactly
>> obvious to me where the memory leak is coming from. @Raman, it would
>> be great to get your advice/thoughts on this.
>
> Yes, please. I think that build stability in general are a big problem.
> And OOM issues specifically seem to be against AsterixDBs goal of not falling over when memory gets tight.
> So we probably should not release, if that’s something that everybody has a high chance of running into on first try.
>
>> - Metadata name changes and Metadata caching consistency fixes are
>> underway by Ildar.
>>
>> - The repackaging and license checker patches still need to be merged
>> in, but this should happen after the above two features are merged.
>> They are otherwise ready for review though.
>>
>> - Now that Feeds is merged, the Apache website should be changed to
>> the new version that has been in draft form for a few weeks now.
>> Before it may have been a little premature, but now it should be
>> accurate. The documentation site should also be reverted to its prior
>> state, before it was quickly patched to serve as an interim website.
>
> Actually, I think that we should find a better way to merge the documentation and the website as they both are based on markdown and so it should be feasible to use the same translation to HTML for both. I’m not sure how much work that is and if that’s a option for the release, but I think that it would be good if we could agree that that’s the goal.
>
> Cheers,
> Till



-- 
Any world that can produce the Taj Mahal, William Shakespeare,
and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

> Yes, please. I think that build stability in general are a big problem.
> And OOM issues specifically seem to be against AsterixDBs goal of not falling over when memory gets tight.
> So we probably should not release, if that’s something that everybody has a high chance of running into on first try.
>

I went ahead and merged a patch to disable the feeds execution tests,
hopefully this will stop the OOM failures from interfering with builds
of totally unrelated patches.

> Actually, I think that we should find a better way to merge the documentation and the website as they both are based on markdown and so it should be feasible to use the same translation to HTML for both. I’m not sure how much work that is and if that’s a option for the release, but I think that it would be good if we could agree that that’s the goal.
>

I think it's ok if we want to use Jekyll for both, but they should
remain separate in terms of maintenance and build. The docs have to
live with the code IMO, because they are (or at least should be) very
tightly coupled in a sense. The website doesn't have that quality at
all, in my mind, there's no "release" of the website in the same sense
that we have for AsterixDB.

- Ian

Re: Tasks remaining for release

[Till Westmann <ti...@apache.org>]

> On Jul 3, 2015, at 3:42 AM, Ian Maxon <im...@uci.edu> wrote:
> 
> We are close to having a release ready, but there's a few things left
> on the checklist before we can cut the first Apache release. I think
> most things on this list are underway, but I'll put them here just for
> reference/visibility. Comments and thoughts are welcomed.
> 
> - Build stability after merging YARN and Feeds seems to have seriously
> declined. It's hard to get a build to go through to the end without
> going OOM at all now honestly, so this is a Problem. I think it may be
> related to Feeds, but even after disabling the tests
> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> Therefore I am not precisely sure what is going on, but it only
> started to happen after we merged those two features. It's not exactly
> obvious to me where the memory leak is coming from. @Raman, it would
> be great to get your advice/thoughts on this.

Yes, please. I think that build stability in general are a big problem. 
And OOM issues specifically seem to be against AsterixDBs goal of not falling over when memory gets tight.
So we probably should not release, if that’s something that everybody has a high chance of running into on first try.

> - Metadata name changes and Metadata caching consistency fixes are
> underway by Ildar.
> 
> - The repackaging and license checker patches still need to be merged
> in, but this should happen after the above two features are merged.
> They are otherwise ready for review though.
> 
> - Now that Feeds is merged, the Apache website should be changed to
> the new version that has been in draft form for a few weeks now.
> Before it may have been a little premature, but now it should be
> accurate. The documentation site should also be reverted to its prior
> state, before it was quickly patched to serve as an interim website.

Actually, I think that we should find a better way to merge the documentation and the website as they both are based on markdown and so it should be feasible to use the same translation to HTML for both. I’m not sure how much work that is and if that’s a option for the release, but I think that it would be good if we could agree that that’s the goal.

Cheers,
Till

Re: Tasks remaining for release

[Toby Lehman <sk...@gmail.com>]

Ian,

I'm just watching from the sidelines, but I've had my share of weird memory
problems with large systems.
It has often been the case for us that we've left some important resource
attached to a data structure (and hence it didn't get freed up and given to
the GC when we all assumed it had been).  Some of our mistakes had to do
with logging/tracking/debugging when we accidentally cached more data than
we were expecting to.

So, that said, here are some thoughts....

Are you watching the activity of the garbage collector?

And, what is the thread limit set to now ... and what happens if you raise
it?

Toby

On Mon, Jul 6, 2015 at 8:25 PM, Ian Maxon <im...@uci.edu> wrote:

> Raman and I worked on getting to the root of what is causing the build
> instability for a while today. The investigation is still ongoing but
> so far we've discovered the following things:
>
> - The OOM error specifically is running out of threads to create on
> the machine, which is odd. We aren't creating more than 500 threads
> per JVM during testing so this is especially puzzling. The heap size
> or permgen size are not the issue.
>
> - The OOM error can be observed at the point where only feeds was
> merged (and not YARN or the managix scripting fix)
>
> - Neither of us can reproduce this locally on our development
> machines. It seems that the environment is a variable in this issue
> (hitting the thread limit on the machine), somehow.
>
> - Where or if the tests run out of threads is not deterministic. It
> tends to fail around the feeds portion of the execution tests, but
> this is only a loose pattern. They can all pass, or the OOM can be hit
> during integration tests, or other totally unrelated execution tests.
>
> - There are a few feeds tests which sometimes fail (namely issue_711
> and feeds_10) but this is totally unrelated to the more major issues
> of running out of threads on the build machine.
>
> Given all the above, it looks like there is at least a degree of
> configuration/environmental influence on this issue.
>
> - Ian
>
>
>
> On Mon, Jul 6, 2015 at 2:14 PM, Raman Grover <ra...@gmail.com>
> wrote:
> > Hi
> >
> > a) The two big commits to the master (YARN integration and feeds)
> happened
> > as atomic units that makes it easier to
> > reset the master to the version prior to each feature and verify if the
> > build began showing OOM after each of the suspected commits. We have a
> > pretty deterministic way of nailing down the commit that introduced the
> > problem. I would suggest, instead of disabling the feeds tests, can we
> > revert to the earlier commit and confirm if the feeds commit did
> introduce
> > the behavior and repeat the test with the YARN commit that followed. We
> > should be able to see sudden increase/drop in build stability by running
> > sufficient number of iterations.
> >
> > b) I have not been able to reproduce the OOM at my setup where I have
> been
> > running the build repeatedly.
> > @Ian are you able to reproduce it at your system? May be I am not running
> > the build sufficient number of times?
> > I am still not able to understand how removal of test cases still causes
> > the OOM? I can go back and look at the precise changes made during the
> > feeds commit that could introduce OOM even if feeds are not involved at
> > all, but as I see it, the changes made do not play a role if feeds are
> not
> > being ingested.
> >
> >
> > Regards,
> > Raman
> >
> >
> > On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:
> >
> >> Hi all,
> >>
> >> We are close to having a release ready, but there's a few things left
> >> on the checklist before we can cut the first Apache release. I think
> >> most things on this list are underway, but I'll put them here just for
> >> reference/visibility. Comments and thoughts are welcomed.
> >>
> >> - Build stability after merging YARN and Feeds seems to have seriously
> >> declined. It's hard to get a build to go through to the end without
> >> going OOM at all now honestly, so this is a Problem. I think it may be
> >> related to Feeds, but even after disabling the tests
> >> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> >> Therefore I am not precisely sure what is going on, but it only
> >> started to happen after we merged those two features. It's not exactly
> >> obvious to me where the memory leak is coming from. @Raman, it would
> >> be great to get your advice/thoughts on this.
> >>
> >> - Metadata name changes and Metadata caching consistency fixes are
> >> underway by Ildar.
> >>
> >> - The repackaging and license checker patches still need to be merged
> >> in, but this should happen after the above two features are merged.
> >> They are otherwise ready for review though.
> >>
> >> - Now that Feeds is merged, the Apache website should be changed to
> >> the new version that has been in draft form for a few weeks now.
> >> Before it may have been a little premature, but now it should be
> >> accurate. The documentation site should also be reverted to its prior
> >> state, before it was quickly patched to serve as an interim website.
> >>
> >>
> >> If there's anything else I am missing that should be in this list,
> >> please feel free to add it into this thread.
> >>
> >> Thanks,
> >> -Ian
> >>
> >
> >
> >
> > --
> > Raman
>

Re: Tasks remaining for release

[Ildar Absalyamov <il...@gmail.com>]

Shouldn’t those threads get reused in a fixed size pool?

> On Jul 7, 2015, at 23:40, Mike Carey <dt...@gmail.com> wrote:
> 
> Wierd....  That many threads seems wrong.....
> 
> On 7/7/15 8:35 PM, Ian Maxon wrote:
>> I think I have at least a workaround to the thread starvation nailed
>> down. We'll have to see, but basically I think the latest few patches
>> cause us to use more threads for whatever reason- and this pushed us
>> over the default thread cap in many circumstances (not always). Going
>> ahead and setting the number of processes to be unlimited within the
>> build server and containers seems to have put out the fire, so to
>> speak. Another confounding factor is the issue that docker containers
>> run within the same host and hence also have their own shared thread
>> limit, in addition to the host's thread limit. It's not clear to me
>> however whether we intend to use that many threads (~500), or if
>> there's a subtle resource leak somewhere.
>> 
>> - Ian
>> 
>> On Tue, Jul 7, 2015 at 5:44 PM, Eldon Carman <ec...@ucr.edu> wrote:
>>> In my branch ("ecarm002/introspection_alternate"), I have adapted some code
>>> I received from Ildar to repeatedly test a set of runtime tests. I am not
>>> sure this testing process will be related to your issue or not. I found
>>> this class very helpful in finding the error that was causing my problem
>>> for introspection. You could add the feeds test to the
>>> repeatedtestsuite.xml and try running it. The process might help you cause
>>> the error locally.
>>> 
>>> https://github.com/ecarm002/incubator-asterixdb/tree/ecarm002/introspection_alternate
>>> 
>>> edu.uci.ics.asterix.test.runtime.RepteatedTest
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Jul 6, 2015 at 8:25 PM, Ian Maxon <im...@uci.edu> wrote:
>>> 
>>>> Raman and I worked on getting to the root of what is causing the build
>>>> instability for a while today. The investigation is still ongoing but
>>>> so far we've discovered the following things:
>>>> 
>>>> - The OOM error specifically is running out of threads to create on
>>>> the machine, which is odd. We aren't creating more than 500 threads
>>>> per JVM during testing so this is especially puzzling. The heap size
>>>> or permgen size are not the issue.
>>>> 
>>>> - The OOM error can be observed at the point where only feeds was
>>>> merged (and not YARN or the managix scripting fix)
>>>> 
>>>> - Neither of us can reproduce this locally on our development
>>>> machines. It seems that the environment is a variable in this issue
>>>> (hitting the thread limit on the machine), somehow.
>>>> 
>>>> - Where or if the tests run out of threads is not deterministic. It
>>>> tends to fail around the feeds portion of the execution tests, but
>>>> this is only a loose pattern. They can all pass, or the OOM can be hit
>>>> during integration tests, or other totally unrelated execution tests.
>>>> 
>>>> - There are a few feeds tests which sometimes fail (namely issue_711
>>>> and feeds_10) but this is totally unrelated to the more major issues
>>>> of running out of threads on the build machine.
>>>> 
>>>> Given all the above, it looks like there is at least a degree of
>>>> configuration/environmental influence on this issue.
>>>> 
>>>> - Ian
>>>> 
>>>> 
>>>> 
>>>> On Mon, Jul 6, 2015 at 2:14 PM, Raman Grover <ra...@gmail.com>
>>>> wrote:
>>>>> Hi
>>>>> 
>>>>> a) The two big commits to the master (YARN integration and feeds)
>>>> happened
>>>>> as atomic units that makes it easier to
>>>>> reset the master to the version prior to each feature and verify if the
>>>>> build began showing OOM after each of the suspected commits. We have a
>>>>> pretty deterministic way of nailing down the commit that introduced the
>>>>> problem. I would suggest, instead of disabling the feeds tests, can we
>>>>> revert to the earlier commit and confirm if the feeds commit did
>>>> introduce
>>>>> the behavior and repeat the test with the YARN commit that followed. We
>>>>> should be able to see sudden increase/drop in build stability by running
>>>>> sufficient number of iterations.
>>>>> 
>>>>> b) I have not been able to reproduce the OOM at my setup where I have
>>>> been
>>>>> running the build repeatedly.
>>>>> @Ian are you able to reproduce it at your system? May be I am not running
>>>>> the build sufficient number of times?
>>>>> I am still not able to understand how removal of test cases still causes
>>>>> the OOM? I can go back and look at the precise changes made during the
>>>>> feeds commit that could introduce OOM even if feeds are not involved at
>>>>> all, but as I see it, the changes made do not play a role if feeds are
>>>> not
>>>>> being ingested.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Raman
>>>>> 
>>>>> 
>>>>> On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:
>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> We are close to having a release ready, but there's a few things left
>>>>>> on the checklist before we can cut the first Apache release. I think
>>>>>> most things on this list are underway, but I'll put them here just for
>>>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>>> 
>>>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>>>> declined. It's hard to get a build to go through to the end without
>>>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>>>> related to Feeds, but even after disabling the tests
>>>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>>>>>> Therefore I am not precisely sure what is going on, but it only
>>>>>> started to happen after we merged those two features. It's not exactly
>>>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>>>> be great to get your advice/thoughts on this.
>>>>>> 
>>>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>>>> underway by Ildar.
>>>>>> 
>>>>>> - The repackaging and license checker patches still need to be merged
>>>>>> in, but this should happen after the above two features are merged.
>>>>>> They are otherwise ready for review though.
>>>>>> 
>>>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>>>> the new version that has been in draft form for a few weeks now.
>>>>>> Before it may have been a little premature, but now it should be
>>>>>> accurate. The documentation site should also be reverted to its prior
>>>>>> state, before it was quickly patched to serve as an interim website.
>>>>>> 
>>>>>> 
>>>>>> If there's anything else I am missing that should be in this list,
>>>>>> please feel free to add it into this thread.
>>>>>> 
>>>>>> Thanks,
>>>>>> -Ian
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Raman
> 

Best regards,
Ildar

Re: Tasks remaining for release

[Mike Carey <dt...@gmail.com>]

Wierd....  That many threads seems wrong.....

On 7/7/15 8:35 PM, Ian Maxon wrote:
> I think I have at least a workaround to the thread starvation nailed
> down. We'll have to see, but basically I think the latest few patches
> cause us to use more threads for whatever reason- and this pushed us
> over the default thread cap in many circumstances (not always). Going
> ahead and setting the number of processes to be unlimited within the
> build server and containers seems to have put out the fire, so to
> speak. Another confounding factor is the issue that docker containers
> run within the same host and hence also have their own shared thread
> limit, in addition to the host's thread limit. It's not clear to me
> however whether we intend to use that many threads (~500), or if
> there's a subtle resource leak somewhere.
>
> - Ian
>
> On Tue, Jul 7, 2015 at 5:44 PM, Eldon Carman <ec...@ucr.edu> wrote:
>> In my branch ("ecarm002/introspection_alternate"), I have adapted some code
>> I received from Ildar to repeatedly test a set of runtime tests. I am not
>> sure this testing process will be related to your issue or not. I found
>> this class very helpful in finding the error that was causing my problem
>> for introspection. You could add the feeds test to the
>> repeatedtestsuite.xml and try running it. The process might help you cause
>> the error locally.
>>
>> https://github.com/ecarm002/incubator-asterixdb/tree/ecarm002/introspection_alternate
>>
>> edu.uci.ics.asterix.test.runtime.RepteatedTest
>>
>>
>>
>>
>> On Mon, Jul 6, 2015 at 8:25 PM, Ian Maxon <im...@uci.edu> wrote:
>>
>>> Raman and I worked on getting to the root of what is causing the build
>>> instability for a while today. The investigation is still ongoing but
>>> so far we've discovered the following things:
>>>
>>> - The OOM error specifically is running out of threads to create on
>>> the machine, which is odd. We aren't creating more than 500 threads
>>> per JVM during testing so this is especially puzzling. The heap size
>>> or permgen size are not the issue.
>>>
>>> - The OOM error can be observed at the point where only feeds was
>>> merged (and not YARN or the managix scripting fix)
>>>
>>> - Neither of us can reproduce this locally on our development
>>> machines. It seems that the environment is a variable in this issue
>>> (hitting the thread limit on the machine), somehow.
>>>
>>> - Where or if the tests run out of threads is not deterministic. It
>>> tends to fail around the feeds portion of the execution tests, but
>>> this is only a loose pattern. They can all pass, or the OOM can be hit
>>> during integration tests, or other totally unrelated execution tests.
>>>
>>> - There are a few feeds tests which sometimes fail (namely issue_711
>>> and feeds_10) but this is totally unrelated to the more major issues
>>> of running out of threads on the build machine.
>>>
>>> Given all the above, it looks like there is at least a degree of
>>> configuration/environmental influence on this issue.
>>>
>>> - Ian
>>>
>>>
>>>
>>> On Mon, Jul 6, 2015 at 2:14 PM, Raman Grover <ra...@gmail.com>
>>> wrote:
>>>> Hi
>>>>
>>>> a) The two big commits to the master (YARN integration and feeds)
>>> happened
>>>> as atomic units that makes it easier to
>>>> reset the master to the version prior to each feature and verify if the
>>>> build began showing OOM after each of the suspected commits. We have a
>>>> pretty deterministic way of nailing down the commit that introduced the
>>>> problem. I would suggest, instead of disabling the feeds tests, can we
>>>> revert to the earlier commit and confirm if the feeds commit did
>>> introduce
>>>> the behavior and repeat the test with the YARN commit that followed. We
>>>> should be able to see sudden increase/drop in build stability by running
>>>> sufficient number of iterations.
>>>>
>>>> b) I have not been able to reproduce the OOM at my setup where I have
>>> been
>>>> running the build repeatedly.
>>>> @Ian are you able to reproduce it at your system? May be I am not running
>>>> the build sufficient number of times?
>>>> I am still not able to understand how removal of test cases still causes
>>>> the OOM? I can go back and look at the precise changes made during the
>>>> feeds commit that could introduce OOM even if feeds are not involved at
>>>> all, but as I see it, the changes made do not play a role if feeds are
>>> not
>>>> being ingested.
>>>>
>>>>
>>>> Regards,
>>>> Raman
>>>>
>>>>
>>>> On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> We are close to having a release ready, but there's a few things left
>>>>> on the checklist before we can cut the first Apache release. I think
>>>>> most things on this list are underway, but I'll put them here just for
>>>>> reference/visibility. Comments and thoughts are welcomed.
>>>>>
>>>>> - Build stability after merging YARN and Feeds seems to have seriously
>>>>> declined. It's hard to get a build to go through to the end without
>>>>> going OOM at all now honestly, so this is a Problem. I think it may be
>>>>> related to Feeds, but even after disabling the tests
>>>>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>>>>> Therefore I am not precisely sure what is going on, but it only
>>>>> started to happen after we merged those two features. It's not exactly
>>>>> obvious to me where the memory leak is coming from. @Raman, it would
>>>>> be great to get your advice/thoughts on this.
>>>>>
>>>>> - Metadata name changes and Metadata caching consistency fixes are
>>>>> underway by Ildar.
>>>>>
>>>>> - The repackaging and license checker patches still need to be merged
>>>>> in, but this should happen after the above two features are merged.
>>>>> They are otherwise ready for review though.
>>>>>
>>>>> - Now that Feeds is merged, the Apache website should be changed to
>>>>> the new version that has been in draft form for a few weeks now.
>>>>> Before it may have been a little premature, but now it should be
>>>>> accurate. The documentation site should also be reverted to its prior
>>>>> state, before it was quickly patched to serve as an interim website.
>>>>>
>>>>>
>>>>> If there's anything else I am missing that should be in this list,
>>>>> please feel free to add it into this thread.
>>>>>
>>>>> Thanks,
>>>>> -Ian
>>>>>
>>>>
>>>>
>>>> --
>>>> Raman

Re: Tasks remaining for release

[Raman Grover <ra...@gmail.com>]

The OOM at Jenkins is of a particular kind - "unable to create native
thread" (different from "OutOfMemoryError Java Heap Space") which means the
stack associated with a thread could not be allocated. The thread stack is
not allocated on the heap. So heap size is immaterial is such OOM error.
(this can be confirmed by setting -XX:+HeapDumpOnOutOfMemoryError
<http://www.oracle.com/technetwork/java/javase/clopts-139448.html#gbzrr> and
verifying the occupied heap from the dump.

Though stack size for a thread can be configured (-Xss) but I don't think
we should fiddle with it.

Ian, do you know what has been the corresponding number of threads from
previous stable builds?

Regards,
Raman

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

I think I have at least a workaround to the thread starvation nailed
down. We'll have to see, but basically I think the latest few patches
cause us to use more threads for whatever reason- and this pushed us
over the default thread cap in many circumstances (not always). Going
ahead and setting the number of processes to be unlimited within the
build server and containers seems to have put out the fire, so to
speak. Another confounding factor is the issue that docker containers
run within the same host and hence also have their own shared thread
limit, in addition to the host's thread limit. It's not clear to me
however whether we intend to use that many threads (~500), or if
there's a subtle resource leak somewhere.

- Ian

On Tue, Jul 7, 2015 at 5:44 PM, Eldon Carman <ec...@ucr.edu> wrote:
> In my branch ("ecarm002/introspection_alternate"), I have adapted some code
> I received from Ildar to repeatedly test a set of runtime tests. I am not
> sure this testing process will be related to your issue or not. I found
> this class very helpful in finding the error that was causing my problem
> for introspection. You could add the feeds test to the
> repeatedtestsuite.xml and try running it. The process might help you cause
> the error locally.
>
> https://github.com/ecarm002/incubator-asterixdb/tree/ecarm002/introspection_alternate
>
> edu.uci.ics.asterix.test.runtime.RepteatedTest
>
>
>
>
> On Mon, Jul 6, 2015 at 8:25 PM, Ian Maxon <im...@uci.edu> wrote:
>
>> Raman and I worked on getting to the root of what is causing the build
>> instability for a while today. The investigation is still ongoing but
>> so far we've discovered the following things:
>>
>> - The OOM error specifically is running out of threads to create on
>> the machine, which is odd. We aren't creating more than 500 threads
>> per JVM during testing so this is especially puzzling. The heap size
>> or permgen size are not the issue.
>>
>> - The OOM error can be observed at the point where only feeds was
>> merged (and not YARN or the managix scripting fix)
>>
>> - Neither of us can reproduce this locally on our development
>> machines. It seems that the environment is a variable in this issue
>> (hitting the thread limit on the machine), somehow.
>>
>> - Where or if the tests run out of threads is not deterministic. It
>> tends to fail around the feeds portion of the execution tests, but
>> this is only a loose pattern. They can all pass, or the OOM can be hit
>> during integration tests, or other totally unrelated execution tests.
>>
>> - There are a few feeds tests which sometimes fail (namely issue_711
>> and feeds_10) but this is totally unrelated to the more major issues
>> of running out of threads on the build machine.
>>
>> Given all the above, it looks like there is at least a degree of
>> configuration/environmental influence on this issue.
>>
>> - Ian
>>
>>
>>
>> On Mon, Jul 6, 2015 at 2:14 PM, Raman Grover <ra...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > a) The two big commits to the master (YARN integration and feeds)
>> happened
>> > as atomic units that makes it easier to
>> > reset the master to the version prior to each feature and verify if the
>> > build began showing OOM after each of the suspected commits. We have a
>> > pretty deterministic way of nailing down the commit that introduced the
>> > problem. I would suggest, instead of disabling the feeds tests, can we
>> > revert to the earlier commit and confirm if the feeds commit did
>> introduce
>> > the behavior and repeat the test with the YARN commit that followed. We
>> > should be able to see sudden increase/drop in build stability by running
>> > sufficient number of iterations.
>> >
>> > b) I have not been able to reproduce the OOM at my setup where I have
>> been
>> > running the build repeatedly.
>> > @Ian are you able to reproduce it at your system? May be I am not running
>> > the build sufficient number of times?
>> > I am still not able to understand how removal of test cases still causes
>> > the OOM? I can go back and look at the precise changes made during the
>> > feeds commit that could introduce OOM even if feeds are not involved at
>> > all, but as I see it, the changes made do not play a role if feeds are
>> not
>> > being ingested.
>> >
>> >
>> > Regards,
>> > Raman
>> >
>> >
>> > On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:
>> >
>> >> Hi all,
>> >>
>> >> We are close to having a release ready, but there's a few things left
>> >> on the checklist before we can cut the first Apache release. I think
>> >> most things on this list are underway, but I'll put them here just for
>> >> reference/visibility. Comments and thoughts are welcomed.
>> >>
>> >> - Build stability after merging YARN and Feeds seems to have seriously
>> >> declined. It's hard to get a build to go through to the end without
>> >> going OOM at all now honestly, so this is a Problem. I think it may be
>> >> related to Feeds, but even after disabling the tests
>> >> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>> >> Therefore I am not precisely sure what is going on, but it only
>> >> started to happen after we merged those two features. It's not exactly
>> >> obvious to me where the memory leak is coming from. @Raman, it would
>> >> be great to get your advice/thoughts on this.
>> >>
>> >> - Metadata name changes and Metadata caching consistency fixes are
>> >> underway by Ildar.
>> >>
>> >> - The repackaging and license checker patches still need to be merged
>> >> in, but this should happen after the above two features are merged.
>> >> They are otherwise ready for review though.
>> >>
>> >> - Now that Feeds is merged, the Apache website should be changed to
>> >> the new version that has been in draft form for a few weeks now.
>> >> Before it may have been a little premature, but now it should be
>> >> accurate. The documentation site should also be reverted to its prior
>> >> state, before it was quickly patched to serve as an interim website.
>> >>
>> >>
>> >> If there's anything else I am missing that should be in this list,
>> >> please feel free to add it into this thread.
>> >>
>> >> Thanks,
>> >> -Ian
>> >>
>> >
>> >
>> >
>> > --
>> > Raman
>>

Re: Tasks remaining for release

[Eldon Carman <ec...@ucr.edu>]

In my branch ("ecarm002/introspection_alternate"), I have adapted some code
I received from Ildar to repeatedly test a set of runtime tests. I am not
sure this testing process will be related to your issue or not. I found
this class very helpful in finding the error that was causing my problem
for introspection. You could add the feeds test to the
repeatedtestsuite.xml and try running it. The process might help you cause
the error locally.

https://github.com/ecarm002/incubator-asterixdb/tree/ecarm002/introspection_alternate

edu.uci.ics.asterix.test.runtime.RepteatedTest




On Mon, Jul 6, 2015 at 8:25 PM, Ian Maxon <im...@uci.edu> wrote:

> Raman and I worked on getting to the root of what is causing the build
> instability for a while today. The investigation is still ongoing but
> so far we've discovered the following things:
>
> - The OOM error specifically is running out of threads to create on
> the machine, which is odd. We aren't creating more than 500 threads
> per JVM during testing so this is especially puzzling. The heap size
> or permgen size are not the issue.
>
> - The OOM error can be observed at the point where only feeds was
> merged (and not YARN or the managix scripting fix)
>
> - Neither of us can reproduce this locally on our development
> machines. It seems that the environment is a variable in this issue
> (hitting the thread limit on the machine), somehow.
>
> - Where or if the tests run out of threads is not deterministic. It
> tends to fail around the feeds portion of the execution tests, but
> this is only a loose pattern. They can all pass, or the OOM can be hit
> during integration tests, or other totally unrelated execution tests.
>
> - There are a few feeds tests which sometimes fail (namely issue_711
> and feeds_10) but this is totally unrelated to the more major issues
> of running out of threads on the build machine.
>
> Given all the above, it looks like there is at least a degree of
> configuration/environmental influence on this issue.
>
> - Ian
>
>
>
> On Mon, Jul 6, 2015 at 2:14 PM, Raman Grover <ra...@gmail.com>
> wrote:
> > Hi
> >
> > a) The two big commits to the master (YARN integration and feeds)
> happened
> > as atomic units that makes it easier to
> > reset the master to the version prior to each feature and verify if the
> > build began showing OOM after each of the suspected commits. We have a
> > pretty deterministic way of nailing down the commit that introduced the
> > problem. I would suggest, instead of disabling the feeds tests, can we
> > revert to the earlier commit and confirm if the feeds commit did
> introduce
> > the behavior and repeat the test with the YARN commit that followed. We
> > should be able to see sudden increase/drop in build stability by running
> > sufficient number of iterations.
> >
> > b) I have not been able to reproduce the OOM at my setup where I have
> been
> > running the build repeatedly.
> > @Ian are you able to reproduce it at your system? May be I am not running
> > the build sufficient number of times?
> > I am still not able to understand how removal of test cases still causes
> > the OOM? I can go back and look at the precise changes made during the
> > feeds commit that could introduce OOM even if feeds are not involved at
> > all, but as I see it, the changes made do not play a role if feeds are
> not
> > being ingested.
> >
> >
> > Regards,
> > Raman
> >
> >
> > On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:
> >
> >> Hi all,
> >>
> >> We are close to having a release ready, but there's a few things left
> >> on the checklist before we can cut the first Apache release. I think
> >> most things on this list are underway, but I'll put them here just for
> >> reference/visibility. Comments and thoughts are welcomed.
> >>
> >> - Build stability after merging YARN and Feeds seems to have seriously
> >> declined. It's hard to get a build to go through to the end without
> >> going OOM at all now honestly, so this is a Problem. I think it may be
> >> related to Feeds, but even after disabling the tests
> >> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> >> Therefore I am not precisely sure what is going on, but it only
> >> started to happen after we merged those two features. It's not exactly
> >> obvious to me where the memory leak is coming from. @Raman, it would
> >> be great to get your advice/thoughts on this.
> >>
> >> - Metadata name changes and Metadata caching consistency fixes are
> >> underway by Ildar.
> >>
> >> - The repackaging and license checker patches still need to be merged
> >> in, but this should happen after the above two features are merged.
> >> They are otherwise ready for review though.
> >>
> >> - Now that Feeds is merged, the Apache website should be changed to
> >> the new version that has been in draft form for a few weeks now.
> >> Before it may have been a little premature, but now it should be
> >> accurate. The documentation site should also be reverted to its prior
> >> state, before it was quickly patched to serve as an interim website.
> >>
> >>
> >> If there's anything else I am missing that should be in this list,
> >> please feel free to add it into this thread.
> >>
> >> Thanks,
> >> -Ian
> >>
> >
> >
> >
> > --
> > Raman
>

Re: Tasks remaining for release

[Ian Maxon <im...@uci.edu>]

Raman and I worked on getting to the root of what is causing the build
instability for a while today. The investigation is still ongoing but
so far we've discovered the following things:

- The OOM error specifically is running out of threads to create on
the machine, which is odd. We aren't creating more than 500 threads
per JVM during testing so this is especially puzzling. The heap size
or permgen size are not the issue.

- The OOM error can be observed at the point where only feeds was
merged (and not YARN or the managix scripting fix)

- Neither of us can reproduce this locally on our development
machines. It seems that the environment is a variable in this issue
(hitting the thread limit on the machine), somehow.

- Where or if the tests run out of threads is not deterministic. It
tends to fail around the feeds portion of the execution tests, but
this is only a loose pattern. They can all pass, or the OOM can be hit
during integration tests, or other totally unrelated execution tests.

- There are a few feeds tests which sometimes fail (namely issue_711
and feeds_10) but this is totally unrelated to the more major issues
of running out of threads on the build machine.

Given all the above, it looks like there is at least a degree of
configuration/environmental influence on this issue.

- Ian



On Mon, Jul 6, 2015 at 2:14 PM, Raman Grover <ra...@gmail.com> wrote:
> Hi
>
> a) The two big commits to the master (YARN integration and feeds) happened
> as atomic units that makes it easier to
> reset the master to the version prior to each feature and verify if the
> build began showing OOM after each of the suspected commits. We have a
> pretty deterministic way of nailing down the commit that introduced the
> problem. I would suggest, instead of disabling the feeds tests, can we
> revert to the earlier commit and confirm if the feeds commit did introduce
> the behavior and repeat the test with the YARN commit that followed. We
> should be able to see sudden increase/drop in build stability by running
> sufficient number of iterations.
>
> b) I have not been able to reproduce the OOM at my setup where I have been
> running the build repeatedly.
> @Ian are you able to reproduce it at your system? May be I am not running
> the build sufficient number of times?
> I am still not able to understand how removal of test cases still causes
> the OOM? I can go back and look at the precise changes made during the
> feeds commit that could introduce OOM even if feeds are not involved at
> all, but as I see it, the changes made do not play a role if feeds are not
> being ingested.
>
>
> Regards,
> Raman
>
>
> On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:
>
>> Hi all,
>>
>> We are close to having a release ready, but there's a few things left
>> on the checklist before we can cut the first Apache release. I think
>> most things on this list are underway, but I'll put them here just for
>> reference/visibility. Comments and thoughts are welcomed.
>>
>> - Build stability after merging YARN and Feeds seems to have seriously
>> declined. It's hard to get a build to go through to the end without
>> going OOM at all now honestly, so this is a Problem. I think it may be
>> related to Feeds, but even after disabling the tests
>> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
>> Therefore I am not precisely sure what is going on, but it only
>> started to happen after we merged those two features. It's not exactly
>> obvious to me where the memory leak is coming from. @Raman, it would
>> be great to get your advice/thoughts on this.
>>
>> - Metadata name changes and Metadata caching consistency fixes are
>> underway by Ildar.
>>
>> - The repackaging and license checker patches still need to be merged
>> in, but this should happen after the above two features are merged.
>> They are otherwise ready for review though.
>>
>> - Now that Feeds is merged, the Apache website should be changed to
>> the new version that has been in draft form for a few weeks now.
>> Before it may have been a little premature, but now it should be
>> accurate. The documentation site should also be reverted to its prior
>> state, before it was quickly patched to serve as an interim website.
>>
>>
>> If there's anything else I am missing that should be in this list,
>> please feel free to add it into this thread.
>>
>> Thanks,
>> -Ian
>>
>
>
>
> --
> Raman

Re: Tasks remaining for release

[Raman Grover <ra...@gmail.com>]

Hi

a) The two big commits to the master (YARN integration and feeds) happened
as atomic units that makes it easier to
reset the master to the version prior to each feature and verify if the
build began showing OOM after each of the suspected commits. We have a
pretty deterministic way of nailing down the commit that introduced the
problem. I would suggest, instead of disabling the feeds tests, can we
revert to the earlier commit and confirm if the feeds commit did introduce
the behavior and repeat the test with the YARN commit that followed. We
should be able to see sudden increase/drop in build stability by running
sufficient number of iterations.

b) I have not been able to reproduce the OOM at my setup where I have been
running the build repeatedly.
@Ian are you able to reproduce it at your system? May be I am not running
the build sufficient number of times?
I am still not able to understand how removal of test cases still causes
the OOM? I can go back and look at the precise changes made during the
feeds commit that could introduce OOM even if feeds are not involved at
all, but as I see it, the changes made do not play a role if feeds are not
being ingested.


Regards,
Raman


On Thu, Jul 2, 2015 at 6:42 PM, Ian Maxon <im...@uci.edu> wrote:

> Hi all,
>
> We are close to having a release ready, but there's a few things left
> on the checklist before we can cut the first Apache release. I think
> most things on this list are underway, but I'll put them here just for
> reference/visibility. Comments and thoughts are welcomed.
>
> - Build stability after merging YARN and Feeds seems to have seriously
> declined. It's hard to get a build to go through to the end without
> going OOM at all now honestly, so this is a Problem. I think it may be
> related to Feeds, but even after disabling the tests
> (https://asterix-gerrit.ics.uci.edu/#/c/312/), I still see it.
> Therefore I am not precisely sure what is going on, but it only
> started to happen after we merged those two features. It's not exactly
> obvious to me where the memory leak is coming from. @Raman, it would
> be great to get your advice/thoughts on this.
>
> - Metadata name changes and Metadata caching consistency fixes are
> underway by Ildar.
>
> - The repackaging and license checker patches still need to be merged
> in, but this should happen after the above two features are merged.
> They are otherwise ready for review though.
>
> - Now that Feeds is merged, the Apache website should be changed to
> the new version that has been in draft form for a few weeks now.
> Before it may have been a little premature, but now it should be
> accurate. The documentation site should also be reverted to its prior
> state, before it was quickly patched to serve as an interim website.
>
>
> If there's anything else I am missing that should be in this list,
> please feel free to add it into this thread.
>
> Thanks,
> -Ian
>



-- 
Raman