Re: Using GitHub Actions for Apache Hudi repo

[Chris Lambertus <cm...@apache.org>]

> On Jan 23, 2021, at 10:15 PM, Jarek Potiuk <ja...@potiuk.com> wrote:

> Let me explain then what I see as the current state of Github Actions.
> 
> And note to Infra:
> 
> If you think it might be helpful, I am happy to Capture it all as a Wiki page - with your comments and such - so that other projects have some information about it). And please correct me if any of the statements below are wrong. 
> Those are my personal views on the matter, and as I have no full insight into agreements of ASF and maybe things are changing in the background without us knowing I might not know the latest "state of affairs”.

Trimmed CC, add builds@ —



Yes, I think this is valuable information to capture. I don’t see that we really have a community editable wiki space dedicated to builds. We have some Infra pages and documentation here and there, some basic info and links on https://ci.apache.org <https://ci.apache.org/>, but perhaps we could set up a more broad community-based wiki to address the issues that come up on the builds@ list and in other places.

Jarek, if you would be so kind, please open an Infra jira ticket requesting a CI community wiki space. I think there’s a lot of value here, but it requires further discussion, and I don’t want it to get lost in the myriad list emails we all receive.

Thanks,
Chris
ASF Infra

Re: Using GitHub Actions for Apache Hudi repo

[Jarek Potiuk <ja...@potiuk.com>]

For everyone who is interested, a quick update of what I know is going on).

I am happy to see that there is significant effort from many parties that
want to help with the whole situation. That's very encouraging :).

* Both security issues I raised via bounty have been triaged - in one of
them I have information that GitHub has already an idea how to fix it and
they are working on it. I will let you all know when they come back to me.
* We are working together with the Apache Pulsar team on optimizing their
workflow (Pulsar seems to contribute a lot of workflow runs to the queue).
There is an issue
<https://github.com/apache/pulsar/pull/9159#issuecomment-769242902> and design
doc
<https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing>
where
we discuss options on how their workflows can be optimised - I provided a
few suggestions and learning from Apache Airflow.
* Kamil Breguła from Apache Airflow, opened a number of PRs to implement
"Cancel Workflow Runs" action (in Pulsar
<https://github.com/apache/pulsar/pull/9159>, Spark
<https://github.com/apache/spark/pull/31104>, Pinot
<https://github.com/apache/incubator-pinot/pull/6429> for example). Some of
them merged, some of them were turned on/off as the "agressive" behaviour
of it was too aggressive (but I helped with fine-tuning).
* We also actively help Apache SuperSet in implementing a working "Cancel
duplicates" workflow - rather than using my cancel action they decided to
use a custom python code to do the same
https://github.com/apache/superset/pull/12625/  - this might be useful for
others as it also allows to run the script manually by committers if you
want to force canceling duplicates
* We are close to have a running POC with self-hosted runners in Apache
Airflow, but it is quite an effort as we initially anticipated. We will let
you know when we have them up and running and hopefully we can come up with
a reusable approach for other ASF projects.

If any of the projects would like to optimise their workflows - I am happy
to help. Also if there are any other ideas how the current situation can be
improved, you are most welcome. We are in this together :).

J.


On Sun, Jan 31, 2021 at 11:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> The space is created (thnks infra!), I turned my message into this page
> describing the status. I hope it will be useful.
>
> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
>
> If you feel you can improve it or correct any omissions there - feel free
> to modify it directly !
>
> J.
>
>
> On Sun, Jan 31, 2021 at 9:06 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Ticket created:
>> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21364
>>
>> J.
>>
>> On Thu, Jan 28, 2021 at 5:59 AM Chris Lambertus <cm...@apache.org> wrote:
>>
>>>
>>>
>>> > On Jan 23, 2021, at 10:15 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> > Let me explain then what I see as the current state of Github Actions.
>>> >
>>> > And note to Infra:
>>> >
>>> > If you think it might be helpful, I am happy to Capture it all as a
>>> Wiki page - with your comments and such - so that other projects have some
>>> information about it). And please correct me if any of the statements below
>>> are wrong.
>>> > Those are my personal views on the matter, and as I have no full
>>> insight into agreements of ASF and maybe things are changing in the
>>> background without us knowing I might not know the latest "state of
>>> affairs”.
>>>
>>> Trimmed CC, add builds@ —
>>>
>>>
>>>
>>> Yes, I think this is valuable information to capture. I don’t see that
>>> we really have a community editable wiki space dedicated to builds. We have
>>> some Infra pages and documentation here and there, some basic info and
>>> links on https://ci.apache.org <https://ci.apache.org/>, but perhaps we
>>> could set up a more broad community-based wiki to address the issues that
>>> come up on the builds@ list and in other places.
>>>
>>> Jarek, if you would be so kind, please open an Infra jira ticket
>>> requesting a CI community wiki space. I think there’s a lot of value here,
>>> but it requires further discussion, and I don’t want it to get lost in the
>>> myriad list emails we all receive.
>>>
>>> Thanks,
>>> Chris
>>> ASF Infra
>>>
>>>
>>
>> --
>> +48 660 796 129
>>
>
>
> --
> +48 660 796 129
>


-- 
+48 660 796 129

Re: Using GitHub Actions for Apache Hudi repo

[Jarek Potiuk <ja...@potiuk.com>]

The space is created (thnks infra!), I turned my message into this page
describing the status. I hope it will be useful.

https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status

If you feel you can improve it or correct any omissions there - feel free
to modify it directly !

J.


On Sun, Jan 31, 2021 at 9:06 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Ticket created:
> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21364
>
> J.
>
> On Thu, Jan 28, 2021 at 5:59 AM Chris Lambertus <cm...@apache.org> wrote:
>
>>
>>
>> > On Jan 23, 2021, at 10:15 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > Let me explain then what I see as the current state of Github Actions.
>> >
>> > And note to Infra:
>> >
>> > If you think it might be helpful, I am happy to Capture it all as a
>> Wiki page - with your comments and such - so that other projects have some
>> information about it). And please correct me if any of the statements below
>> are wrong.
>> > Those are my personal views on the matter, and as I have no full
>> insight into agreements of ASF and maybe things are changing in the
>> background without us knowing I might not know the latest "state of
>> affairs”.
>>
>> Trimmed CC, add builds@ —
>>
>>
>>
>> Yes, I think this is valuable information to capture. I don’t see that we
>> really have a community editable wiki space dedicated to builds. We have
>> some Infra pages and documentation here and there, some basic info and
>> links on https://ci.apache.org <https://ci.apache.org/>, but perhaps we
>> could set up a more broad community-based wiki to address the issues that
>> come up on the builds@ list and in other places.
>>
>> Jarek, if you would be so kind, please open an Infra jira ticket
>> requesting a CI community wiki space. I think there’s a lot of value here,
>> but it requires further discussion, and I don’t want it to get lost in the
>> myriad list emails we all receive.
>>
>> Thanks,
>> Chris
>> ASF Infra
>>
>>
>
> --
> +48 660 796 129
>


-- 
+48 660 796 129

Re: Using GitHub Actions for Apache Hudi repo

[Jarek Potiuk <ja...@potiuk.com>]

Ticket created:
https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21364

J.

On Thu, Jan 28, 2021 at 5:59 AM Chris Lambertus <cm...@apache.org> wrote:

>
>
> > On Jan 23, 2021, at 10:15 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Let me explain then what I see as the current state of Github Actions.
> >
> > And note to Infra:
> >
> > If you think it might be helpful, I am happy to Capture it all as a Wiki
> page - with your comments and such - so that other projects have some
> information about it). And please correct me if any of the statements below
> are wrong.
> > Those are my personal views on the matter, and as I have no full insight
> into agreements of ASF and maybe things are changing in the background
> without us knowing I might not know the latest "state of affairs”.
>
> Trimmed CC, add builds@ —
>
>
>
> Yes, I think this is valuable information to capture. I don’t see that we
> really have a community editable wiki space dedicated to builds. We have
> some Infra pages and documentation here and there, some basic info and
> links on https://ci.apache.org <https://ci.apache.org/>, but perhaps we
> could set up a more broad community-based wiki to address the issues that
> come up on the builds@ list and in other places.
>
> Jarek, if you would be so kind, please open an Infra jira ticket
> requesting a CI community wiki space. I think there’s a lot of value here,
> but it requires further discussion, and I don’t want it to get lost in the
> myriad list emails we all receive.
>
> Thanks,
> Chris
> ASF Infra
>
>

-- 
+48 660 796 129