You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Karan Kumar <ka...@gmail.com> on 2023/02/15 21:38:47 UTC
[Discuss] S3 buckets or IT tests
Hey Folks
S3 read write tests currently are not executed in github actions since we
do not have public creds to read/write from s3.
Have raised an ASF infra ticket
https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
so that they can give us a bucket.
They require a PMC approval.
As discussed on this slack thread
https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839,
starting a formal discussion around it.
The objective would be to integrate our s3 IT's tests to run on each PR.
--
Thanks
Karan
Re: [Discuss] S3 buckets or IT tests
Posted by Karan Kumar <ka...@gmail.com>.
Thanks everyone. I have closed the ticket
https://issues.apache.org/jira/browse/INFRA-23952 since min.io seems like
the path ahead.
On Tue, Mar 7, 2023 at 12:24 AM Austin Bennett <au...@apache.org> wrote:
> +1 for Minio to keep things sufficiently simple. Full functional testing [
> ex: using s3 ] is great. But, it doesn't seem critical for this use case?
>
>
> On Sun, Mar 5, 2023, 8:27 PM Abhishek Agarwal <ab...@apache.org> wrote:
>
> > I would prefer using something like min.io than running tests against
> S3.
> > Primarily because tests no longer remain portable if we use S3. Which
> also
> > makes them harder to debug. Setting up minio using GHA seems
> > straightforward (https://github.com/mozilla/sccache/pull/1513/files).
> >
> > On Mon, Mar 6, 2023 at 7:01 AM Karan Kumar <ka...@gmail.com>
> > wrote:
> >
> > > Oh yes, https://issues.apache.org/jira/browse/INFRA-23952 Is the
> ticket
> > I
> > > am talking about. As the INFRA folks require approval to proceed, can
> one
> > > of the PMC's drop a comment regarding approval/rejection on the same
> > jira.
> > > If we are not able to procure a bucket, then we can start exploring
> mini
> > > IO.
> > >
> > > On Wed, Feb 22, 2023 at 6:07 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > I think the ticket you're referring to is
> > > > https://issues.apache.org/jira/browse/INFRA-23952.
> > > >
> > > > It would definitely be valuable to run S3 integration tests as part
> of
> > > the
> > > > automated test suite in GitHub Actions. If Infra is willing to
> provide
> > a
> > > > bucket for this purpose then we would certainly be able to use that.
> I
> > > bet
> > > > we could also use Minio (https://min.io/) or something similar.
> > > >
> > > > Gian
> > > >
> > > > On 2023/02/15 21:38:47 Karan Kumar wrote:
> > > > > Hey Folks
> > > > > S3 read write tests currently are not executed in github actions
> > since
> > > we
> > > > > do not have public creds to read/write from s3.
> > > > > Have raised an ASF infra ticket
> > > > >
> > > >
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > > > > so that they can give us a bucket.
> > > > > They require a PMC approval.
> > > > > As discussed on this slack thread
> > > > >
> > > >
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > > > ,
> > > > > starting a formal discussion around it.
> > > > >
> > > > > The objective would be to integrate our s3 IT's tests to run on
> each
> > > PR.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks
> > > > > Karan
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > >
> > > >
> > >
> > > --
> > > Thanks
> > > Karan
> > >
> >
>
--
Thanks
Karan
Re: [Discuss] S3 buckets or IT tests
Posted by Austin Bennett <au...@apache.org>.
+1 for Minio to keep things sufficiently simple. Full functional testing [
ex: using s3 ] is great. But, it doesn't seem critical for this use case?
On Sun, Mar 5, 2023, 8:27 PM Abhishek Agarwal <ab...@apache.org> wrote:
> I would prefer using something like min.io than running tests against S3.
> Primarily because tests no longer remain portable if we use S3. Which also
> makes them harder to debug. Setting up minio using GHA seems
> straightforward (https://github.com/mozilla/sccache/pull/1513/files).
>
> On Mon, Mar 6, 2023 at 7:01 AM Karan Kumar <ka...@gmail.com>
> wrote:
>
> > Oh yes, https://issues.apache.org/jira/browse/INFRA-23952 Is the ticket
> I
> > am talking about. As the INFRA folks require approval to proceed, can one
> > of the PMC's drop a comment regarding approval/rejection on the same
> jira.
> > If we are not able to procure a bucket, then we can start exploring mini
> > IO.
> >
> > On Wed, Feb 22, 2023 at 6:07 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I think the ticket you're referring to is
> > > https://issues.apache.org/jira/browse/INFRA-23952.
> > >
> > > It would definitely be valuable to run S3 integration tests as part of
> > the
> > > automated test suite in GitHub Actions. If Infra is willing to provide
> a
> > > bucket for this purpose then we would certainly be able to use that. I
> > bet
> > > we could also use Minio (https://min.io/) or something similar.
> > >
> > > Gian
> > >
> > > On 2023/02/15 21:38:47 Karan Kumar wrote:
> > > > Hey Folks
> > > > S3 read write tests currently are not executed in github actions
> since
> > we
> > > > do not have public creds to read/write from s3.
> > > > Have raised an ASF infra ticket
> > > >
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > > > so that they can give us a bucket.
> > > > They require a PMC approval.
> > > > As discussed on this slack thread
> > > >
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > > ,
> > > > starting a formal discussion around it.
> > > >
> > > > The objective would be to integrate our s3 IT's tests to run on each
> > PR.
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks
> > > > Karan
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >
> > >
> >
> > --
> > Thanks
> > Karan
> >
>
Re: [Discuss] S3 buckets or IT tests
Posted by Abhishek Agarwal <ab...@apache.org>.
I would prefer using something like min.io than running tests against S3.
Primarily because tests no longer remain portable if we use S3. Which also
makes them harder to debug. Setting up minio using GHA seems
straightforward (https://github.com/mozilla/sccache/pull/1513/files).
On Mon, Mar 6, 2023 at 7:01 AM Karan Kumar <ka...@gmail.com> wrote:
> Oh yes, https://issues.apache.org/jira/browse/INFRA-23952 Is the ticket I
> am talking about. As the INFRA folks require approval to proceed, can one
> of the PMC's drop a comment regarding approval/rejection on the same jira.
> If we are not able to procure a bucket, then we can start exploring mini
> IO.
>
> On Wed, Feb 22, 2023 at 6:07 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I think the ticket you're referring to is
> > https://issues.apache.org/jira/browse/INFRA-23952.
> >
> > It would definitely be valuable to run S3 integration tests as part of
> the
> > automated test suite in GitHub Actions. If Infra is willing to provide a
> > bucket for this purpose then we would certainly be able to use that. I
> bet
> > we could also use Minio (https://min.io/) or something similar.
> >
> > Gian
> >
> > On 2023/02/15 21:38:47 Karan Kumar wrote:
> > > Hey Folks
> > > S3 read write tests currently are not executed in github actions since
> we
> > > do not have public creds to read/write from s3.
> > > Have raised an ASF infra ticket
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > > so that they can give us a bucket.
> > > They require a PMC approval.
> > > As discussed on this slack thread
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > ,
> > > starting a formal discussion around it.
> > >
> > > The objective would be to integrate our s3 IT's tests to run on each
> PR.
> > >
> > >
> > >
> > > --
> > > Thanks
> > > Karan
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
> >
> >
>
> --
> Thanks
> Karan
>
Re: [Discuss] S3 buckets or IT tests
Posted by Karan Kumar <ka...@gmail.com>.
Oh yes, https://issues.apache.org/jira/browse/INFRA-23952 Is the ticket I
am talking about. As the INFRA folks require approval to proceed, can one
of the PMC's drop a comment regarding approval/rejection on the same jira.
If we are not able to procure a bucket, then we can start exploring mini
IO.
On Wed, Feb 22, 2023 at 6:07 PM Gian Merlino <gi...@apache.org> wrote:
> I think the ticket you're referring to is
> https://issues.apache.org/jira/browse/INFRA-23952.
>
> It would definitely be valuable to run S3 integration tests as part of the
> automated test suite in GitHub Actions. If Infra is willing to provide a
> bucket for this purpose then we would certainly be able to use that. I bet
> we could also use Minio (https://min.io/) or something similar.
>
> Gian
>
> On 2023/02/15 21:38:47 Karan Kumar wrote:
> > Hey Folks
> > S3 read write tests currently are not executed in github actions since we
> > do not have public creds to read/write from s3.
> > Have raised an ASF infra ticket
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > so that they can give us a bucket.
> > They require a PMC approval.
> > As discussed on this slack thread
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> ,
> > starting a formal discussion around it.
> >
> > The objective would be to integrate our s3 IT's tests to run on each PR.
> >
> >
> >
> > --
> > Thanks
> > Karan
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>
--
Thanks
Karan
Re: [Discuss] S3 buckets or IT tests
Posted by Gian Merlino <gi...@apache.org>.
I think the ticket you're referring to is https://issues.apache.org/jira/browse/INFRA-23952.
It would definitely be valuable to run S3 integration tests as part of the automated test suite in GitHub Actions. If Infra is willing to provide a bucket for this purpose then we would certainly be able to use that. I bet we could also use Minio (https://min.io/) or something similar.
Gian
On 2023/02/15 21:38:47 Karan Kumar wrote:
> Hey Folks
> S3 read write tests currently are not executed in github actions since we
> do not have public creds to read/write from s3.
> Have raised an ASF infra ticket
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> so that they can give us a bucket.
> They require a PMC approval.
> As discussed on this slack thread
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839,
> starting a formal discussion around it.
>
> The objective would be to integrate our s3 IT's tests to run on each PR.
>
>
>
> --
> Thanks
> Karan
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org