You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Tyson Norris <tn...@adobe.com.INVALID> on 2018/10/02 16:07:26 UTC
Logstore usage during `activation get`
Hi –
I created this PR [1] due to noticing that `wsk activation get` does NOT return logs from a LogStore which stores logs outside of the Activation entity.
But it bring up a question of: Does IBM or any other operator who might use a custom LogStore desire to have logs included with `activation get`?
Currently returning logs is only possible using `wsk activation logs`
Personally, I think it is “nice” to have a separate explicit request for logs and activation metadata, and this is the way that the current OW Activation API operates with regards to an external LogStore (splunk, elk, etc), but after all it is inconsistent from the case where logs are NOT using external storage.
WDYT?
Thanks
Tyson
[1] https://github.com/apache/incubator-openwhisk/pull/4044
Re: Logstore usage during `activation get`
Posted by David P Grove <gr...@us.ibm.com>.
Nick Mitchell <mo...@gmail.com> wrote on 10/04/2018 12:06:14 PM:
>
> from a usability perspective, there is something quite attractive about
> having ready access to logs (and activation records generally)... while
> developing. i think most of the woes we encounter with logs come from
high
> volume actions (either exercised in prod, or during a high volume test)?
Nick, I agree with this, but I think it is separable from the question of
whether or not the logs are included in the activation API or if the
user/tooling needs to make a separate call to a logs API.
If we cleanly separated the two APIs, it might actually be easier to
provide higher-quality implementations of both of them by using the
appropriate backend technology for each.
--dave
>
> i realize that this gets us into a broader topic, but it would be nice if
> we could at least support ready access to say the last 1MB of activation
> information per user.
>
> otherwise, we are sacrificing developer experience at the altar of
service
> provider bottom line. don't get me wrong, i am sympathetic with the
> conundrum, but there are technical solutions that can satisfy both, if we
> put our minds to it.
>
> nick
>
> On Thu, Oct 4, 2018 at 11:49 AM James W Dubee <jw...@us.ibm.com> wrote:
>
> > Hey guys,
> >
> > For scalability concerns, I think it is preferable to separate logs
from
> > activation records entirely. Doing so would require users to use the
logs
> > API, or whatever tools are provided by the underly logging service. We
> > should focus on serverless technology and delegate log handling to
services
> > that specialize in such areas.
> >
> > From experimenting with storing logs in a different data store other
than
> > where activations are stored, I've seen that there may be intermittent
test
> > failures for tests that rely on data being present in an activation
> > record's log field. This may result when the activation record is
retrieved
> > before all the logs are stored in the separate log store.
> >
> > On a similar note, I don't think we should be storing activation
responses
> > at all.
> >
> > Regards,
> > James Dubee
> >
> >
> >
> > [image: Inactive hide details for Rodric Rabbah ---10/02/2018 07:30:22
> > PM---> By "break this" do you mean at some point we should remov]Rodric
> > Rabbah ---10/02/2018 07:30:22 PM---> By "break this" do you mean at
some
> > point we should remove the logs from the GET?
> >
> > From: Rodric Rabbah <ro...@gmail.com>
> > To: dev@openwhisk.apache.org
> > Date: 10/02/2018 07:30 PM
> > Subject: Re: Logstore usage during `activation get`
> > ------------------------------
> >
> >
> >
> > > By "break this" do you mean at some point we should remove the logs
from
> > the GET?
> >
> > Yes.
> >
> > @dubee thoughts? Since you've worked on the elastic plugin.
> >
> > -r
> >
> >
> >
> >
Re: Logstore usage during `activation get`
Posted by Nick Mitchell <mo...@gmail.com>.
from a usability perspective, there is something quite attractive about
having ready access to logs (and activation records generally)... while
developing. i think most of the woes we encounter with logs come from high
volume actions (either exercised in prod, or during a high volume test)?
i realize that this gets us into a broader topic, but it would be nice if
we could at least support ready access to say the last 1MB of activation
information per user.
otherwise, we are sacrificing developer experience at the altar of service
provider bottom line. don't get me wrong, i am sympathetic with the
conundrum, but there are technical solutions that can satisfy both, if we
put our minds to it.
nick
On Thu, Oct 4, 2018 at 11:49 AM James W Dubee <jw...@us.ibm.com> wrote:
> Hey guys,
>
> For scalability concerns, I think it is preferable to separate logs from
> activation records entirely. Doing so would require users to use the logs
> API, or whatever tools are provided by the underly logging service. We
> should focus on serverless technology and delegate log handling to services
> that specialize in such areas.
>
> From experimenting with storing logs in a different data store other than
> where activations are stored, I've seen that there may be intermittent test
> failures for tests that rely on data being present in an activation
> record's log field. This may result when the activation record is retrieved
> before all the logs are stored in the separate log store.
>
> On a similar note, I don't think we should be storing activation responses
> at all.
>
> Regards,
> James Dubee
>
>
>
> [image: Inactive hide details for Rodric Rabbah ---10/02/2018 07:30:22
> PM---> By "break this" do you mean at some point we should remov]Rodric
> Rabbah ---10/02/2018 07:30:22 PM---> By "break this" do you mean at some
> point we should remove the logs from the GET?
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 10/02/2018 07:30 PM
> Subject: Re: Logstore usage during `activation get`
> ------------------------------
>
>
>
> > By "break this" do you mean at some point we should remove the logs from
> the GET?
>
> Yes.
>
> @dubee thoughts? Since you've worked on the elastic plugin.
>
> -r
>
>
>
>
Re: Logstore usage during `activation get`
Posted by David P Grove <gr...@us.ibm.com>.
"James W Dubee" <jw...@us.ibm.com> wrote on 10/04/2018 11:45:34 AM:
> From: "James W Dubee" <jw...@us.ibm.com>
> To: dev@openwhisk.apache.org
> Date: 10/04/2018 11:49 AM
> Subject: Re: Logstore usage during `activation get`
>
> On a similar note, I don't think we should be storing activation
> responses at all.
>
+1 Responses can contain sensitive data; we should not be persisting them.
--dave
Re: Logstore usage during `activation get`
Posted by James W Dubee <jw...@us.ibm.com>.
Hey guys,
For scalability concerns, I think it is preferable to separate logs from
activation records entirely. Doing so would require users to use the logs
API, or whatever tools are provided by the underly logging service. We
should focus on serverless technology and delegate log handling to services
that specialize in such areas.
From experimenting with storing logs in a different data store other than
where activations are stored, I've seen that there may be intermittent test
failures for tests that rely on data being present in an activation
record's log field. This may result when the activation record is retrieved
before all the logs are stored in the separate log store.
On a similar note, I don't think we should be storing activation responses
at all.
Regards,
James Dubee
From: Rodric Rabbah <ro...@gmail.com>
To: dev@openwhisk.apache.org
Date: 10/02/2018 07:30 PM
Subject: Re: Logstore usage during `activation get`
> By "break this" do you mean at some point we should remove the logs from
the GET?
Yes.
@dubee thoughts? Since you've worked on the elastic plugin.
-r
Re: Logstore usage during `activation get`
Posted by Rodric Rabbah <ro...@gmail.com>.
> By "break this" do you mean at some point we should remove the logs from
the GET?
Yes.
@dubee thoughts? Since you've worked on the elastic plugin.
-r
Re: Logstore usage during `activation get`
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
By "break this" do you mean at some point we should remove the logs from the GET?
In any case I will close the PR.
Thanks
Tyson
On 10/2/18, 4:21 PM, "Rodric Rabbah" <ro...@gmail.com> wrote:
Hi Tyson - this was the intent of the API design: there is a separate resource for LOGS and the RESULT. The reasoning also that the metadata is typically small but the logs could be much larger. Separating the two was also intended for easier streaming of the responses.
Because of implementation made it easier to bundle the response, we have the current “feature” where GET on the activation id returns the entire record. I think we can break this because the clients can sugar the underlying calls.
-r
> On Oct 2, 2018, at 12:07 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>
> Hi –
> I created this PR [1] due to noticing that `wsk activation get` does NOT return logs from a LogStore which stores logs outside of the Activation entity.
> But it bring up a question of: Does IBM or any other operator who might use a custom LogStore desire to have logs included with `activation get`?
> Currently returning logs is only possible using `wsk activation logs`
>
> Personally, I think it is “nice” to have a separate explicit request for logs and activation metadata, and this is the way that the current OW Activation API operates with regards to an external LogStore (splunk, elk, etc), but after all it is inconsistent from the case where logs are NOT using external storage.
>
> WDYT?
>
> Thanks
> Tyson
>
> [1] https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F4044&data=02%7C01%7Ctnorris%40adobe.com%7C30a7422c5e8f4184015408d628bdcb36%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636741192954948579&sdata=uvYACmWD3PWVl3pWzCZpzZer1jVFhdpv8pc%2Fz0Bh7z4%3D&reserved=0
Re: Logstore usage during `activation get`
Posted by Rodric Rabbah <ro...@gmail.com>.
Hi Tyson - this was the intent of the API design: there is a separate resource for LOGS and the RESULT. The reasoning also that the metadata is typically small but the logs could be much larger. Separating the two was also intended for easier streaming of the responses.
Because of implementation made it easier to bundle the response, we have the current “feature” where GET on the activation id returns the entire record. I think we can break this because the clients can sugar the underlying calls.
-r
> On Oct 2, 2018, at 12:07 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>
> Hi –
> I created this PR [1] due to noticing that `wsk activation get` does NOT return logs from a LogStore which stores logs outside of the Activation entity.
> But it bring up a question of: Does IBM or any other operator who might use a custom LogStore desire to have logs included with `activation get`?
> Currently returning logs is only possible using `wsk activation logs`
>
> Personally, I think it is “nice” to have a separate explicit request for logs and activation metadata, and this is the way that the current OW Activation API operates with regards to an external LogStore (splunk, elk, etc), but after all it is inconsistent from the case where logs are NOT using external storage.
>
> WDYT?
>
> Thanks
> Tyson
>
> [1] https://github.com/apache/incubator-openwhisk/pull/4044