You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Leif Hedstrom <zw...@apache.org> on 2015/04/21 16:06:12 UTC
[FEATURE REMOVAL v6.0.0] Log collation
We have a feature to do remote logging, called “log collation”. This runs either as part of traffic_server itself, or using the stand-alone collector, traffic_sac. The issues with log collation are
* Not well supported, no development / fixes are done on critical bugs
* It has some severe shortcomings, such as orphaned logs never being collected
* We feel that there are better out-of-band transports for logging, and we shouldn’t be in this business
Some options for collecting logs include
* syslog-ng
* Apache Flume
* Splunk (commercial)
* etc.
The Jira for this is
https://issues.apache.org/jira/browse/TS-3168
Unless we hear otherwise within 1 week, this feature will be removed for 6.0.0. If there’s concerns about upgrades from 5.x to 6.0, please voice your concerns asap. We could consider making this a deprecation in 6.0, and removal for 7.0, but only if people really need this feature during an upgrade period. v6.0.0 is schedule for release in ~july time frame.
— Leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 28, 2015, at 5:18 AM, Yongming Zhao <mi...@gmail.com> wrote:
>
>>
>> 1) You use this feature?
>
> Yes, we do use this feature.
>
>>
>> 2) You will support this feature, and work on the code?
>
> sure we will support it, because we use it.
Alright, cool. Lets keep log collation, I’ll reopen the closed Jira’s, and if necessary, we will revisit this decision for 7.0.0 (~1 year from now).
Cheers,
- leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 28, 2015, at 5:18 AM, Yongming Zhao <mi...@gmail.com> wrote:
>
>>
>> 1) You use this feature?
>
> Yes, we do use this feature.
>
>>
>> 2) You will support this feature, and work on the code?
>
> sure we will support it, because we use it.
Alright, cool. Lets keep log collation, I’ll reopen the closed Jira’s, and if necessary, we will revisit this decision for 7.0.0 (~1 year from now).
Cheers,
- leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Yongming Zhao <mi...@gmail.com>.
>
> 1) You use this feature?
Yes, we do use this feature.
>
> 2) You will support this feature, and work on the code?
sure we will support it, because we use it.
just go thorough the jira you listed below, looks not that bad, I think we can fix it, let us fix them first to in the ‘probation’.
> 在 2015年4月28日，上午1:38，Leif Hedstrom <zw...@apache.org> 写道：
>
>
>> On Apr 27, 2015, at 12:15 AM, Yongming Zhao <mi...@gmail.com> wrote:
>>
>> on the collation:
>> I don’t get that many critical bugs on collation: project = TS AND component = Logging AND resolution = Unresolved AND text ~ "collation" ORDER BY due ASC, priority DESC, created ASC
>> so, what is the problem we would like to remove it completely?
>
>
> We closed a few of them as Won’t Fix. But the issues that were crippling were
>
> - There is no way to replay an orphaned log
>
> and together with
>
> https://issues.apache.org/jira/browse/TS-2629
> https://issues.apache.org/jira/browse/TS-2062
> https://issues.apache.org/jira/browse/TS-2101
> https://issues.apache.org/jira/browse/TS-2781
> https://issues.apache.org/jira/browse/TS-910
>
>
> this becomes completely crippling (it filled up our disks).
>
>
>
>>
>> the current ATS logging a ugly but very good in performance thinking log system, the SAC is a dedicated commercial server to offer log collation service, due to that we are now open sourced, that is a good idea removing the duplicated SAC codes. In our using with syslog-ng, it’s performance may affect very busy ATS systems, that is why Yunkai pushed many collation improvements.
>
> No one has said anything about the logging infrastructure itself. In fact, I do not think it’s ugly, I think it’s very good. This has nothing to do with that, logging will work as it does.
>
>>
>> yes, logging has some ugly bugs, but collation should not that bad after Yunkai’s work, and it does not bother you if you just logging to local disk and collection with the other systems, right?
>
>
> Yes, it did bother us, because the misbehavior around orphaned logs are crippling.
>
>>
>> please consider keep this feature, I am very happy that a simple collation works for me out of box.
>>
>> sure you may find someone will take interest in supporting it, only if it is still there.
>
>
> So, I take it that
>
> 1) You use this feature?
>
> 2) You will support this feature, and work on the code?
>
>
>
> I could probably be convinced to give this feature another year “probation”, but only under the conditions that you answered “yes” to both questions above. If so, I’ll file a Jira on the biggest issue, that orphaned logs are abandoned with no way to get inserted again.
>
>
> Anyone else have any thoughts on this?
>
>
> — Leif
>
- Yongming Zhao 赵永明
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Yongming Zhao <mi...@gmail.com>.
>
> 1) You use this feature?
Yes, we do use this feature.
>
> 2) You will support this feature, and work on the code?
sure we will support it, because we use it.
just go thorough the jira you listed below, looks not that bad, I think we can fix it, let us fix them first to in the ‘probation’.
> 在 2015年4月28日，上午1:38，Leif Hedstrom <zw...@apache.org> 写道：
>
>
>> On Apr 27, 2015, at 12:15 AM, Yongming Zhao <mi...@gmail.com> wrote:
>>
>> on the collation:
>> I don’t get that many critical bugs on collation: project = TS AND component = Logging AND resolution = Unresolved AND text ~ "collation" ORDER BY due ASC, priority DESC, created ASC
>> so, what is the problem we would like to remove it completely?
>
>
> We closed a few of them as Won’t Fix. But the issues that were crippling were
>
> - There is no way to replay an orphaned log
>
> and together with
>
> https://issues.apache.org/jira/browse/TS-2629
> https://issues.apache.org/jira/browse/TS-2062
> https://issues.apache.org/jira/browse/TS-2101
> https://issues.apache.org/jira/browse/TS-2781
> https://issues.apache.org/jira/browse/TS-910
>
>
> this becomes completely crippling (it filled up our disks).
>
>
>
>>
>> the current ATS logging a ugly but very good in performance thinking log system, the SAC is a dedicated commercial server to offer log collation service, due to that we are now open sourced, that is a good idea removing the duplicated SAC codes. In our using with syslog-ng, it’s performance may affect very busy ATS systems, that is why Yunkai pushed many collation improvements.
>
> No one has said anything about the logging infrastructure itself. In fact, I do not think it’s ugly, I think it’s very good. This has nothing to do with that, logging will work as it does.
>
>>
>> yes, logging has some ugly bugs, but collation should not that bad after Yunkai’s work, and it does not bother you if you just logging to local disk and collection with the other systems, right?
>
>
> Yes, it did bother us, because the misbehavior around orphaned logs are crippling.
>
>>
>> please consider keep this feature, I am very happy that a simple collation works for me out of box.
>>
>> sure you may find someone will take interest in supporting it, only if it is still there.
>
>
> So, I take it that
>
> 1) You use this feature?
>
> 2) You will support this feature, and work on the code?
>
>
>
> I could probably be convinced to give this feature another year “probation”, but only under the conditions that you answered “yes” to both questions above. If so, I’ll file a Jira on the biggest issue, that orphaned logs are abandoned with no way to get inserted again.
>
>
> Anyone else have any thoughts on this?
>
>
> — Leif
>
- Yongming Zhao 赵永明
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Theo Schlossnagle <je...@omniti.com>.
Yes. -1 from me.
On Apr 28, 2015 12:54 AM, "Leif Hedstrom" <zw...@apache.org> wrote:
>
> > On Apr 27, 2015, at 10:40 AM, Theo Schlossnagle <je...@omniti.com>
> wrote:
> >
> > I'm pretty sure we use log collation here. Also, we tend to not have any
> > issues with it.
>
>
> Roger. Surprised you are at least not running into the orphaned log issues
> :).
>
> Should I consider this a -1 vote then to not eliminate this feature for
> now?
>
> — Leif
>
>
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 27, 2015, at 10:40 AM, Theo Schlossnagle <je...@omniti.com> wrote:
>
> I'm pretty sure we use log collation here. Also, we tend to not have any
> issues with it.
Roger. Surprised you are at least not running into the orphaned log issues :).
Should I consider this a -1 vote then to not eliminate this feature for now?
— Leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 27, 2015, at 10:40 AM, Theo Schlossnagle <je...@omniti.com> wrote:
>
> I'm pretty sure we use log collation here. Also, we tend to not have any
> issues with it.
Roger. Surprised you are at least not running into the orphaned log issues :).
Should I consider this a -1 vote then to not eliminate this feature for now?
— Leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Theo Schlossnagle <je...@omniti.com>.
I'm pretty sure we use log collation here. Also, we tend to not have any
issues with it.
On Mon, Apr 27, 2015 at 1:38 PM, Leif Hedstrom <zw...@apache.org> wrote:
>
> > On Apr 27, 2015, at 12:15 AM, Yongming Zhao <mi...@gmail.com> wrote:
> >
> > on the collation:
> > I don’t get that many critical bugs on collation: project = TS AND
> component = Logging AND resolution = Unresolved AND text ~ "collation"
> ORDER BY due ASC, priority DESC, created ASC
> > so, what is the problem we would like to remove it completely?
>
>
> We closed a few of them as Won’t Fix. But the issues that were crippling
> were
>
> - There is no way to replay an orphaned log
>
> and together with
>
> https://issues.apache.org/jira/browse/TS-2629
> https://issues.apache.org/jira/browse/TS-2062
> https://issues.apache.org/jira/browse/TS-2101
> https://issues.apache.org/jira/browse/TS-2781
> https://issues.apache.org/jira/browse/TS-910
>
>
> this becomes completely crippling (it filled up our disks).
>
>
>
> >
> > the current ATS logging a ugly but very good in performance thinking
> log system, the SAC is a dedicated commercial server to offer log collation
> service, due to that we are now open sourced, that is a good idea removing
> the duplicated SAC codes. In our using with syslog-ng, it’s performance
> may affect very busy ATS systems, that is why Yunkai pushed many collation
> improvements.
>
> No one has said anything about the logging infrastructure itself. In fact,
> I do not think it’s ugly, I think it’s very good. This has nothing to do
> with that, logging will work as it does.
>
> >
> > yes, logging has some ugly bugs, but collation should not that bad
> after Yunkai’s work, and it does not bother you if you just logging to
> local disk and collection with the other systems, right?
>
>
> Yes, it did bother us, because the misbehavior around orphaned logs are
> crippling.
>
> >
> > please consider keep this feature, I am very happy that a simple
> collation works for me out of box.
> >
> > sure you may find someone will take interest in supporting it, only if
> it is still there.
>
>
> So, I take it that
>
> 1) You use this feature?
>
> 2) You will support this feature, and work on the code?
>
>
>
> I could probably be convinced to give this feature another year
> “probation”, but only under the conditions that you answered “yes” to both
> questions above. If so, I’ll file a Jira on the biggest issue, that
> orphaned logs are abandoned with no way to get inserted again.
>
>
> Anyone else have any thoughts on this?
>
>
> — Leif
>
>
--
Theo Schlossnagle
http://omniti.com/is/theo-schlossnagle
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 27, 2015, at 12:15 AM, Yongming Zhao <mi...@gmail.com> wrote:
>
> on the collation:
> I don’t get that many critical bugs on collation: project = TS AND component = Logging AND resolution = Unresolved AND text ~ "collation" ORDER BY due ASC, priority DESC, created ASC
> so, what is the problem we would like to remove it completely?
We closed a few of them as Won’t Fix. But the issues that were crippling were
- There is no way to replay an orphaned log
and together with
https://issues.apache.org/jira/browse/TS-2629
https://issues.apache.org/jira/browse/TS-2062
https://issues.apache.org/jira/browse/TS-2101
https://issues.apache.org/jira/browse/TS-2781
https://issues.apache.org/jira/browse/TS-910
this becomes completely crippling (it filled up our disks).
>
> the current ATS logging a ugly but very good in performance thinking log system, the SAC is a dedicated commercial server to offer log collation service, due to that we are now open sourced, that is a good idea removing the duplicated SAC codes. In our using with syslog-ng, it’s performance may affect very busy ATS systems, that is why Yunkai pushed many collation improvements.
No one has said anything about the logging infrastructure itself. In fact, I do not think it’s ugly, I think it’s very good. This has nothing to do with that, logging will work as it does.
>
> yes, logging has some ugly bugs, but collation should not that bad after Yunkai’s work, and it does not bother you if you just logging to local disk and collection with the other systems, right?
Yes, it did bother us, because the misbehavior around orphaned logs are crippling.
>
> please consider keep this feature, I am very happy that a simple collation works for me out of box.
>
> sure you may find someone will take interest in supporting it, only if it is still there.
So, I take it that
1) You use this feature?
2) You will support this feature, and work on the code?
I could probably be convinced to give this feature another year “probation”, but only under the conditions that you answered “yes” to both questions above. If so, I’ll file a Jira on the biggest issue, that orphaned logs are abandoned with no way to get inserted again.
Anyone else have any thoughts on this?
— Leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 27, 2015, at 12:15 AM, Yongming Zhao <mi...@gmail.com> wrote:
>
> on the collation:
> I don’t get that many critical bugs on collation: project = TS AND component = Logging AND resolution = Unresolved AND text ~ "collation" ORDER BY due ASC, priority DESC, created ASC
> so, what is the problem we would like to remove it completely?
We closed a few of them as Won’t Fix. But the issues that were crippling were
- There is no way to replay an orphaned log
and together with
https://issues.apache.org/jira/browse/TS-2629
https://issues.apache.org/jira/browse/TS-2062
https://issues.apache.org/jira/browse/TS-2101
https://issues.apache.org/jira/browse/TS-2781
https://issues.apache.org/jira/browse/TS-910
this becomes completely crippling (it filled up our disks).
>
> the current ATS logging a ugly but very good in performance thinking log system, the SAC is a dedicated commercial server to offer log collation service, due to that we are now open sourced, that is a good idea removing the duplicated SAC codes. In our using with syslog-ng, it’s performance may affect very busy ATS systems, that is why Yunkai pushed many collation improvements.
No one has said anything about the logging infrastructure itself. In fact, I do not think it’s ugly, I think it’s very good. This has nothing to do with that, logging will work as it does.
>
> yes, logging has some ugly bugs, but collation should not that bad after Yunkai’s work, and it does not bother you if you just logging to local disk and collection with the other systems, right?
Yes, it did bother us, because the misbehavior around orphaned logs are crippling.
>
> please consider keep this feature, I am very happy that a simple collation works for me out of box.
>
> sure you may find someone will take interest in supporting it, only if it is still there.
So, I take it that
1) You use this feature?
2) You will support this feature, and work on the code?
I could probably be convinced to give this feature another year “probation”, but only under the conditions that you answered “yes” to both questions above. If so, I’ll file a Jira on the biggest issue, that orphaned logs are abandoned with no way to get inserted again.
Anyone else have any thoughts on this?
— Leif
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Yongming Zhao <mi...@gmail.com>.
on the collation:
I don’t get that many critical bugs on collation: project = TS AND component = Logging AND resolution = Unresolved AND text ~ "collation" ORDER BY due ASC, priority DESC, created ASC
so, what is the problem we would like to remove it completely?
the current ATS logging a ugly but very good in performance thinking log system, the SAC is a dedicated commercial server to offer log collation service, due to that we are now open sourced, that is a good idea removing the duplicated SAC codes. In our using with syslog-ng, it’s performance may affect very busy ATS systems, that is why Yunkai pushed many collation improvements.
yes, logging has some ugly bugs, but collation should not that bad after Yunkai’s work, and it does not bother you if you just logging to local disk and collection with the other systems, right?
please consider keep this feature, I am very happy that a simple collation works for me out of box.
sure you may find someone will take interest in supporting it, only if it is still there.
thanks
> 在 2015年4月21日，下午10:06，Leif Hedstrom <zw...@apache.org> 写道：
>
> We have a feature to do remote logging, called “log collation”. This runs either as part of traffic_server itself, or using the stand-alone collector, traffic_sac. The issues with log collation are
>
> * Not well supported, no development / fixes are done on critical bugs
> * It has some severe shortcomings, such as orphaned logs never being collected
> * We feel that there are better out-of-band transports for logging, and we shouldn’t be in this business
>
> Some options for collecting logs include
>
> * syslog-ng
> * Apache Flume
> * Splunk (commercial)
> * etc.
>
>
> The Jira for this is
>
> https://issues.apache.org/jira/browse/TS-3168
>
> Unless we hear otherwise within 1 week, this feature will be removed for 6.0.0. If there’s concerns about upgrades from 5.x to 6.0, please voice your concerns asap. We could consider making this a deprecation in 6.0, and removal for 7.0, but only if people really need this feature during an upgrade period. v6.0.0 is schedule for release in ~july time frame.
>
> — Leif
>
- Yongming Zhao 赵永明
Re: [FEATURE REMOVAL v6.0.0] Log collation
Posted by Yongming Zhao <mi...@gmail.com>.
on the collation:
I don’t get that many critical bugs on collation: project = TS AND component = Logging AND resolution = Unresolved AND text ~ "collation" ORDER BY due ASC, priority DESC, created ASC
so, what is the problem we would like to remove it completely?
the current ATS logging a ugly but very good in performance thinking log system, the SAC is a dedicated commercial server to offer log collation service, due to that we are now open sourced, that is a good idea removing the duplicated SAC codes. In our using with syslog-ng, it’s performance may affect very busy ATS systems, that is why Yunkai pushed many collation improvements.
yes, logging has some ugly bugs, but collation should not that bad after Yunkai’s work, and it does not bother you if you just logging to local disk and collection with the other systems, right?
please consider keep this feature, I am very happy that a simple collation works for me out of box.
sure you may find someone will take interest in supporting it, only if it is still there.
thanks
> 在 2015年4月21日，下午10:06，Leif Hedstrom <zw...@apache.org> 写道：
>
> We have a feature to do remote logging, called “log collation”. This runs either as part of traffic_server itself, or using the stand-alone collector, traffic_sac. The issues with log collation are
>
> * Not well supported, no development / fixes are done on critical bugs
> * It has some severe shortcomings, such as orphaned logs never being collected
> * We feel that there are better out-of-band transports for logging, and we shouldn’t be in this business
>
> Some options for collecting logs include
>
> * syslog-ng
> * Apache Flume
> * Splunk (commercial)
> * etc.
>
>
> The Jira for this is
>
> https://issues.apache.org/jira/browse/TS-3168
>
> Unless we hear otherwise within 1 week, this feature will be removed for 6.0.0. If there’s concerns about upgrades from 5.x to 6.0, please voice your concerns asap. We could consider making this a deprecation in 6.0, and removal for 7.0, but only if people really need this feature during an upgrade period. v6.0.0 is schedule for release in ~july time frame.
>
> — Leif
>
- Yongming Zhao 赵永明