Idea: Is it useful to have an dedicated AWS S3 Appender?

[Christian Schneider <cs...@gmail.com>]

Hi,
for our ec2 instances (with limited diskspace) we like to have a simple
Appender that ships its logs to s3.
It seems there is no "standard" way of doing this.
There are some ideas, but they doesn't seem completed at the first glance
[0] ... [3]

But before starting a new n-th S3Appender project, I like to ask whether
you think it would be a great idea?

At the most simplest stage we could think of an extension of the
RollingFileAppender.
That Appender is then usable on any machine and in every infrastructure
that has http access to s3

If this is done, maybe it would be useful to have a more ec2 coupled
appender.
That appender could know more about its environment.
With this, it would be possible to let log4j organize the "upload"-path.
E.g.: <bucket>/<instanceId || ipaddress>.
Or Information about the VPC settings, userscripts, ...

What do you think?

Thanks for you feedback,
Christian.


[0] https://github.com/ryanhosp/mldx-log-appenders
[1] http://aws.amazon.com/code/Amazon-S3/4032
[2] http://s3appender.codeplex.com/
[3] https://github.com/dialt0ne/rotate-to-s3

Re: Idea: Is it useful to have an dedicated AWS S3 Appender?

[Remko Popma <re...@gmail.com>]

You may want to create a Jira ticket that you can attach patch files to
(the mailing list software doesn't like attachments).

Regards,
Remko

On Thursday, October 31, 2013, Remko Popma wrote:

> Hi Christian,
>
> I'd say go for it! If it's useful to you then it's probably useful for
> other people too.
>
> The projects you mention are either for other languages or no longer
> actively maintained (afaics), so I don't think you'd be duplicating effort.
>
> Regarding testing, I'm not sure who in the community has access to a ec2
> system to test such an appender. So we'd rely on junit tests and community
> feedback to fix any issues (but that's not a first).
>
> The way to proceed for this is to supply a patch that one of the
> committers can integrate with trunk. You'd need to check out trunk and base
> your patch off of that.
>
> To protect the ASF from legal issues, we ask that you submit a CLA (
> http://www.apache.org/licenses/#clas) before we can commit your
> contribution.
>
> Looking forward to your contribution!
>
> Best regards,
> Remko
>
> On Wednesday, October 30, 2013, Christian Schneider wrote:
>
>> Hi,
>> for our ec2 instances (with limited diskspace) we like to have a simple
>> Appender that ships its logs to s3.
>> It seems there is no "standard" way of doing this.
>> There are some ideas, but they doesn't seem completed at the first glance
>> [0] ... [3]
>>
>> But before starting a new n-th S3Appender project, I like to ask whether
>> you think it would be a great idea?
>>
>> At the most simplest stage we could think of an extension of the
>> RollingFileAppender.
>> That Appender is then usable on any machine and in every infrastructure
>> that has http access to s3
>>
>> If this is done, maybe it would be useful to have a more ec2 coupled
>> appender.
>> That appender could know more about its environment.
>> With this, it would be possible to let log4j organize the "upload"-path.
>> E.g.: <bucket>/<instanceId || ipaddress>.
>> Or Information about the VPC settings, userscripts, ...
>>
>> What do you think?
>>
>> Thanks for you feedback,
>> Christian.
>>
>>
>> [0] https://github.com/ryanhosp/mldx-log-appenders
>> [1] http://aws.amazon.com/code/Amazon-S3/4032
>> [2] http://s3appender.codeplex.com/
>> [3] https://github.com/dialt0ne/rotate-to-s3
>>
>

Re: Idea: Is it useful to have an dedicated AWS S3 Appender?

[Remko Popma <re...@gmail.com>]

Hi Christian,

I'd say go for it! If it's useful to you then it's probably useful for
other people too.

The projects you mention are either for other languages or no longer
actively maintained (afaics), so I don't think you'd be duplicating effort.

Regarding testing, I'm not sure who in the community has access to a ec2
system to test such an appender. So we'd rely on junit tests and community
feedback to fix any issues (but that's not a first).

The way to proceed for this is to supply a patch that one of the committers
can integrate with trunk. You'd need to check out trunk and base your patch
off of that.

To protect the ASF from legal issues, we ask that you submit a CLA (
http://www.apache.org/licenses/#clas) before we can commit your
contribution.

Looking forward to your contribution!

Best regards,
Remko

On Wednesday, October 30, 2013, Christian Schneider wrote:

> Hi,
> for our ec2 instances (with limited diskspace) we like to have a simple
> Appender that ships its logs to s3.
> It seems there is no "standard" way of doing this.
> There are some ideas, but they doesn't seem completed at the first glance
> [0] ... [3]
>
> But before starting a new n-th S3Appender project, I like to ask whether
> you think it would be a great idea?
>
> At the most simplest stage we could think of an extension of the
> RollingFileAppender.
> That Appender is then usable on any machine and in every infrastructure
> that has http access to s3
>
> If this is done, maybe it would be useful to have a more ec2 coupled
> appender.
> That appender could know more about its environment.
> With this, it would be possible to let log4j organize the "upload"-path.
> E.g.: <bucket>/<instanceId || ipaddress>.
> Or Information about the VPC settings, userscripts, ...
>
> What do you think?
>
> Thanks for you feedback,
> Christian.
>
>
> [0] https://github.com/ryanhosp/mldx-log-appenders
> [1] http://aws.amazon.com/code/Amazon-S3/4032
> [2] http://s3appender.codeplex.com/
> [3] https://github.com/dialt0ne/rotate-to-s3
>