log transfering into hadoop/spark

[pengyh <pe...@web.de>]

since flume is not continued to develop.
what's the current opensource tool to transfer webserver logs into
hdfs/spark?

thank you.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: log transfering into hadoop/spark

[Gourav Sengupta <go...@gmail.com>]

hi,
I do it with simple bash scripts to transfer to s3. Takes less than 1
minute to write it, and another 1 min to include it bootstrap scripts.

Never saw the need for so much hype for such simple tasks.

Regards,
Gourav Sengupta

On Tue, Aug 2, 2022 at 2:16 PM ayan guha <gu...@gmail.com> wrote:

> ELK or Splunk agents typically.
>
> Or if you are in cloud then there are cloud native solutions which can
> forward logs to object store, which can then be read like hdfs.
>
> On Tue, 2 Aug 2022 at 4:43 pm, pengyh <pe...@web.de> wrote:
>
>> since flume is not continued to develop.
>> what's the current opensource tool to transfer webserver logs into
>> hdfs/spark?
>>
>> thank you.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>> --
> Best Regards,
> Ayan Guha
>

Re: log transfering into hadoop/spark

[ayan guha <gu...@gmail.com>]

ELK or Splunk agents typically.

Or if you are in cloud then there are cloud native solutions which can
forward logs to object store, which can then be read like hdfs.

On Tue, 2 Aug 2022 at 4:43 pm, pengyh <pe...@web.de> wrote:

> since flume is not continued to develop.
> what's the current opensource tool to transfer webserver logs into
> hdfs/spark?
>
> thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
> --
Best Regards,
Ayan Guha