You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by prasad <pr...@gmail.com> on 2014/12/08 12:20:53 UTC
Can we load log files into some database using Camel?
Hi,
I just started using camel for ETL Operations.I have loaded the data
from database to a file and to another database too.But now i want to
transform and load the continuous web server log files into database ,Can
we do this with camel?
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by "Preben.Asmussen" <pr...@dr.dk>.
Hi,
A word of advice here since we have done something similar where we load GB
of zipped http log files from our http servers, unzip them, parse them and
insert the parsed content into a structured db tables using camel.
Even though the solution works it is somewhat cumbersome to maintain when
log file format changes, error handling when dealing with GB of log files
asf.
Depending on your requirements and http traffic I really think you should
consider a more flexible solution that can handle unstructured/big data
(nosql).
Properly we will replace our current Camel solution with something like
Splunk, that can realtime ingest raw http log files, and data can be
analysed on the original content.
/Preben
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760325.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by akoufoudakis <ak...@gmail.com>.
Nice...
So can it be done like:
stream:file?fileName=access.log
?
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760306.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by prasad <pr...@gmail.com>.
Thank you so much for your response.
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760319.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
There is the stream component which can read log files while they are in use
http://camel.apache.org/stream
On Mon, Dec 8, 2014 at 1:24 PM, akoufoudakis <ak...@gmail.com> wrote:
> A small update.
> You can read an uncompleted file. But, then you might want to use the
> readLock option to be sure that when you read the file, nothing is writing
> to it at the same time.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760302.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Can we load log files into some database using Camel?
Posted by akoufoudakis <ak...@gmail.com>.
A small update.
You can read an uncompleted file. But, then you might want to use the
readLock option to be sure that when you read the file, nothing is writing
to it at the same time.
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760302.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by akoufoudakis <ak...@gmail.com>.
Well, to read a file you definitely need to use camel file2 component.
However, I don't know whether reading a non-completed files (because, it is
a log file and, as you said, is appended by new log entries) is a proper way
to use this component.
Say, it can be done and it is not a misuse. Do you want to have a new row
for each new version of the file or do you want to update the existing row,
if a row with the file already exists in the database.
Finally, do you want to keep log files in their initial directory or to move
them somewhere?
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760299.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AW: Can we load log files into some database using Camel?
Posted by akoufoudakis <ak...@gmail.com>.
So, finally, you could use the combination of stream component and jpa
component, as Jan noted:
from("stream:file?fileName...")
.to("jpa:my.etl.LogContent");
Alternatively, you can also use SQL or JDBC component to store it in the
database.
You can also use bean component between stream and JPA/SQL/JDBC, if you want
to do something tricky with the content.
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760308.html
Sent from the Camel - Users mailing list archive at Nabble.com.
AW: Can we load log files into some database using Camel?
Posted by "Jan Matèrne (jhm)" <ap...@materne.de>.
http://camel.apache.org/file2.html
This is a polling consumer (in your case) so it will fetch all new added
files.
Storing its content could be something simple like
from("file:logDirectory")
.to("jpa:my.etl.LogContent")
where LogContent in its simplest implementation just stores the string
content plus an additional generated id.
With a custom TypeConverter you could also do some parsing ...
Jan
> -----Ursprüngliche Nachricht-----
> Von: prasad [mailto:prasadbezavada@gmail.com]
> Gesendet: Montag, 8. Dezember 2014 13:03
> An: users@camel.apache.org
> Betreff: Re: Can we load log files into some database using Camel?
>
> Hi,
>
> Thank you so much for your Prompt response .
> yeah that is my requirement ,but continuously the newly generating log
> files
> also have to come into my database.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Can-we-
> load-log-files-into-some-database-using-Camel-tp5760290p5760296.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by prasad <pr...@gmail.com>.
Hi,
Thank you so much for your Prompt response .
yeah that is my requirement ,but continuously the newly generating log files
also have to come into my database.
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760296.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can we load log files into some database using Camel?
Posted by akoufoudakis <ak...@gmail.com>.
Hello, Prasad!
So, you want to take a file from a directory and to put its contents to the
database.
Did I get you correctly?
--
View this message in context: http://camel.465427.n5.nabble.com/Can-we-load-log-files-into-some-database-using-Camel-tp5760290p5760295.html
Sent from the Camel - Users mailing list archive at Nabble.com.