You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by andcrow <an...@gmail.com> on 2017/04/12 14:06:37 UTC
Using Split with JSON File
Hello,
I have a route that triggers a SELECT in a database and creates a JSON file
in a folder (this file which on average generates something around 100k to
150k of records).
Then another route that sends the data of this file to an endpoint using the
HTTP4 component and performing a POST and REST API (the endpoint is already
prepared to perform batch processing).
I noticed something curious, that in certain cases Camel sends the data, Web
Services is processing correctly, but after a few seconds Camel returns me a
DeadLetterChannel, with some exceptions:
- "org.apache.http.wire" "http-outgoing-5" "end of stream"
- "org.apache.http.nohttpresponseexception" "failed to respond"
- "h.i.c.DefaultManagedHttpClientConnection" "http-outgoing-5" "Close
connection"
- "h.i.c.DefaultManagedHttpClientConnection" "http-outgoing-5" "Shutdown
connection"
- "o.a.http.impl.execchain.MainClientExec" "Connection discarded"
"Connection released"
But the process continues normally, it tries to resend again, but gives the
error again, but in AWS web services it is processing normally and I have
verified that it returns the normal status codes.
Searching the Camel documentation and found httpClient. I made the
configuration by increasing the timeout, but nothing worked, it seems that
it ignores these settings (httpClient.socketTimeout = 300000).
I'm thinking of splitting the JSON file, with some rule like every 1000
records generating 1 JSON file. So I will have several smaller files that
will be processed faster and thus get the response from the server, avoiding
this error.
My actual route thats find all records in database and create a json file
is:
from("timer://query?fixedRate=true&delay=5s&period=10800s")
.routeId("route-product")
.setBody(constant(new ProductQuery().getProductSelect()))
.to("jdbc:myDataSource")
.marshal()
.json(JsonLibrary.Gson)
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.setHeader(Exchange.HTTP_CHARACTER_ENCODING, constant("UTF-8"))
.setHeader("CamelFileName", constant("products.json"))
.to("file:products");
However, searching the documentation, I was able to perform the split:
from("timer://query?fixedRate=true&delay=5s&period=10800s")
.routeId("route-product")
.setBody(constant(new ProductQuery().getProductSelect()))
.to("jdbc:myDataSource?outputType=StreamList")
.split(body()).streaming()
.marshal()
.json(JsonLibrary.Gson)
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.setHeader("CamelFileName", simple("${id}.json"))
.to("file:product");
In this way each record creates a JSON file, but my case, I want to group
these records, again in the documentation is suggested as follows:
.split().tokenize("\n", 1000).streaming()
But in this way the output of the log shows only 1 file with the following
content:
"org.apache.camel.component.jdbc.ResultSetIterator@7be7d81f"
If I remove the option ?outputType=StreamList from the JDBC component's URI,
then the following exception is thrown:
No converter found capable of converting from type [java.util.ArrayList<?>]
to type [java.lang.String]
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Using Split with JSON File
Posted by andcrow <an...@gmail.com>.
Does Apache Camel have any component that can perform auditing on a database?
Or audit records that will be sent to a web service using a JSON file?
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5798258.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Using Split with JSON File
Posted by andcrow <an...@gmail.com>.
I'll try
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797774.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Using Split with JSON File
Posted by souciance <so...@gmail.com>.
I would suggest get the whole resultset and save the data on a queue or
even as a file. Then you can parse it as a stream or every x number of rows
and split.
On Mon, Apr 17, 2017 at 7:49 PM, andcrow [via Camel] <
ml-node+s465427n5797772h95@n5.nabble.com> wrote:
> Did not work using maxMessagesPerPoll.
>
> Does not group every 1000 records.
>
> Creates 1 JSON file for each record.
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Using-Split-with-JSON-
> File-tp5797508p5797772.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797773.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Using Split with JSON File
Posted by andcrow <an...@gmail.com>.
Did not work using maxMessagesPerPoll.
Does not group every 1000 records.
Creates 1 JSON file for each record.
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797772.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Using Split with JSON File
Posted by souciance <so...@gmail.com>.
Couldn't you split the resultset from the database and for each split
produce a json and send that batch to your web service?
On Mon, Apr 17, 2017 at 6:45 PM, andcrow [via Camel] <
ml-node+s465427n5797770h33@n5.nabble.com> wrote:
> Still in the battle with this case ..... hehehehe
>
> Viewing the documentation I found the component SQL Component:
>
> In the options:
>
> maxMessagesPerPoll: Camel 2.11: SQL consumer only: An integer value to
> define the maximum number of messages to gather per poll. By default, no
> maximum is set.
>
> But also with this component he ignores this option:
>
> from("timer://query?fixedRate=true&delay=5s&period=10800s")
> .routeId("route-product")
> .setBody(constant(new ProductQuery().getProductSelect()))
> to("sql:" + new ProductQuery().getProductSelect() +
> "?maxMessagesPerPoll=1000&dataSource=myDataSource")
> .marshal()
> .json(JsonLibrary.Gson)
> .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
> .setHeader("CamelFileName", simple("${id}.json"))
> .to("file:product");
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Using-Split-with-JSON-
> File-tp5797508p5797770.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797771.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Using Split with JSON File
Posted by andcrow <an...@gmail.com>.
Still in the battle with this case ..... hehehehe
Viewing the documentation I found the component SQL Component:
In the options:
maxMessagesPerPoll: Camel 2.11: SQL consumer only: An integer value to
define the maximum number of messages to gather per poll. By default, no
maximum is set.
But also with this component he ignores this option:
from("timer://query?fixedRate=true&delay=5s&period=10800s")
.routeId("route-product")
.setBody(constant(new ProductQuery().getProductSelect()))
to("sql:" + new ProductQuery().getProductSelect() +
"?maxMessagesPerPoll=1000&dataSource=myDataSource")
.marshal()
.json(JsonLibrary.Gson)
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.setHeader("CamelFileName", simple("${id}.json"))
.to("file:product");
--
View this message in context: http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508p5797770.html
Sent from the Camel - Users mailing list archive at Nabble.com.