Camel to ingest data more than 200 HTTP url endpoints

[Vins Yu <vi...@gmail.com>]

Hi,

I'm building an app that grabs DATA from dynamic URLs which can reach
more than 200.

I tried the following

// this bean just generates dynamic url

 public void createUrls(Exchange exchange) {
        List<String> list = new ArrayList<String>();
        for (int i = 0; i < 9; i++) {
            list.add("resources" + i + data[i]);
        }

        exchange.getIn().setHeader("theList", list);

    }


then I have a camel route

camel routes:


        <route id="testing_loop">
            <from uri="timer://execute?delay=5000&amp;repeatCount=1"/>
            <to uri="bean:dataProcessor?method=createUrls"/>
            <split>
                <header>theList</header>
                <setHeader headerName="currentURL">
                    <simple>${body}</simple>
                </setHeader>
                <setBody>
                    <simple>${null}</simple>
                </setBody>
                <to uri="direct:go"/>
            </split>
        </route>

        <route id="testing_get">
            <from uri="direct:go"/>
            <setHeader headerName="HTTP_METHOD">
                <constant>GET</constant>
            </setHeader>
            <recipientList>
                <simple>http://${in.header.currentURL}</simple>
            </recipientList>
            <to
uri="file:/project/data?fileName=${date:now:yyyyMMddHHmmss.SSSSSS}.json"/>
        </route>


The above works if the list of URL is less than 90 but if it's more
than that, all the requests are getting

On delivery attempt: 0 caught: java.net.SocketException: Connection reset


I've also tried using:

            <to uri="bean:dataProcessor?method=createUrls"/>
           <removeHeaders pattern="CamelHttp*"/>
            <removeHeaders pattern="Http*"/>
            <split>
                <header>codeURLS</header>
                <setHeader headerName="currentUrl">
                    <simple>${body}</simple>
                </setHeader>
                <log message="Current URL: ${in.header.currentUrl}"/>
                <setBody>
                    <simple>${null}</simple>
                </setBody>
                <setHeader headerName="Exchange.HTTP_URI">
                    <simple>${in.header.currentUrl}</simple>
                </setHeader>
                <to uri="http://dummyhost"/>
                <to
uri="file:/tmp/output?fileName=${date:now:yyyyMMddHHmmss.SSSSSS}.js"/>


but same thing. It works for few URLS but anything more throws Connect reset.

Any idea what's the best way to do this?


I'm using Camel ver 2.12.3 and JDK 7


Thanks,
/Vins