DIH transformer script size limitations with Jetty?

[harrysmith <ha...@gmail.com>]

I am attempting to use a javascript transformer script within data-config.xml
with the DataImportHandler. It seems if I have over a certain number of
characters in the javascript function I get a jetty error. Below are the
relevant snippets.

>From data-config.xml

<entity name="metadata" query="select * from vw_item_metadata where
item_id=${item.DIVID_PK} AND core_desc_term='${terms.CORE_DESC_TERM}'"
transformer="script:f1"/>

The function f1 is called as part of the transform. I stripped out all logic
and only left comments. If I keep everything up to line 8 comment, all works
well, but if I add line 9, I get a stack trace.

<script><![CDATA[
      function f1(row) {
        	// Comment line 1 - Still working
			// Comment line 2 - Still working
			// Comment line 3 - Still working
			// Comment line 4 - Still working
			// Comment line 5 - Still working
			// Comment line 6 - Still working
			// Comment line 7 - Still working
			// Comment line 8 - Still working
			// Comment line 9 - Toast
        return row;
      }
 ]]></script> 


The above function results in the following stack trace and failure to
import.

_header=[9838890,5990238,m=4036,g=4096,p=4096,c=4096]={http://localhost:8983/sol
r/admin/debug.jsp?handler=/dataimpo}{}
_buffer=[9838890,5990238,m=4036,g=4096,p=4096,c=4096]={http://localhost:8983/sol
r/admin/debug.jsp?handler=/dataimpo}{}
2010-08-11 10:39:27.070::WARN:  handle failed
java.io.IOException: FULL
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.
java:226)
        at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool
.java:442)

It looks like this is a jetty GET vs POST issue? Any idea how to resolve.
-- 
View this message in context: http://lucene.472066.n3.nabble.com/DIH-transformer-script-size-limitations-with-Jetty-tp1091246p1091246.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH transformer script size limitations with Jetty?

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Thu, Aug 12, 2010 at 5:42 AM, harrysmith <ha...@gmail.com> wrote:

>
> To follow up on my own question, it appears this is only an issue when
> using
> the DataImport console debugging tools. It looks like when submitting the
> debugging request, the data-config.xml is sent via a GET request, which
> would fail.  However, using the exact same data-config.xml via a
> full-import
> operation (ie not a dry run debug), it looks like the request is sent POST
> and the import works fine.
>

You are right. In debug mode, the data-config is sent as a GET request. Can
you open a Jira issue?

-- 
Regards,
Shalin Shekhar Mangar.

Re: DIH transformer script size limitations with Jetty?

[Girish Pandit <pa...@gmail.com>]

Have you tried changing the -Xmx value to bump to -Xmx1300m?

I had some problem with DIH loading the data and when I bumped the 
memory everything worked fine!

harrysmith wrote:
> To follow up on my own question, it appears this is only an issue when using
> the DataImport console debugging tools. It looks like when submitting the
> debugging request, the data-config.xml is sent via a GET request, which
> would fail.  However, using the exact same data-config.xml via a full-import
> operation (ie not a dry run debug), it looks like the request is sent POST
> and the import works fine.
>   

Re: DIH transformer script size limitations with Jetty?

[harrysmith <ha...@gmail.com>]

To follow up on my own question, it appears this is only an issue when using
the DataImport console debugging tools. It looks like when submitting the
debugging request, the data-config.xml is sent via a GET request, which
would fail.  However, using the exact same data-config.xml via a full-import
operation (ie not a dry run debug), it looks like the request is sent POST
and the import works fine.
-- 
View this message in context: http://lucene.472066.n3.nabble.com/DIH-transformer-script-size-limitations-with-Jetty-tp1091246p1100285.html
Sent from the Solr - User mailing list archive at Nabble.com.