too large XSP logicsheet? aka How to cut XML in to pieces and generate from each one independly (using Cocoon not text editor ;) )

[Cocoon User <in...@vp.pl>]

Hi,

I've got a problem with too large XSP logicsheet (hypothesis).

Problem origin:
    I must INSERT INTO a Postgres database a large amount of data from
XML source.
    XML source is something about 1MB. Lots of records like:

<ENTRY start="2005-03-05 05:45" next="2005-03-05 06:15" duration="26"
language="PL" age_category="NA">
  <TITLE>Pod słońcem Afryki</TITLE>
  <INSTALMENT number="22/26" />
  <DESCRIPTION short="">Serial o przyrodzie Afryki. Autorzy filmu
koncentrują się na problemach związanych z ochroną        zagrożonych
gatunków i ekosystemów.</DESCRIPTION>
  <PRODUCTION country="RPA" />
  <HIT>no</HIT>
  <CATEGORY formal="SER" a="DOK">serial dokumentalny</CATEGORY>
  <FEATURES x="no" k="no" z="no" p="no" s="no" b="no" m="no" d="no" />
</ENTRY>

I thought it would work fine with one big XSP generated by XSL
transformer:
for each <ENTRY> in source XML target XSP gets:

<esql:connection>
    <esql:pool>geozeta</esql:pool>
      <xsp:logic>
        cal.set(<xsl:value-of select="$y"/>,<xsl:value-of select="$m -
1"/>,<xsl:value-of select="$d"/>);
        dayofweek = cal.get(Calendar.DAY_OF_WEEK);
      </xsp:logic>
      <result>
      <esql:execute-query>
        <esql:query>
          INSERT INTO programs ( Channel, Title, Chapter, MaxChapters,
Description, ProductionCountry, Category, Date, DayOfWeek, StartTime,
EndTime)
          VALUES ('<xsl:value-of select="parent::*/@name"/>'
          ,<esql:parameter><xsl:value-of
select="normalize-space(XSLTExtension:escapeDoubleQuotes(TITLE))"/></esq
l:parameter>
          ,0<xsl:value-of
select="substring-before(INSTALMENT/@number,'/')"/>
          ,0<xsl:value-of
select="substring-after(INSTALMENT/@number,'/')"/>
          ,<esql:parameter><xsl:value-of
select="normalize-space(XSLTExtension:escapeDoubleQuotes(DESCRIPTION))"/
></esql:parameter>
          ,'<xsl:value-of select="PRODUCTION/@country"/>'
          ,'<xsl:value-of select="CATEGORY"/>','<xsl:value-of
select="substring-before(@start,' ')"/>'
          ,<xsp:expr>dayofweek</xsp:expr>
          ,'<xsl:value-of select="substring-after(@start,'
')"/>','<xsl:value-of select="substring-after(@next,' ')"/>');
        </esql:query>
        <esql:results>Dodano do bazy</esql:results>
        <esql:no-results>Nie dodano do bazy</esql:no-results>
        <esql:error-results>Error typu Błont</esql:error-results>
      </esql:execute-query>
      </result>
</esql:connection>


(Each INSERT has its own connection in case of possible connection
timeout postgres feature)

It works fine for several <ENTRY> tags but when I try to run pipeline
for whole XML (really big number of <ENTRY>) it chokes for several
minutes then browser returns err. 500. Nothing is inserted into DB.

My guess is: too large XSP. (XML structure and content defects would
return different errors probably)

So I'm looking for some way to iteratively cut source XML into pieces,
transform into smaller XSP's and generate from each indepedently and
sequentially. But I can't find a way how to do it in Cocoon. (maybe flow
control could be of some use).

Any suggetions, maybe some smarter way how to insert a lot of data from
XML to DB, or any hint about a possible trival mistake i might have
done, would be appreciated.

Thanx and sorry for my clumsiness in en.

-- 
Marcel Gołuński
  Netventure, http://www.netventure.pl/
  ul. Kozłowskiej 1, 00-710 Warszawa
  tel.: (0-22) 6519430, fax: (0-22) 6518604



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org