You are viewing a plain text version of this content. The canonical link for it is here.
Posted to j-users@xalan.apache.org by "Eric J. Schwarzenbach" <Er...@wrycan.com> on 2014/09/30 22:02:13 UTC
No more DTM IDs are available
Hi,
We're getting the dreaded "No more DTM IDs are available" and are
looking for some pointers on how to debug this.
First of all, yes we know this is (or at least was with past JDKs)
usually an issue with people either running an old version of Xalan or
picking up the JDK's built-in branch of Xalan instead of the intended
Xalan jar, and we are positive this is not the case. We've checked
object classnames and checked the versions with
org.apache.xalan.xslt.EnvironmentCheck and are definitely using 2.7.2
(2.7.1 until today, as I wanted to make sure this newest version made no
difference).
Our application is probably putting more stress on Xalan than most, as
there is a lot of calling to Java extension functions from our XSLs some
of which return large NodeLists, and some of which even kick off
additional transforms. We've been using this application like this for
years, with pretty large data sets, and the only thing that has changed
to cause us to run into the error is some combination of the amount of
data and the particular XSLs.
So what I'm looking for are some tips on how to debug or ameliorate
this. Are there particular scenarios / operations where Xalan is
particularly likely to use (or hold onto) more DTMs than usual. Is there
any debugging we can turn on to have it output any DTM usage info that
we could perhaps use to determine where is the course of our XSL
execution DTM usage starts getting out of hand?
Worse come to worst we will dig into the source and see if we can add
logging of our own to debug this, or perhaps mess with changing
IDENT_DTM_NODE_BITS and DEFAULT_BLOCKSIZE, which some posts on this list
(from 2006!) say can help with this. But we'd rather explore other
options before going there...
Thanks,
Eric
Re: No more DTM IDs are available
Posted by Joseph Kesselman/Watson/IBM <ke...@us.ibm.com>.
I believe the DTM cache/pool is per Transformer object.
______________________________________
"Everything should be as simple as possible. But not simpler." --
attributed to Albert Einstein
From:
"Eric J. Schwarzenbach" <Er...@wrycan.com>
To:
j-users@xalan.apache.org
Date:
09/30/2014 05:11 PM
Subject:
Re: No more DTM IDs are available
Thanks!
What is the scope of these DTM limitations? Is it per Transformer object?
TransformerFactory? Xalan as a whole?
Eric
On 09/30/2014 04:52 PM, Joseph Kesselman/Watson/IBM wrote:
A DTM ID is consumed by each document tree Xalan is working with.
Temporary result trees, for example, each occupy a DTM.
For reasons having to do with DTM's history and the use cases it was tuned
for, there are a fixed number of bits that need to be split between
selecting DTMs and selecting nodes within a DTM. If you're using many
small trees, reducing the number of bits used for node selection will
increase the number of trees available, at the cost of reducing the space
in each DTM.
At one point, IBM modified the DTM code so documents could overflow from
one DTM into another. That allowed us to push the DTM Node Bits down
(allowing more documents) while still being able to handle larger
documents. I don't remember whether that change was checked into Xalan or
was applied only to the Xalan derivative IBM was shipping in its own JDKs,
but if you're ambitious you could go digging through the code to check on
that.
Not that it helps Apache, but IBM's second-generation XSLT processor
(which ships with the recent IBM JDKs, and in an enhanced version as the
WebSphere XML Feature), redesigned the document model code to avoid the
multiplicity-versus-size tradeoff.
______________________________________
"Everything should be as simple as possible. But not simpler." --
attributed to Albert Einstein
From:
"Eric J. Schwarzenbach" <Er...@wrycan.com>
To:
j-users@xalan.apache.org
Date:
09/30/2014 04:06 PM
Subject:
No more DTM IDs are available
Hi,
We're getting the dreaded "No more DTM IDs are available" and are
looking for some pointers on how to debug this.
First of all, yes we know this is (or at least was with past JDKs)
usually an issue with people either running an old version of Xalan or
picking up the JDK's built-in branch of Xalan instead of the intended
Xalan jar, and we are positive this is not the case. We've checked
object classnames and checked the versions with
org.apache.xalan.xslt.EnvironmentCheck and are definitely using 2.7.2
(2.7.1 until today, as I wanted to make sure this newest version made no
difference).
Our application is probably putting more stress on Xalan than most, as
there is a lot of calling to Java extension functions from our XSLs some
of which return large NodeLists, and some of which even kick off
additional transforms. We've been using this application like this for
years, with pretty large data sets, and the only thing that has changed
to cause us to run into the error is some combination of the amount of
data and the particular XSLs.
So what I'm looking for are some tips on how to debug or ameliorate
this. Are there particular scenarios / operations where Xalan is
particularly likely to use (or hold onto) more DTMs than usual. Is there
any debugging we can turn on to have it output any DTM usage info that
we could perhaps use to determine where is the course of our XSL
execution DTM usage starts getting out of hand?
Worse come to worst we will dig into the source and see if we can add
logging of our own to debug this, or perhaps mess with changing
IDENT_DTM_NODE_BITS and DEFAULT_BLOCKSIZE, which some posts on this list
(from 2006!) say can help with this. But we'd rather explore other
options before going there...
Thanks,
Eric
Re: No more DTM IDs are available
Posted by "Eric J. Schwarzenbach" <Er...@wrycan.com>.
Thanks!
What is the scope of these DTM limitations? Is it per Transformer
object? TransformerFactory? Xalan as a whole?
Eric
On 09/30/2014 04:52 PM, Joseph Kesselman/Watson/IBM wrote:
> A DTM ID is consumed by each document tree Xalan is working with.
> Temporary result trees, for example, each occupy a DTM.
>
> For reasons having to do with DTM's history and the use cases it was
> tuned for, there are a fixed number of bits that need to be split
> between selecting DTMs and selecting nodes within a DTM. If you're
> using many small trees, reducing the number of bits used for node
> selection will increase the number of trees available, at the cost of
> reducing the space in each DTM.
>
> At one point, IBM modified the DTM code so documents could overflow
> from one DTM into another. That allowed us to push the DTM Node Bits
> down (allowing more documents) while still being able to handle larger
> documents. I don't remember whether that change was checked into Xalan
> or was applied only to the Xalan derivative IBM was shipping in its
> own JDKs, but if you're ambitious you could go digging through the
> code to check on that.
>
> Not that it helps Apache, but IBM's second-generation XSLT processor
> (which ships with the recent IBM JDKs, and in an enhanced version as
> the WebSphere XML Feature), redesigned the document model code to
> avoid the multiplicity-versus-size tradeoff.
>
>
> ______________________________________
> "Everything should be as simple as possible. But not simpler." --
> attributed to Albert Einstein
>
>
> From: "Eric J. Schwarzenbach" <Er...@wrycan.com>
> To: j-users@xalan.apache.org
> Date: 09/30/2014 04:06 PM
> Subject: No more DTM IDs are available
>
>
> ------------------------------------------------------------------------
>
>
>
> Hi,
>
> We're getting the dreaded "No more DTM IDs are available" and are
> looking for some pointers on how to debug this.
>
> First of all, yes we know this is (or at least was with past JDKs)
> usually an issue with people either running an old version of Xalan or
> picking up the JDK's built-in branch of Xalan instead of the intended
> Xalan jar, and we are positive this is not the case. We've checked
> object classnames and checked the versions with
> org.apache.xalan.xslt.EnvironmentCheck and are definitely using 2.7.2
> (2.7.1 until today, as I wanted to make sure this newest version made no
> difference).
>
> Our application is probably putting more stress on Xalan than most, as
> there is a lot of calling to Java extension functions from our XSLs some
> of which return large NodeLists, and some of which even kick off
> additional transforms. We've been using this application like this for
> years, with pretty large data sets, and the only thing that has changed
> to cause us to run into the error is some combination of the amount of
> data and the particular XSLs.
>
> So what I'm looking for are some tips on how to debug or ameliorate
> this. Are there particular scenarios / operations where Xalan is
> particularly likely to use (or hold onto) more DTMs than usual. Is there
> any debugging we can turn on to have it output any DTM usage info that
> we could perhaps use to determine where is the course of our XSL
> execution DTM usage starts getting out of hand?
>
> Worse come to worst we will dig into the source and see if we can add
> logging of our own to debug this, or perhaps mess with changing
> IDENT_DTM_NODE_BITS and DEFAULT_BLOCKSIZE, which some posts on this list
> (from 2006!) say can help with this. But we'd rather explore other
> options before going there...
>
> Thanks,
>
> Eric
>
>
Are DTM IDs ever "recycled"?
Posted by "Eric J. Schwarzenbach" <er...@wrycan.com>.
I have a question regarding DTM IDs and the dreaded "No more DTM IDs are
available" error. (No, not using an old version Xalan, just creating
lots of temporary nodesets during processing.) Are DTM IDs ever
"recycled"? Does a node set variable going out of scope ever free up the
DTM IDs that it was using for reuse by a later nodeset?
(I'm including Joseph Kesselman's comments about how these work from a
previous thread, just for reference.)
Thanks,
Eric
On 09/30/2014 04:52 PM, Joseph Kesselman/Watson/IBM wrote:
> A DTM ID is consumed by each document tree Xalan is working with.
> Temporary result trees, for example, each occupy a DTM.
>
> For reasons having to do with DTM's history and the use cases it was
> tuned for, there are a fixed number of bits that need to be split
> between selecting DTMs and selecting nodes within a DTM. If you're
> using many small trees, reducing the number of bits used for node
> selection will increase the number of trees available, at the cost of
> reducing the space in each DTM.
>
Re: No more DTM IDs are available
Posted by Joseph Kesselman/Watson/IBM <ke...@us.ibm.com>.
A DTM ID is consumed by each document tree Xalan is working with.
Temporary result trees, for example, each occupy a DTM.
For reasons having to do with DTM's history and the use cases it was tuned
for, there are a fixed number of bits that need to be split between
selecting DTMs and selecting nodes within a DTM. If you're using many
small trees, reducing the number of bits used for node selection will
increase the number of trees available, at the cost of reducing the space
in each DTM.
At one point, IBM modified the DTM code so documents could overflow from
one DTM into another. That allowed us to push the DTM Node Bits down
(allowing more documents) while still being able to handle larger
documents. I don't remember whether that change was checked into Xalan or
was applied only to the Xalan derivative IBM was shipping in its own JDKs,
but if you're ambitious you could go digging through the code to check on
that.
Not that it helps Apache, but IBM's second-generation XSLT processor
(which ships with the recent IBM JDKs, and in an enhanced version as the
WebSphere XML Feature), redesigned the document model code to avoid the
multiplicity-versus-size tradeoff.
______________________________________
"Everything should be as simple as possible. But not simpler." --
attributed to Albert Einstein
From:
"Eric J. Schwarzenbach" <Er...@wrycan.com>
To:
j-users@xalan.apache.org
Date:
09/30/2014 04:06 PM
Subject:
No more DTM IDs are available
Hi,
We're getting the dreaded "No more DTM IDs are available" and are
looking for some pointers on how to debug this.
First of all, yes we know this is (or at least was with past JDKs)
usually an issue with people either running an old version of Xalan or
picking up the JDK's built-in branch of Xalan instead of the intended
Xalan jar, and we are positive this is not the case. We've checked
object classnames and checked the versions with
org.apache.xalan.xslt.EnvironmentCheck and are definitely using 2.7.2
(2.7.1 until today, as I wanted to make sure this newest version made no
difference).
Our application is probably putting more stress on Xalan than most, as
there is a lot of calling to Java extension functions from our XSLs some
of which return large NodeLists, and some of which even kick off
additional transforms. We've been using this application like this for
years, with pretty large data sets, and the only thing that has changed
to cause us to run into the error is some combination of the amount of
data and the particular XSLs.
So what I'm looking for are some tips on how to debug or ameliorate
this. Are there particular scenarios / operations where Xalan is
particularly likely to use (or hold onto) more DTMs than usual. Is there
any debugging we can turn on to have it output any DTM usage info that
we could perhaps use to determine where is the course of our XSL
execution DTM usage starts getting out of hand?
Worse come to worst we will dig into the source and see if we can add
logging of our own to debug this, or perhaps mess with changing
IDENT_DTM_NODE_BITS and DEFAULT_BLOCKSIZE, which some posts on this list
(from 2006!) say can help with this. But we'd rather explore other
options before going there...
Thanks,
Eric