Xindice internals

[James Bates <ja...@amplexor.com>]

I've been looking closely at some of the core code in Xindice, that handles 
filing, collection organization, querying etc... and in order to do some real
"core" development, I feal some kind of documentation explaining HOW Xindice 
works internally would be a useful, if not necessary addition.

I have therefore started a "Xindice internals" guide in the Forrest 
documentation tree that should explain these various aspects. I was not 
involved in the initial core design, so some of the things I'm assuming may 
not be true, but only an original core developer (Kimbro? Tom?) would know 
how to correct them.

An outline of table of contents is included, as are the first few chapters. 
I'm still working on the rest.

I've been reading up in some database-theory books as to how we might start 
tackling true transactioning, coupled with failure recovery.  There's quite a 
bit of work still starting from thye current state of Xindice, but after my 
internals guide is finished, it should become more clear how to proceed...

Comments are, as always, welcome.

The file is in src/documentation/content/xdocs/dev/guide-internals.xml btw, 
and a link to it exists in the book.xml file in dev/ (as I don't know how to 
tell Forrest to render the file if it's not referenced).

James

Re: Xindice internals

[Gianugo Rabellino <gi...@apache.org>]

James Bates wrote:

> I've been looking closely at some of the core code in Xindice, that 
> handles
> filing, collection organization, querying etc... and in order to do 
> some real
> "core" development, I feal some kind of documentation explaining HOW 
> Xindice
> works internally would be a useful, if not necessary addition.
>
> I have therefore started a "Xindice internals" guide in the Forrest
> documentation tree that should explain these various aspects. 


YES! This is great news, James. Looks like we are in sync: I've just 
started to look at some core optimizations and possible improvements (in 
speed & scalability & document size and so on). I just came back from 
ApacheCon where everyone was talking about Xindice with almost nobody 
using it in production due to its current limitations, so I feel this is 
a very good time to start thinking about 2.0.

I hope that Tom will join us in this effort: I'm currently reading 
papers about storage of semi-structured data (take a look at 
http://www-db.stanford.edu/lore/pubs/index.html) and I have a few ideas 
about new filers and indexes. Time to start again? I will have a good 
amount of time to devote to OSS again in a short while, and I'm willing 
to tackle Xindice seriously. Ready to start? :-)

Oh, and great kudos to Vladimir for the impressive work being done on 
the docs: I will start contributing shortly, expect some RT Real Soon Now.

Ciao,

-- 
Gianugo Rabellino

Re: Xindice internals

["Vladimir R. Bossicard" <vl...@apache.org>]

James,

> Comments are, as always, welcome.

Wouah!  I can't wait to print it and read it!

I would even promote the guide into the main documentation.  But that's 
another story.

Once again, very nice job.

-Vladimir

-- 
Vladimir R. Bossicard
Apache Xindice - http://xml.apache.org/xindice

SysSymbols

[Lixin Meng <lx...@yahoo.com>]

I the 1.1b on Win2K, and tried to run
	java/test/src/org.apache.xindice.IntegrationEmbedTests.java

All 34 test cases were succeed.

I noticed that the master test suite calls CollectionTest.java, which
creates and drops collections. However, after above the unit test, I wrote
some code to exam all collections and found the db/system/SysSymbols is not
empty (as below). I would assume that after dropping a collection, all those
symbols associated with the collection should be dropped too.

Did I miss anything here?
Lixin

------------------ output of my code ------
---------- collections : ------system
---------- collections : ------SysConfig
---------- docs : ------database.xml
---------- collections : ------SysSymbols
---------- docs : ------system_SysConfig
---------- docs : ------testing
---------- docs : ------testing_current
---------- docs : ------testing_current_child1
---------- docs : ------testing_current_child2
---------- docs : ------testing_current_childcol
---------- docs : ------testing_current_colcount
---------- docs : ------testing_current_count
---------- docs : ------testing_current_droptwice
---------- docs : ------testing_current_duplicate
---------- docs : ------testing_current_getname
---------- docs : ------testing_current_nested
---------- docs : ------testing_current_nested_child2
---------- docs : ------testing_current_nested_child3
--------------------------------------------

----------------- my code ----------------
	// starts with 'db'
	private void iterateCollections(XmlDbClient client, String parent) {
		try {
			iterateDocuments(client, parent);
			String [] collections = client.listCollections(parent);
			for (int i = 0; i < collections.length; i++) {
				System.out.println("---------- collections : ------" + collections[i]);
				String childPath = parent + "/" + collections[i];
				iterateCollections(client, childPath);
			}
		} catch (Exception e) {
			e.printStackTrace();
		}

	}

	private void iterateDocuments(XmlDbClient client, String path) {
		try {
			String [] docs = client.listDocuments(path);
			for (int i = 0; i < docs.length; i++) {
				System.out.println("---------- docs : ------" + docs[i]);
			}
		} catch (Exception e) {
			// ignore it, especially for 'db' and 'db/system'
		}

	}

------------------------------------------