data migration / ETL

[daniel williams <da...@gmail.com>]

My organization has run into a handful of issues around migration of
structures of documents from varying versions of software built upon
couchdb.  Has anyone in the nets come up with a good solution for structural
changes of documents that coincide with major/minor/maintenance releases of
software leverage couchdb?  perhaps a couch ETL?

Thanks.

dan

Re: data migration / ETL

[Nestor Urquiza <ne...@gmail.com>]

Hi,

I would recommend looking at Talend Open Studio (TOS) Probably the
JSON components will not be enough but they could easily be extended.

I suggest opening a question in TOS forum about having this support.
If anyone decides to do so I would appreciate replying back with the
link to the post either to the mailing list or my gmail address. I
know I will be jumping into the same question soon once our product
starts producing.

Thanks!
-Nestor

Re: data migration / ETL

[Ryan Ramage <ry...@gmail.com>]

There is Costco:
https://github.com/harthur/costco

Or Recline:
https://github.com/maxogden/recline

Not sure if those will have enough horsepower for you though.

After reading your question again, it seems to be more about release
management...the versions of software using docs in couch, and
managing the document 'schema'.  In the sql world I have used liqibase
[http://www.liquibase.org/manual/home]. Although I find schema-less
much more relaxing,  I could see some cases where a tool like
liquibase for couchdb would be useful.

Ryan


On Fri, Oct 21, 2011 at 8:20 AM, Sean Copenhaver
<se...@gmail.com> wrote:
> I want to say someone announced a basic utility that safely rolls through
> all docs in the database and performs a task. I think it was a node.js
> command line tool and you provided it a script to run on each document. Not
> positive.
>
> If you have a view that gives you all the documents based on a version
> attribute, then you could roll through all documents in the view for a
> particular version (null, 1.1, etc) perform your ETL process, update the
> version, and store them back. Eventually you should have no documents with a
> version below the one you are migrating to. Only concern is if you run into
> conflict because someone updates the doc while you are migrating it, but I
> suppose in this situation you would follow your normal conflict resolution
> then migrate the resulting doc.
>
> Anyway just some thoughts off the top of my head. I hope that helps some.
>
> On Tue, Oct 18, 2011 at 9:13 AM, daniel williams
> <da...@gmail.com>wrote:
>
>> My organization has run into a handful of issues around migration of
>> structures of documents from varying versions of software built upon
>> couchdb.  Has anyone in the nets come up with a good solution for
>> structural
>> changes of documents that coincide with major/minor/maintenance releases of
>> software leverage couchdb?  perhaps a couch ETL?
>>
>> Thanks.
>>
>> dan
>>
>
>
>
> --
> “The limits of language are the limits of one's world. “ - Ludwig von
> Wittgenstein
>
> "Water is fluid, soft and yielding. But water will wear away rock, which is
> rigid and cannot yield. As a rule, whatever is fluid, soft and yielding will
> overcome whatever is rigid and hard. This is another paradox: what is soft
> is strong." - Lao-Tzu
>


-- 
Twitter: @eckoit

Re: data migration / ETL

[Sean Copenhaver <se...@gmail.com>]

I want to say someone announced a basic utility that safely rolls through
all docs in the database and performs a task. I think it was a node.js
command line tool and you provided it a script to run on each document. Not
positive.

If you have a view that gives you all the documents based on a version
attribute, then you could roll through all documents in the view for a
particular version (null, 1.1, etc) perform your ETL process, update the
version, and store them back. Eventually you should have no documents with a
version below the one you are migrating to. Only concern is if you run into
conflict because someone updates the doc while you are migrating it, but I
suppose in this situation you would follow your normal conflict resolution
then migrate the resulting doc.

Anyway just some thoughts off the top of my head. I hope that helps some.

On Tue, Oct 18, 2011 at 9:13 AM, daniel williams
<da...@gmail.com>wrote:

> My organization has run into a handful of issues around migration of
> structures of documents from varying versions of software built upon
> couchdb.  Has anyone in the nets come up with a good solution for
> structural
> changes of documents that coincide with major/minor/maintenance releases of
> software leverage couchdb?  perhaps a couch ETL?
>
> Thanks.
>
> dan
>



-- 
“The limits of language are the limits of one's world. “ - Ludwig von
Wittgenstein

"Water is fluid, soft and yielding. But water will wear away rock, which is
rigid and cannot yield. As a rule, whatever is fluid, soft and yielding will
overcome whatever is rigid and hard. This is another paradox: what is soft
is strong." - Lao-Tzu