ETL with Ignite

[Konstantin Boudnik <co...@apache.org>]

Guys,

We have a data-pipeline ingesting some data into Ignite cache from a JSON
source. While JSON ingestion isn't an issue by itself, it seems that there's
no mechanism in place to detect and create a schema class for JSON data
structure on the fly.

I know there's a functionality to automatically detect and import DB schemas
(which is an easier thing to do, considering the availability of declarative
DDL). I wonder if there's something we've overlooked wrt JSON? If not, any
ideas on how to proceed with such implementation would be very appreciated!

Thanks,
  Cos

Re: ETL with Ignite

[Dmitriy Setrakyan <ds...@apache.org>]

Cos,

I see that you have already found the JSON ticket:
https://issues.apache.org/jira/browse/IGNITE-962

This ticket will make it possible to index JSON object in Ignite cache and
use them in SQL queries. If you don’t need indexes, why not just store
JSONs as String objects or byte arrays?

D.

On Fri, Nov 13, 2015 at 11:40 AM, Konstantin Boudnik <co...@apache.org> wrote:

> Guys,
>
> We have a data-pipeline ingesting some data into Ignite cache from a JSON
> source. While JSON ingestion isn't an issue by itself, it seems that
> there's
> no mechanism in place to detect and create a schema class for JSON data
> structure on the fly.
>
> I know there's a functionality to automatically detect and import DB
> schemas
> (which is an easier thing to do, considering the availability of
> declarative
> DDL). I wonder if there's something we've overlooked wrt JSON? If not, any
> ideas on how to proceed with such implementation would be very appreciated!
>
> Thanks,
>   Cos
>