Manipulating JSON from inputStream

[erd <ev...@gmail.com>]

Hello,

I am  building a camel route and I am looking to manipulate an inputStream
formatted as JSON into a different "hierarchy". 

The endgame is to turn the service's response

{"foo":
     {"bar":{"type": "motor","status":"spinning"}},
     {"baz":{"type": "calculator","status":"crunching"}}
}
into a series of JSON strings with name and properties a la:
{"foo":
{"name":"bar", properties:{"type": "motor","status":"spinning"}}
{"name":"baz", properties:{"type": "calculator","status":"crunching"}}
}

What's the best way to approach this?



--
View this message in context: http://camel.465427.n5.nabble.com/Manipulating-JSON-from-inputStream-tp5766240.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Manipulating JSON from inputStream

[Reji Mathews <co...@gmail.com>]

Hello

I had similar use case for one of my customers. It was a bulk dump of
electricity consumption data of subscribers which was to be loaded into
charging engine. The records ran to millions.

I doubt if there is something out of box for json to json transformation. I
did following

1) Source Json ->Equalent Source XML  ( json unmarshalling component in
camel )
2) Equalent Source XML->Equalent Target XML ( transformation using
xslt,xquery etc)
3) Equalent Target XML -> Equalent Target Json ( xml-json marshalling )

Note - Don't attempt step 1 with huge payloads. It kills the memory and i
had a crash.

I did some java coding to chunk out the huge json into smaller chunks and
processed each of them in parallel. You can adopt some cool algo's for
chunking. I adopted stack approach. a small snippet of the code below using
java.util.Stack class.

Read json char by char and pass to this function. When it returns a boolean
true, that means you now have the point where you have a valid json. Pls
manage the outer envelops / roots if any.


try {
char current;
int counter = 0;
while (stream.available() > 0) {
jsonProcessed=false;
current = (char) stream.read();
validJson.append(current);
if (current == '{' || current == '(' || current == ')'
|| current == '}') {

if (IsValidJsonRecord(current) && validJson.length()>500000) {
/* System.out.println((++counter) + "This is valid"
+ validJson.toString());*/
producer.sendBody("vm:SplittedRecords",
"{\"CollectDataList\":{\"CollectData\":["+validJson.toString()+"]}}");
//The outer envelopes being attached here to the chunks
jsonProcessed=true;
validJson.delete(0, validJson.length()); //Clearing the buffer variable
stream.read(); // To skip the comma
}
else if (stream.available() < 1 && validJson.length()>0 &&
jsonProcessed==false)
{
System.out.println("PROCESSINGLASTRECORD- "+  validJson.toString());
producer.sendBody("vm:SplittedRecords",
"{\"CollectDataList\":{\"CollectData\":["+validJson.toString()+"]}}");
jsonProcessed=true;
 }
}

}
} catch (java.util.EmptyStackException e){}


private static boolean IsValidJsonRecord(char c) {
// TODO Auto-generated method stub
if (c == '(')
stack.push(c);
else if (c == '{')
stack.push(c);
else if (c == ')') {
if (stack.peek() == '(') {
stack.pop();
return stack.empty();
} else {
stack.push(c);
return false;
}
}

else if (c == '}') {
if (stack.peek() == '{') {
stack.pop();
return stack.empty();
} else {
stack.push(c);
return false;
}
}

return stack.empty();

}
}



Cheers
Reji

On Thu, Apr 23, 2015 at 10:23 AM, erd <ev...@gmail.com> wrote:

> Hello,
>
> I am  building a camel route and I am looking to manipulate an inputStream
> formatted as JSON into a different "hierarchy".
>
> The endgame is to turn the service's response
>
> {"foo":
>      {"bar":{"type": "motor","status":"spinning"}},
>      {"baz":{"type": "calculator","status":"crunching"}}
> }
> into a series of JSON strings with name and properties a la:
> {"foo":
> {"name":"bar", properties:{"type": "motor","status":"spinning"}}
> {"name":"baz", properties:{"type": "calculator","status":"crunching"}}
> }
>
> What's the best way to approach this?
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Manipulating-JSON-from-inputStream-tp5766240.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Re: Manipulating JSON from inputStream

[Mark Frazier <mm...@me.com>]

Use the Message Translator EIP

http://camel.apache.org/message-translator.html <http://camel.apache.org/message-translator.html>


Inside that,you can use GSON and do one of two:

1) use the GSON api to directly manipulate JsonObject instances after you unmarshall

2) unmarshall into an object model that you define, translate from one object model to another in java, then
marshal back into JSON


> On Apr 23, 2015, at 10:23 AM, erd <ev...@gmail.com> wrote:
> 
> Hello,
> 
> I am  building a camel route and I am looking to manipulate an inputStream
> formatted as JSON into a different "hierarchy". 
> 
> The endgame is to turn the service's response
> 
> {"foo":
>     {"bar":{"type": "motor","status":"spinning"}},
>     {"baz":{"type": "calculator","status":"crunching"}}
> }
> into a series of JSON strings with name and properties a la:
> {"foo":
> {"name":"bar", properties:{"type": "motor","status":"spinning"}}
> {"name":"baz", properties:{"type": "calculator","status":"crunching"}}
> }
> 
> What's the best way to approach this?
> 
> 
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Manipulating-JSON-from-inputStream-tp5766240.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Manipulating JSON from inputStream

[erd <ev...@gmail.com>]

Hi,
Sorry for going AWOL for a while... anyways, I'm in the process of doing
essentially what Mark suggested. I ended up splitting the large file, using
Jackson and JAXB to unmarshal the entries into a TreeMap, translating the
TreeMap into my own defined POJO, and marshaling back into JSON. Another
curveball I am running into is a second level of nesting in some of the
data. I'm fixing that by making a class to define the types of each element
within the properties.



--
View this message in context: http://camel.465427.n5.nabble.com/Manipulating-JSON-from-inputStream-tp5766240p5766556.html
Sent from the Camel - Users mailing list archive at Nabble.com.