Re: hello from granary

[sblackmon <sb...@apache.org>]

Ryan,

Thanks for the note.  Good to know that another effort like this is happening in the python world.

It would be really cool if we could write and apply (and share) JSON document conversions using a cross-language DSL.

We’ve done a bit of that using HOCON (see links below) but there’s only so much that can do, and anything beyond that we have to write code - so nearly all converters wind up written with code.

https://github.com/apache/incubator-streams/blob/master/streams-components/streams-converters/src/test/java/org/apache/streams/converter/test/HoconConverterTest.java

https://github.com/apache/incubator-streams/tree/master/streams-components/streams-converters/src/test/resources

Best,
Steve

On June 15, 2016 at 12:16:07 AM, Ryan B (public@ryanb.org) wrote:
hi steve, all! i just discovered apache streams. it looks great! can't wait  
to dig in more.  

i'm the main developer on a similar project called granary:  

https://github.com/snarfed/granary  
https://granary-demo.appspot.com  

it has a lot of the same functionality as streams. it's focused on social  
networking data, so it doesn't do databases or data processing pipelines  
like streams, but it supports writing to silo APIs as well as reading them,  
and it supports a few formats streams doesn't: Flickr, Atom, microformats  
HTML and JSON.  

we've spent a fair amount of effort adapting to silo changes over time, eg  
Facebook's 2.x API, reconstructing Twitter @-reply chains, scraping  
Instagram's HTML now that they've disabled their API for many apps. I'd be  
happy to compare notes if you're interested.  

keep up the good work!  

--  
https://snarfed.org/  

Re: hello from granary

[Ryan B <pu...@ryanb.org>]

On Tue, Sep 27, 2016 at 10:37 AM, sblackmon <sb...@apache.org> wrote:

>
> It would be really cool if we could write and apply (and share) JSON
> document conversions using a cross-language DSL.
>
> We’ve done a bit of that using HOCON (see links below) but there’s only so
> much that can do, and anything beyond that we have to write code - so
> nearly all converters wind up written with code.
>

interesting! agreed, declarative schema mapping tools like that have been
around forever, especially in academia, and they can definitely work and be
more maintainable than code! you're right though, they can't usually do the
whole thing on their own, so we often need to fall back to code for some
things.

keep us posted if you end up using HOCON or something similar more broadly!



> https://github.com/apache/incubator-streams/blob/master/
> streams-components/streams-converters/src/test/java/org/
> apache/streams/converter/test/HoconConverterTest.java
>
> https://github.com/apache/incubator-streams/tree/master/
> streams-components/streams-converters/src/test/resources
>
> Best,
> Steve
>
> On June 15, 2016 at 12:16:07 AM, Ryan B (public@ryanb.org) wrote:
>
> hi steve, all! i just discovered apache streams. it looks great! can't
> wait
> to dig in more.
>
> i'm the main developer on a similar project called granary:
>
> https://github.com/snarfed/granary
> https://granary-demo.appspot.com
>
> it has a lot of the same functionality as streams. it's focused on social
> networking data, so it doesn't do databases or data processing pipelines
> like streams, but it supports writing to silo APIs as well as reading
> them,
> and it supports a few formats streams doesn't: Flickr, Atom, microformats
> HTML and JSON.
>
> we've spent a fair amount of effort adapting to silo changes over time, eg
> Facebook's 2.x API, reconstructing Twitter @-reply chains, scraping
> Instagram's HTML now that they've disabled their API for many apps. I'd be
> happy to compare notes if you're interested.
>
> keep up the good work!
>
> --
> https://snarfed.org/
>
>


-- 
https://snarfed.org/