You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br> on 2013/02/20 19:18:15 UTC
If we Open Source our platform, would it be interesting to you?
Hello All,
I’m sending this email because I think it may be interesting for Solr
users, as this project have a strong usage of Solr platform.
We are strongly considering opening the source of our DMP (Data Management
Platform), if it proves to be technically interesting to other developers /
companies.
More details: http://www.s1mbi0se.com/s1mbi0se_DMP.html
All comments, questions and critics happening at HN:
http://news.ycombinator.com/item?id=5251780
Please, feel free to send questions, comments and critics... We will try to
reply them all.
Regards,
Marcelo
Re: If we Open Source our platform, would it be interesting to
you?
Posted by Chris Hostetter <ho...@fucit.org>.
: We are strongly considering opening the source of our DMP (Data Management
: Platform), if it proves to be technically interesting to other developers /
: companies.
:
: More details: http://www.s1mbi0se.com/s1mbi0se_DMP.html
If you do decide you want to open source your platform, an important
consideration is logistically how you plan to do that.
You could zip up all the source code along with a LICENSE file and toss it
on dropbox and, strictly speaking, you have now "open sourced your
platform" -- but that doesn't mean it's a good idea, or that it will
really benefit you (or anyone else interested in trying ot use, maintain,
improve your code) in the long run.
You have to consider important aspects like: license type, source hosting,
community discussion & bug track hosting, project name branding &
trademarks, release management, etc...
For projects intersted in becoming part of the ASF developer community,
this is what the "Incubator" project is for, and they have a lot of great
docs on what is neccessary for a company to do a code grant to bring an
existing code base into the Apache fold and build a community arround
it...
http://incubator.apache.org/
http://incubator.apache.org/podling-story.html
http://incubator.apache.org/faq.html
http://incubator.apache.org/guides/index.html
http://incubator.apache.org/guides/proposal.html
https://www.apache.org/foundation/how-it-works.html
-Hoss
Re: If we Open Source our platform, would it be interesting to you?
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
Hello James,
First of all, thanks for your feedback. I will try to clarify some
questions bellow.
2013/2/20 Dyer, James <Ja...@ingramcontent.com>
> I only looked at your link super fast, but this seems like a very viable
> alternative to Solr's DIH. DIH does the job fairly well but we've
> struggled to have developers who are willing to maintain it. The problem,
> I think, is that DIH appeals to non-programmers who want to index their
> data without writing (much) code, hence the user base is unable to help out.
>
Indeed, one of the next steps we were considering to improve is the ability
for non-programmers to use pre-made transformations easily, thought a web
interface. In other words, to do transformations like this:
http://www.zorba-xquery.com/html/demo users wouldn't need to do code.
However, we think it's important to assure the possibility to code is still
open without losing features. I like to see it more as a framework to make
the programmers life much easier than as a product for non-programmers,
that's the idea.
> I saw that you can take data from NoSql databases, from a rdbms and
> emails. Can it handle flat tiles, integrate with tika to parse out
> different file formats, etc? Can it join and denormalize data from a
> variety of sources and combine that into 1 flat-schema document?
>
That's exactly what we do. This is our main strengh, as we could capture
data from any source. The idea is that you can write java code to import to
our repository, or you can use some of our already pre-build importers. We
have importers for JSON, CSV and one for DBs nowadays, although the DB one
might be improved a bit. However, you are free to write any java code that
imports from whatever you wants.
The only possible drawback is: we need the repository. We import from
anything and put on Cassandra, so users can capture data first and index it
later. The indexing process is split from the capture process, so we need
somewhere to keep the data. Suppose you capture all data you can about
users today, but only index part of it. In a later time, you could choose
to index the rest, the data is already there.
However, if this is important, we could think in adding the possibility to
import data directly to Solr, without this repo in the middle.
> I could see your project as something we could use in Solr & Lucene, but
> with its appeal to more projects than just ours, possibly would not have
> such a problem attracting developers.
That's exactly the kind of feedback we want, thank you very much.
Best regards,
Marcelo
Re: If we Open Source our platform, would it be interesting to you?
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
Hello Alexandre,
2013/2/20 Alexandre Rafalovitch <ar...@gmail.com>
> How does this compare to Pentaho Kettle? http://kettle.pentaho.com/
>
> I am not terribly familiar with this space, so - if the question is stupid,
> feel free to address it appropriately. :-)
Don't worry, it's not stupid. I will make my best to try to answer to you.
Pentaho Kettle is more to a product, what we have something that is more to
a framework. Besides that, it seems more like an ETL or EAI tool, which you
could use to extract data from some source, apply visual transformations
without needing to code and load them on a database or nosql tool. It's not
entity oriented and it doesn't have it's own repository.
Our project address more specific cases, as we don't just extract data, we
identify data on the process. Suppose for example you have several legacy
systems in your company, like a CRM, an ERP and a billing system. In each
of these systems, you have data related to users, that you want to process
and analyse for each user. What our system does is aggregate all data for
each user in a single logical place, so you can perform actions like:
- suppose you have a replicated field in both CRM and ERP systems. You can
choose which one to use;
- as more billing data is generated, you want to aggregate the new
invoices to the index, without needing to process the entire set of data
again;
- you might have the same data represented differently in you CRM and in
your ERP, so you want to normalize it to a canonical format before choosing
which one to index, or even before indexing both.
This is just a possible scenario and it's not what we do as a company, ok?
Of course you could do these things using a tool like Pentaho, but you
would have to do them manually and you would be narrowed to their
interface. Our system gives you these funcionalities as pre-built and you
keep being able to use any framework or technology you choose to do other
tasks that might be interesting to you in the middle. For instance, you
could easilly integrate a java machine learning framework, call facebook
graph api, use spring or hibernate, or anything like that in your processes.
Hope I was able to answer your question, but I will be glad to try again if
you have more doubts.
Best regards,
Marcelo
Re: If we Open Source our platform, would it be interesting to you?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
How does this compare to Pentaho Kettle? http://kettle.pentaho.com/
I am not terribly familiar with this space, so - if the question is stupid,
feel free to address it appropriately. :-)
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all at
once. Lately, it doesn't seem to be working. (Anonymous - via GTD book)
On Wed, Feb 20, 2013 at 1:37 PM, Dyer, James
<Ja...@ingramcontent.com>wrote:
> I only looked at your link super fast, but this seems like a very viable
> alternative to Solr's DIH. DIH does the job fairly well but we've
> struggled to have developers who are willing to maintain it. The problem,
> I think, is that DIH appeals to non-programmers who want to index their
> data without writing (much) code, hence the user base is unable to help out.
>
> I saw that you can take data from NoSql databases, from a rdbms and
> emails. Can it handle flat tiles, integrate with tika to parse out
> different file formats, etc? Can it join and denormalize data from a
> variety of sources and combine that into 1 flat-schema document?
>
> I could see your project as something we could use in Solr & Lucene, but
> with its appeal to more projects than just ours, possibly would not have
> such a problem attracting developers.
>
> James Dyer
> Ingram Content Group
> (615) 213-4311
>
> -----Original Message-----
> From: Marcelo Elias Del Valle [mailto:marcelo@s1mbi0se.com.br]
> Sent: Wednesday, February 20, 2013 12:18 PM
> To: solr-user@lucene.apache.org
> Subject: If we Open Source our platform, would it be interesting to you?
>
> Hello All,
>
> I'm sending this email because I think it may be interesting for Solr
> users, as this project have a strong usage of Solr platform.
>
> We are strongly considering opening the source of our DMP (Data Management
> Platform), if it proves to be technically interesting to other developers /
> companies.
>
> More details: http://www.s1mbi0se.com/s1mbi0se_DMP.html
>
> All comments, questions and critics happening at HN:
> http://news.ycombinator.com/item?id=5251780
>
> Please, feel free to send questions, comments and critics... We will try to
> reply them all.
>
> Regards,
> Marcelo
>
>
RE: If we Open Source our platform, would it be interesting to you?
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
I only looked at your link super fast, but this seems like a very viable alternative to Solr's DIH. DIH does the job fairly well but we've struggled to have developers who are willing to maintain it. The problem, I think, is that DIH appeals to non-programmers who want to index their data without writing (much) code, hence the user base is unable to help out.
I saw that you can take data from NoSql databases, from a rdbms and emails. Can it handle flat tiles, integrate with tika to parse out different file formats, etc? Can it join and denormalize data from a variety of sources and combine that into 1 flat-schema document?
I could see your project as something we could use in Solr & Lucene, but with its appeal to more projects than just ours, possibly would not have such a problem attracting developers.
James Dyer
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Marcelo Elias Del Valle [mailto:marcelo@s1mbi0se.com.br]
Sent: Wednesday, February 20, 2013 12:18 PM
To: solr-user@lucene.apache.org
Subject: If we Open Source our platform, would it be interesting to you?
Hello All,
I'm sending this email because I think it may be interesting for Solr
users, as this project have a strong usage of Solr platform.
We are strongly considering opening the source of our DMP (Data Management
Platform), if it proves to be technically interesting to other developers /
companies.
More details: http://www.s1mbi0se.com/s1mbi0se_DMP.html
All comments, questions and critics happening at HN:
http://news.ycombinator.com/item?id=5251780
Please, feel free to send questions, comments and critics... We will try to
reply them all.
Regards,
Marcelo
Re: If we Open Source our platform, would it be interesting to you?
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
Jack,
I didn't know this concept, but indeed, it seems like a good way to
define our application, and also opens possibilities of improvement.
Thanks for the link!
Best regards,
Marcelo.
2013/2/21 Jack Park <ja...@topicquests.org>
> Marcelo
>
> In some sense, it sounds like you are aiming at building a topic map
> of all your resources.
>
> Jack
>
> On Thu, Feb 21, 2013 at 11:54 AM, Marcelo Elias Del Valle
> <ma...@s1mbi0se.com.br> wrote:
> > Hello David,
> >
> > First of all, thanks for answering!
> >
> > 2013/2/21 David Quarterman <da...@corexe.com>
> >
> >> Looked through your site and the framework looks very powerful as an
> >> aggregator. We do a lot of data aggregation from many different sources
> in
> >> many different formats (XML, JSON, text, CSV, etc) using RDBMS as the
> main
> >> repository for eventual SOLR indexing. A 'one-stop-shop' for all this
> would
> >> be very appealing.
> >>
> >
> > Actually, just to clarify, it uses Cassandra as repository, not an
> > RDMS. We want to use it for large scale, so you could import entire
> company
> > databases into the repo and relate the data from one another. However,
> If I
> > understood you right, you got the idea, an intermediate repo before
> > indexing, so you could postpone decisions about what to index and how...
> >
> >
> >> Have you looked at products like Talend & Jitterbit? These offer
> >> transformation from almost anything to almost anything using graphical
> >> interfaces (Jitterbit is better) and a PHP-like coding format for
> trickier
> >> work. If you (or somebody) could add a graphical interface, the world
> would
> >> beat a path to your door!
> >
> >
> > This is very interesting, actually! We considered using Talend when
> we
> > started our business, but we decided to go ahead with the development of
> a
> > new product. The reason was: Talend is great, but it limits a good
> > programmer, if he is more agile coding than using graphical interfaces.
> > Have user interfaces as a possibility is nice, but as something you HAVE
> TO
> > use is awful. Besides, it has a learning curve and seems to run better
> and
> > you hire their own platform, and we wanted to choose the fine grain of
> our
> > platform.
> > However, your question made me think a lot about it. Do you think
> > integrating to jitterbit or talend could be interesting? Or did you mean
> > developing a new user interface? The bad thing I see in integrating with
> a
> > talend like program is that you start to be dependent on the graphical
> > interface, I feel it's hard to use my own java code... I might be wrong.
> > Anyway, I will consider this possibility, but if you could explain
> > better why you think one or other could be such a good idea would help
> us a
> > lot. Would you be interested in using such a tool yourself?
> >
> > Best regards,
> > Marcelo.
>
Re: If we Open Source our platform, would it be interesting to you?
Posted by Jack Park <ja...@topicquests.org>.
Marcelo
In some sense, it sounds like you are aiming at building a topic map
of all your resources.
Jack
On Thu, Feb 21, 2013 at 11:54 AM, Marcelo Elias Del Valle
<ma...@s1mbi0se.com.br> wrote:
> Hello David,
>
> First of all, thanks for answering!
>
> 2013/2/21 David Quarterman <da...@corexe.com>
>
>> Looked through your site and the framework looks very powerful as an
>> aggregator. We do a lot of data aggregation from many different sources in
>> many different formats (XML, JSON, text, CSV, etc) using RDBMS as the main
>> repository for eventual SOLR indexing. A 'one-stop-shop' for all this would
>> be very appealing.
>>
>
> Actually, just to clarify, it uses Cassandra as repository, not an
> RDMS. We want to use it for large scale, so you could import entire company
> databases into the repo and relate the data from one another. However, If I
> understood you right, you got the idea, an intermediate repo before
> indexing, so you could postpone decisions about what to index and how...
>
>
>> Have you looked at products like Talend & Jitterbit? These offer
>> transformation from almost anything to almost anything using graphical
>> interfaces (Jitterbit is better) and a PHP-like coding format for trickier
>> work. If you (or somebody) could add a graphical interface, the world would
>> beat a path to your door!
>
>
> This is very interesting, actually! We considered using Talend when we
> started our business, but we decided to go ahead with the development of a
> new product. The reason was: Talend is great, but it limits a good
> programmer, if he is more agile coding than using graphical interfaces.
> Have user interfaces as a possibility is nice, but as something you HAVE TO
> use is awful. Besides, it has a learning curve and seems to run better and
> you hire their own platform, and we wanted to choose the fine grain of our
> platform.
> However, your question made me think a lot about it. Do you think
> integrating to jitterbit or talend could be interesting? Or did you mean
> developing a new user interface? The bad thing I see in integrating with a
> talend like program is that you start to be dependent on the graphical
> interface, I feel it's hard to use my own java code... I might be wrong.
> Anyway, I will consider this possibility, but if you could explain
> better why you think one or other could be such a good idea would help us a
> lot. Would you be interested in using such a tool yourself?
>
> Best regards,
> Marcelo.
Re: If we Open Source our platform, would it be interesting to you?
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
Hello David,
First of all, thanks for answering!
2013/2/21 David Quarterman <da...@corexe.com>
> Looked through your site and the framework looks very powerful as an
> aggregator. We do a lot of data aggregation from many different sources in
> many different formats (XML, JSON, text, CSV, etc) using RDBMS as the main
> repository for eventual SOLR indexing. A 'one-stop-shop' for all this would
> be very appealing.
>
Actually, just to clarify, it uses Cassandra as repository, not an
RDMS. We want to use it for large scale, so you could import entire company
databases into the repo and relate the data from one another. However, If I
understood you right, you got the idea, an intermediate repo before
indexing, so you could postpone decisions about what to index and how...
> Have you looked at products like Talend & Jitterbit? These offer
> transformation from almost anything to almost anything using graphical
> interfaces (Jitterbit is better) and a PHP-like coding format for trickier
> work. If you (or somebody) could add a graphical interface, the world would
> beat a path to your door!
This is very interesting, actually! We considered using Talend when we
started our business, but we decided to go ahead with the development of a
new product. The reason was: Talend is great, but it limits a good
programmer, if he is more agile coding than using graphical interfaces.
Have user interfaces as a possibility is nice, but as something you HAVE TO
use is awful. Besides, it has a learning curve and seems to run better and
you hire their own platform, and we wanted to choose the fine grain of our
platform.
However, your question made me think a lot about it. Do you think
integrating to jitterbit or talend could be interesting? Or did you mean
developing a new user interface? The bad thing I see in integrating with a
talend like program is that you start to be dependent on the graphical
interface, I feel it's hard to use my own java code... I might be wrong.
Anyway, I will consider this possibility, but if you could explain
better why you think one or other could be such a good idea would help us a
lot. Would you be interested in using such a tool yourself?
Best regards,
Marcelo.
RE: If we Open Source our platform, would it be interesting to you?
Posted by David Quarterman <da...@corexe.com>.
Hi Marcelo,
Looked through your site and the framework looks very powerful as an aggregator. We do a lot of data aggregation from many different sources in many different formats (XML, JSON, text, CSV, etc) using RDBMS as the main repository for eventual SOLR indexing. A 'one-stop-shop' for all this would be very appealing.
Have you looked at products like Talend & Jitterbit? These offer transformation from almost anything to almost anything using graphical interfaces (Jitterbit is better) and a PHP-like coding format for trickier work. If you (or somebody) could add a graphical interface, the world would beat a path to your door!
Regards,
DQ
-----Original Message-----
From: Marcelo Elias Del Valle [mailto:marcelo@s1mbi0se.com.br]
Sent: 20 February 2013 18:18
To: solr-user@lucene.apache.org
Subject: If we Open Source our platform, would it be interesting to you?
Hello All,
I’m sending this email because I think it may be interesting for Solr users, as this project have a strong usage of Solr platform.
We are strongly considering opening the source of our DMP (Data Management Platform), if it proves to be technically interesting to other developers / companies.
More details: http://www.s1mbi0se.com/s1mbi0se_DMP.html
All comments, questions and critics happening at HN:
http://news.ycombinator.com/item?id=5251780
Please, feel free to send questions, comments and critics... We will try to reply them all.
Regards,
Marcelo