You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@zeppelin.apache.org by madhuka udantha <ma...@gmail.com> on 2015/03/20 06:27:30 UTC

[GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Hi All,

I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
project, 0COMDEV-1190 captures my interest.

I have abundant experiences of visualization techniques creating numerous
dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.

My current research area comprises of big data where I have worked with
various types of data sets. Also I'm working with cluster representation
and classification techniques where visualization amounts to a considerable
part. I was following COMDEV-119 (jira) with Alexander Bezzubov and CORNEAU
Damien for more than week.

Thanks

[1] http://wso2.com/products/user-engagement-server/
[2] https://github.com/wso2/jaggery
-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi, Alex

Thanks for the Zeppelin Community and GSoC Management for their interest on
my GSoC Proposal Zeppelin COMDEV-119. Also thank you Alexander for your
response.

I hope it will get selected and I'm looking forward to working with you
all. Even though I have few experience on working with the Microsoft
Community the only major exposure to OpenSource will be at WSO2 as senior
software engineer. In WSO2 my work  was mostly for front end[1], APIS[3,
4], Javascript/jaggeryjs [5], dashboard[2], etc. Currently I am doing
Master in Computer Science Engineer (full time RA) in area of datamining
and CEP

Therefore in order to extend my experience with the Opensource Community
I'm eagerly looking forward to contribute for this project and for a fun
filled and educational time together ahead.

Thanks. And I humbly expect you all will assist me in this journey
together.

[1] https://www.youtube.com/watch?v=aWikaNZPvVU
[2] http://wso2.com/products/user-engagement-server/
[3] http://wso2.com/products/enterprise-store/
[4] http://wso2.com/api-management/
[5] http://jaggeryjs.org/


On Tue, Mar 31, 2015 at 8:58 AM, Alex B. <ab...@nflabs.com> wrote:

> Thank you very much for all the effort that you put in this project!
>
> We and GSoC management we'll keep you posted on the status of the proposal.
>
> On Mon, Mar 30, 2015 at 2:45 AM, madhuka udantha <madhukaudantha@gmail.com
> > wrote:
>
>> Hi,
>>
>> I have submitted the GSoC Proposal 2015 under the title of "Extending
>> visualization of Zeppelin with Rich GUI and Charting Manager".
>>
>> My gratitude goes to the active Community of Zeppelin,
>> specially Alexander, Damien, Eran and Moon Soo Lee for your valuable ideas
>> and feedback.
>>
>> I hope to write few blog posts on the current tasks I'm doing with regard
>> to this project and share with you all.
>>
>> On Fri, Mar 27, 2015 at 12:20 PM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi Alex,
>>>
>>> This is the email thread which I used to collaborate with the Zeppelin
>>> Community[1] .
>>> Currently I'm writing the Proposal upon the points discussed here and on
>>> JIRA. If I have missed anything regard to the task please feel free to
>>> share it.
>>> I'll share my proposal when I finish writing it.
>>>
>>> Thanks.
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/COMDEV-119
>>>
>>> On Wed, Mar 25, 2015 at 11:09 PM, madhuka udantha <
>>> madhukaudantha@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> According to the discussion to get a clear understanding I just drew 2
>>>> sequence diagrams that  will explain
>>>> how chart will react to changing the pivot.
>>>>
>>>> Safe Level
>>>>
>>>> https://issues.apache.org/jira/secure/attachment/12707251/Changing%20the%20pivot%20-%20Safe%20Level.png
>>>>
>>>>
>>>> In safe level (default level) only limited amount of data is
>>>> retrieved(sufficient to draw the chart).
>>>> At initial stage local storage don't contain data. But when you make a
>>>> pivot change data will be there to draw the graph. If data is out-dated we
>>>> will get it from back-end.
>>>>
>>>> Restricted Level
>>>>
>>>> https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png
>>>> <https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png>
>>>>
>>>> User will reach Restricted Level after he successfully pass the Safe
>>>> Level. Then in local storage we will have up-to-date data. But for this
>>>> level it will be using all the data in the database. So Charting will grab
>>>> the data from storage and back-end.
>>>>
>>>> Your ideas are mostly welcomed.
>>>>
>>>>
>>>> On Wed, Mar 25, 2015 at 2:55 PM, madhuka udantha <
>>>> madhukaudantha@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want to know about the code structure and zeppelin architecture? Is
>>>>> there any good post / article / wiki regarding the said.
>>>>> Also if there is any quick start guide regarding development of
>>>>> Zeppelin please share it with me.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Mon, Mar 23, 2015 at 10:44 AM, madhuka udantha <
>>>>> madhukaudantha@gmail.com> wrote:
>>>>>
>>>>>> Hi, moon
>>>>>>
>>>>>> Yes, Since
>>>>>>
>>>>>>> "Moving computation is cheaper than moving data"
>>>>>>
>>>>>> We can do computation in computing framework.
>>>>>>
>>>>>> For simple pivot changing or filtering can be handle in local storage
>>>>>> with indexing databases depending on the current user level.
>>>>>> As you saw, computations will be handle in the back ends.
>>>>>>
>>>>>> Great to hear about the building rich GUI, I will give me chart
>>>>>> library ideas on there.
>>>>>>
>>>>>> Your ideas are always welcome, those will be helpful for my task and
>>>>>> draft proposal
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, madhuka udantha
>>>>>>>
>>>>>>> I think your idea about chart library and data transformation engine
>>>>>>> sounds
>>>>>>> cool. For the data transform modules, it's good idea to make this
>>>>>>> pluggable
>>>>>>> to data transform engine. But i'm not sure getting result locally
>>>>>>> and do
>>>>>>> transform for pivot or filtering to prevent run query again is good
>>>>>>> idea.
>>>>>>> Because of Zeppelin is (not limited but) trying to build analytical
>>>>>>> environment on top of distributed computing framework, like Spark,
>>>>>>> Flink,
>>>>>>> Ignite, etc. Most of distributed computing framework Zeppelin trying
>>>>>>> to
>>>>>>> integrate is following the same paradigm "Moving computation is
>>>>>>> cheaper
>>>>>>> than moving data". In this manner, size of data that transform
>>>>>>> engine need
>>>>>>> to handle can be easily multiple TB. Which will take long time to
>>>>>>> copy to
>>>>>>> local machine and process. So i think transform module should be run
>>>>>>> on
>>>>>>> underlying distributed computing framework.
>>>>>>>
>>>>>>> And about Chart library, we have started discussion thread about
>>>>>>> building
>>>>>>> rich GUI inside of notebook. it might be related.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> moon
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <
>>>>>>> madhukaudantha@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <
>>>>>>> corneadoug@apache.org>
>>>>>>> > wrote:
>>>>>>> >
>>>>>>> > > Hi,
>>>>>>> > >
>>>>>>> > > Being able to aggregate on the query side is a great idea and
>>>>>>> would allow
>>>>>>> > > us to transfer less data as well as having a full query
>>>>>>> representation of
>>>>>>> > > the visualization.
>>>>>>> > >
>>>>>>> > > However creating a SQL query dynamically is a pretty difficult
>>>>>>> task, and
>>>>>>> > > might be too much for that scope.
>>>>>>> > >
>>>>>>> > > Also I see some possible problems with this method:
>>>>>>> > >  - Changing the pivot or simple filtering would mean running the
>>>>>>> query
>>>>>>> > > again
>>>>>>> > >
>>>>>>> > No, the query wont run again.
>>>>>>> > In the first run of the query data is collected and stored
>>>>>>> locally- local
>>>>>>> > storage [1](using indexing techniques to make retrieval faster) So
>>>>>>> changing
>>>>>>> > pivot or simple filtering will use the local storage.
>>>>>>> > If any attribute or data is missing in local storage then it will
>>>>>>> retrieve
>>>>>>> > only that and save the network bandwidth as well.
>>>>>>> > Does my explanation make sense.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > >  - Being able to make pivot style SQL query would be really hard,
>>>>>>> > >    we would need multiple sub-queries or even some times
>>>>>>> multiple queries
>>>>>>> > > (I tried a few times and could have the result wanted only with
>>>>>>> > > visualization side pivot).
>>>>>>> > >    It would end up with really bad SQL queries, especially with
>>>>>>> the Hive
>>>>>>> > > SQL or Spark SQL limitations and would take way more time to
>>>>>>> process.
>>>>>>> > >
>>>>>>> > Agreed. I'm not planing to use pivot style queries.
>>>>>>> >
>>>>>>> > Any suggestions?
>>>>>>> >
>>>>>>> >
>>>>>>> > Thanks.
>>>>>>> >
>>>>>>> >
>>>>>>> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > > Hi,
>>>>>>> > > >
>>>>>>> > > > The Chart library features sounds promising.
>>>>>>> > > > As  for the data engine - one thing that I think is missing is
>>>>>>> the
>>>>>>> > > ability
>>>>>>> > > > to use the visualization to drive the aggregation in the SQL.
>>>>>>> today,
>>>>>>> > you
>>>>>>> > > > first write the SQL, you execute it, *limited by the number of
>>>>>>> results
>>>>>>> > > sent
>>>>>>> > > > to the client*, and then you use viz to understand the results.
>>>>>>> > > > Alternatively, if through the visualization I can generate a
>>>>>>> better SQL
>>>>>>> > > > which returns returns an aggregated data-set then I can
>>>>>>> analyze a
>>>>>>> > bigger
>>>>>>> > > > amount of data.
>>>>>>> > > >
>>>>>>> > > > I hope I was clear enough in my explanation :-)
>>>>>>> > > >
>>>>>>> > > > Eran
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
>>>>>>> > > madhukaudantha@gmail.com
>>>>>>> > > > >
>>>>>>> > > > wrote:
>>>>>>> > > >
>>>>>>> > > > > Hi,
>>>>>>> > > > >
>>>>>>> > > > > Here is my proposing ideas.
>>>>>>> > > > > According to COMDEV-119 jira. Charts are hard coded until
>>>>>>> now and
>>>>>>> > data
>>>>>>> > > > > transformation issue was highlighted since different charts
>>>>>>> have
>>>>>>> > > > different
>>>>>>> > > > > pivot fields eg: Area charts, Scatter, Surface charts,
>>>>>>> Bubble charts,
>>>>>>> > > > Radar
>>>>>>> > > > > charts. etc..
>>>>>>> > > > >
>>>>>>> > > > > To solve this I am introducing a two major component one is
>>>>>>> called
>>>>>>> > > 'Chart
>>>>>>> > > > > library' and 'Data transformation engine'. Chart library is
>>>>>>> located
>>>>>>> > > where
>>>>>>> > > > > it shows the chats that are currently plugged. There we can
>>>>>>> plug
>>>>>>> > chart
>>>>>>> > > > > types and those can be reused.
>>>>>>> > > > >
>>>>>>> > > > > *Chart library features *
>>>>>>> > > > >
>>>>>>> > > > >    - Users can select the chart from library
>>>>>>> > > > >    - Those charts are pluggable to library
>>>>>>> > > > >    - Charts can be plugged by config(json)/UI with wizard
>>>>>>> > > > >    - Configuration/Meta file of the chart contains
>>>>>>> interface, libs,
>>>>>>> > > > themes
>>>>>>> > > > >    and a data transformation types/mappings
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > > *Data Transformation Engine*
>>>>>>> > > > > 'Data transformation engine' contains data transformation
>>>>>>> modules.
>>>>>>> > > Those
>>>>>>> > > > > modules are also pluggable to engine. Those have connections
>>>>>>> to
>>>>>>> > charts.
>>>>>>> > > > > Data transformation engine sit between the data (sql) and
>>>>>>> chart. So
>>>>>>> > > this
>>>>>>> > > > > module  converts data and map them to each chart pivot field
>>>>>>> > > > >
>>>>>>> > > > >    - This module will look at pivot fields of the chart
>>>>>>> > > > >    - Selected attributes of the SQL query
>>>>>>> > > > >    - Attribute value operations improvement (string split,
>>>>>>> value
>>>>>>> > > > >    aggregation, round number round)
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > > Another improvement that I notice is that
>>>>>>> > > > >
>>>>>>> > > > >    - Query Edit auto-completion support (with Ctrl+space)
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > > Your ideas are welcome here
>>>>>>> > > > > Thanks
>>>>>>> > > > >
>>>>>>> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>>>>> > > > > madhukaudantha@gmail.com>
>>>>>>> > > > > wrote:
>>>>>>> > > > >
>>>>>>> > > > > > Hi All,
>>>>>>> > > > > >
>>>>>>> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This
>>>>>>> GSoC 2015
>>>>>>> > > > > > project, 0COMDEV-1190 captures my interest.
>>>>>>> > > > > >
>>>>>>> > > > > > I have abundant experiences of visualization techniques
>>>>>>> creating
>>>>>>> > > > numerous
>>>>>>> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3
>>>>>>> charting etc.
>>>>>>> > > > > >
>>>>>>> > > > > > My current research area comprises of big data where I
>>>>>>> have worked
>>>>>>> > > with
>>>>>>> > > > > > various types of data sets. Also I'm working with cluster
>>>>>>> > > > representation
>>>>>>> > > > > > and classification techniques where visualization amounts
>>>>>>> to a
>>>>>>> > > > > considerable
>>>>>>> > > > > > part. I was following COMDEV-119 (jira) with Alexander
>>>>>>> Bezzubov and
>>>>>>> > > > > CORNEAU
>>>>>>> > > > > > Damien for more than week.
>>>>>>> > > > > >
>>>>>>> > > > > > Thanks
>>>>>>> > > > > >
>>>>>>> > > > > > [1] http://wso2.com/products/user-engagement-server/
>>>>>>> > > > > > [2] https://github.com/wso2/jaggery
>>>>>>> > > > > > --
>>>>>>> > > > > > Cheers,
>>>>>>> > > > > > Madhuka Udantha
>>>>>>> > > > > > http://madhukaudantha.blogspot.com
>>>>>>> > > > > >
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > > --
>>>>>>> > > > > Cheers,
>>>>>>> > > > > Madhuka Udantha
>>>>>>> > > > > http://madhukaudantha.blogspot.com
>>>>>>> > > > >
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > --
>>>>>>> > > > Eran | CTO
>>>>>>> > > >
>>>>>>> > >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Cheers,
>>>>>>> > Madhuka Udantha
>>>>>>> > http://madhukaudantha.blogspot.com
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Madhuka Udantha
>>>>>> http://madhukaudantha.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Madhuka Udantha
>>>>> http://madhukaudantha.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Madhuka Udantha
>>>> http://madhukaudantha.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>


-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by "Alex B." <ab...@nflabs.com>.

Thank you very much for all the effort that you put in this project!

We and GSoC management we'll keep you posted on the status of the proposal.

On Mon, Mar 30, 2015 at 2:45 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi,
>
> I have submitted the GSoC Proposal 2015 under the title of "Extending
> visualization of Zeppelin with Rich GUI and Charting Manager".
>
> My gratitude goes to the active Community of Zeppelin,
> specially Alexander, Damien, Eran and Moon Soo Lee for your valuable ideas
> and feedback.
>
> I hope to write few blog posts on the current tasks I'm doing with regard
> to this project and share with you all.
>
> On Fri, Mar 27, 2015 at 12:20 PM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi Alex,
>>
>> This is the email thread which I used to collaborate with the Zeppelin
>> Community[1] .
>> Currently I'm writing the Proposal upon the points discussed here and on
>> JIRA. If I have missed anything regard to the task please feel free to
>> share it.
>> I'll share my proposal when I finish writing it.
>>
>> Thanks.
>>
>>
>> [1] https://issues.apache.org/jira/browse/COMDEV-119
>>
>> On Wed, Mar 25, 2015 at 11:09 PM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> According to the discussion to get a clear understanding I just drew 2
>>> sequence diagrams that  will explain
>>> how chart will react to changing the pivot.
>>>
>>> Safe Level
>>>
>>> https://issues.apache.org/jira/secure/attachment/12707251/Changing%20the%20pivot%20-%20Safe%20Level.png
>>>
>>>
>>> In safe level (default level) only limited amount of data is
>>> retrieved(sufficient to draw the chart).
>>> At initial stage local storage don't contain data. But when you make a
>>> pivot change data will be there to draw the graph. If data is out-dated we
>>> will get it from back-end.
>>>
>>> Restricted Level
>>>
>>> https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png
>>> <https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png>
>>>
>>> User will reach Restricted Level after he successfully pass the Safe
>>> Level. Then in local storage we will have up-to-date data. But for this
>>> level it will be using all the data in the database. So Charting will grab
>>> the data from storage and back-end.
>>>
>>> Your ideas are mostly welcomed.
>>>
>>>
>>> On Wed, Mar 25, 2015 at 2:55 PM, madhuka udantha <
>>> madhukaudantha@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I want to know about the code structure and zeppelin architecture? Is
>>>> there any good post / article / wiki regarding the said.
>>>> Also if there is any quick start guide regarding development of
>>>> Zeppelin please share it with me.
>>>>
>>>> Thanks.
>>>>
>>>> On Mon, Mar 23, 2015 at 10:44 AM, madhuka udantha <
>>>> madhukaudantha@gmail.com> wrote:
>>>>
>>>>> Hi, moon
>>>>>
>>>>> Yes, Since
>>>>>
>>>>>> "Moving computation is cheaper than moving data"
>>>>>
>>>>> We can do computation in computing framework.
>>>>>
>>>>> For simple pivot changing or filtering can be handle in local storage
>>>>> with indexing databases depending on the current user level.
>>>>> As you saw, computations will be handle in the back ends.
>>>>>
>>>>> Great to hear about the building rich GUI, I will give me chart
>>>>> library ideas on there.
>>>>>
>>>>> Your ideas are always welcome, those will be helpful for my task and
>>>>> draft proposal
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org> wrote:
>>>>>
>>>>>> Hi, madhuka udantha
>>>>>>
>>>>>> I think your idea about chart library and data transformation engine
>>>>>> sounds
>>>>>> cool. For the data transform modules, it's good idea to make this
>>>>>> pluggable
>>>>>> to data transform engine. But i'm not sure getting result locally and
>>>>>> do
>>>>>> transform for pivot or filtering to prevent run query again is good
>>>>>> idea.
>>>>>> Because of Zeppelin is (not limited but) trying to build analytical
>>>>>> environment on top of distributed computing framework, like Spark,
>>>>>> Flink,
>>>>>> Ignite, etc. Most of distributed computing framework Zeppelin trying
>>>>>> to
>>>>>> integrate is following the same paradigm "Moving computation is
>>>>>> cheaper
>>>>>> than moving data". In this manner, size of data that transform engine
>>>>>> need
>>>>>> to handle can be easily multiple TB. Which will take long time to
>>>>>> copy to
>>>>>> local machine and process. So i think transform module should be run
>>>>>> on
>>>>>> underlying distributed computing framework.
>>>>>>
>>>>>> And about Chart library, we have started discussion thread about
>>>>>> building
>>>>>> rich GUI inside of notebook. it might be related.
>>>>>>
>>>>>> Thanks,
>>>>>> moon
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <
>>>>>> madhukaudantha@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <
>>>>>> corneadoug@apache.org>
>>>>>> > wrote:
>>>>>> >
>>>>>> > > Hi,
>>>>>> > >
>>>>>> > > Being able to aggregate on the query side is a great idea and
>>>>>> would allow
>>>>>> > > us to transfer less data as well as having a full query
>>>>>> representation of
>>>>>> > > the visualization.
>>>>>> > >
>>>>>> > > However creating a SQL query dynamically is a pretty difficult
>>>>>> task, and
>>>>>> > > might be too much for that scope.
>>>>>> > >
>>>>>> > > Also I see some possible problems with this method:
>>>>>> > >  - Changing the pivot or simple filtering would mean running the
>>>>>> query
>>>>>> > > again
>>>>>> > >
>>>>>> > No, the query wont run again.
>>>>>> > In the first run of the query data is collected and stored locally-
>>>>>> local
>>>>>> > storage [1](using indexing techniques to make retrieval faster) So
>>>>>> changing
>>>>>> > pivot or simple filtering will use the local storage.
>>>>>> > If any attribute or data is missing in local storage then it will
>>>>>> retrieve
>>>>>> > only that and save the network bandwidth as well.
>>>>>> > Does my explanation make sense.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > >  - Being able to make pivot style SQL query would be really hard,
>>>>>> > >    we would need multiple sub-queries or even some times multiple
>>>>>> queries
>>>>>> > > (I tried a few times and could have the result wanted only with
>>>>>> > > visualization side pivot).
>>>>>> > >    It would end up with really bad SQL queries, especially with
>>>>>> the Hive
>>>>>> > > SQL or Spark SQL limitations and would take way more time to
>>>>>> process.
>>>>>> > >
>>>>>> > Agreed. I'm not planing to use pivot style queries.
>>>>>> >
>>>>>> > Any suggestions?
>>>>>> >
>>>>>> >
>>>>>> > Thanks.
>>>>>> >
>>>>>> >
>>>>>> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com>
>>>>>> wrote:
>>>>>> > >
>>>>>> > > > Hi,
>>>>>> > > >
>>>>>> > > > The Chart library features sounds promising.
>>>>>> > > > As  for the data engine - one thing that I think is missing is
>>>>>> the
>>>>>> > > ability
>>>>>> > > > to use the visualization to drive the aggregation in the SQL.
>>>>>> today,
>>>>>> > you
>>>>>> > > > first write the SQL, you execute it, *limited by the number of
>>>>>> results
>>>>>> > > sent
>>>>>> > > > to the client*, and then you use viz to understand the results.
>>>>>> > > > Alternatively, if through the visualization I can generate a
>>>>>> better SQL
>>>>>> > > > which returns returns an aggregated data-set then I can analyze
>>>>>> a
>>>>>> > bigger
>>>>>> > > > amount of data.
>>>>>> > > >
>>>>>> > > > I hope I was clear enough in my explanation :-)
>>>>>> > > >
>>>>>> > > > Eran
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
>>>>>> > > madhukaudantha@gmail.com
>>>>>> > > > >
>>>>>> > > > wrote:
>>>>>> > > >
>>>>>> > > > > Hi,
>>>>>> > > > >
>>>>>> > > > > Here is my proposing ideas.
>>>>>> > > > > According to COMDEV-119 jira. Charts are hard coded until now
>>>>>> and
>>>>>> > data
>>>>>> > > > > transformation issue was highlighted since different charts
>>>>>> have
>>>>>> > > > different
>>>>>> > > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble
>>>>>> charts,
>>>>>> > > > Radar
>>>>>> > > > > charts. etc..
>>>>>> > > > >
>>>>>> > > > > To solve this I am introducing a two major component one is
>>>>>> called
>>>>>> > > 'Chart
>>>>>> > > > > library' and 'Data transformation engine'. Chart library is
>>>>>> located
>>>>>> > > where
>>>>>> > > > > it shows the chats that are currently plugged. There we can
>>>>>> plug
>>>>>> > chart
>>>>>> > > > > types and those can be reused.
>>>>>> > > > >
>>>>>> > > > > *Chart library features *
>>>>>> > > > >
>>>>>> > > > >    - Users can select the chart from library
>>>>>> > > > >    - Those charts are pluggable to library
>>>>>> > > > >    - Charts can be plugged by config(json)/UI with wizard
>>>>>> > > > >    - Configuration/Meta file of the chart contains interface,
>>>>>> libs,
>>>>>> > > > themes
>>>>>> > > > >    and a data transformation types/mappings
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > *Data Transformation Engine*
>>>>>> > > > > 'Data transformation engine' contains data transformation
>>>>>> modules.
>>>>>> > > Those
>>>>>> > > > > modules are also pluggable to engine. Those have connections
>>>>>> to
>>>>>> > charts.
>>>>>> > > > > Data transformation engine sit between the data (sql) and
>>>>>> chart. So
>>>>>> > > this
>>>>>> > > > > module  converts data and map them to each chart pivot field
>>>>>> > > > >
>>>>>> > > > >    - This module will look at pivot fields of the chart
>>>>>> > > > >    - Selected attributes of the SQL query
>>>>>> > > > >    - Attribute value operations improvement (string split,
>>>>>> value
>>>>>> > > > >    aggregation, round number round)
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > Another improvement that I notice is that
>>>>>> > > > >
>>>>>> > > > >    - Query Edit auto-completion support (with Ctrl+space)
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > Your ideas are welcome here
>>>>>> > > > > Thanks
>>>>>> > > > >
>>>>>> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>>>> > > > > madhukaudantha@gmail.com>
>>>>>> > > > > wrote:
>>>>>> > > > >
>>>>>> > > > > > Hi All,
>>>>>> > > > > >
>>>>>> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This
>>>>>> GSoC 2015
>>>>>> > > > > > project, 0COMDEV-1190 captures my interest.
>>>>>> > > > > >
>>>>>> > > > > > I have abundant experiences of visualization techniques
>>>>>> creating
>>>>>> > > > numerous
>>>>>> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3
>>>>>> charting etc.
>>>>>> > > > > >
>>>>>> > > > > > My current research area comprises of big data where I have
>>>>>> worked
>>>>>> > > with
>>>>>> > > > > > various types of data sets. Also I'm working with cluster
>>>>>> > > > representation
>>>>>> > > > > > and classification techniques where visualization amounts
>>>>>> to a
>>>>>> > > > > considerable
>>>>>> > > > > > part. I was following COMDEV-119 (jira) with Alexander
>>>>>> Bezzubov and
>>>>>> > > > > CORNEAU
>>>>>> > > > > > Damien for more than week.
>>>>>> > > > > >
>>>>>> > > > > > Thanks
>>>>>> > > > > >
>>>>>> > > > > > [1] http://wso2.com/products/user-engagement-server/
>>>>>> > > > > > [2] https://github.com/wso2/jaggery
>>>>>> > > > > > --
>>>>>> > > > > > Cheers,
>>>>>> > > > > > Madhuka Udantha
>>>>>> > > > > > http://madhukaudantha.blogspot.com
>>>>>> > > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > --
>>>>>> > > > > Cheers,
>>>>>> > > > > Madhuka Udantha
>>>>>> > > > > http://madhukaudantha.blogspot.com
>>>>>> > > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > --
>>>>>> > > > Eran | CTO
>>>>>> > > >
>>>>>> > >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Cheers,
>>>>>> > Madhuka Udantha
>>>>>> > http://madhukaudantha.blogspot.com
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Madhuka Udantha
>>>>> http://madhukaudantha.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Madhuka Udantha
>>>> http://madhukaudantha.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
--
Kind regards,
Alexander.

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

I have submitted the GSoC Proposal 2015 under the title of "Extending
visualization of Zeppelin with Rich GUI and Charting Manager".

My gratitude goes to the active Community of Zeppelin, specially Alexander,
Damien, Eran and Moon Soo Lee for your valuable ideas and feedback.

I hope to write few blog posts on the current tasks I'm doing with regard
to this project and share with you all.

On Fri, Mar 27, 2015 at 12:20 PM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi Alex,
>
> This is the email thread which I used to collaborate with the Zeppelin
> Community[1] .
> Currently I'm writing the Proposal upon the points discussed here and on
> JIRA. If I have missed anything regard to the task please feel free to
> share it.
> I'll share my proposal when I finish writing it.
>
> Thanks.
>
>
> [1] https://issues.apache.org/jira/browse/COMDEV-119
>
> On Wed, Mar 25, 2015 at 11:09 PM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi,
>>
>> According to the discussion to get a clear understanding I just drew 2
>> sequence diagrams that  will explain
>> how chart will react to changing the pivot.
>>
>> Safe Level
>>
>> https://issues.apache.org/jira/secure/attachment/12707251/Changing%20the%20pivot%20-%20Safe%20Level.png
>>
>>
>> In safe level (default level) only limited amount of data is
>> retrieved(sufficient to draw the chart).
>> At initial stage local storage don't contain data. But when you make a
>> pivot change data will be there to draw the graph. If data is out-dated we
>> will get it from back-end.
>>
>> Restricted Level
>>
>> https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png
>> <https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png>
>>
>> User will reach Restricted Level after he successfully pass the Safe
>> Level. Then in local storage we will have up-to-date data. But for this
>> level it will be using all the data in the database. So Charting will grab
>> the data from storage and back-end.
>>
>> Your ideas are mostly welcomed.
>>
>>
>> On Wed, Mar 25, 2015 at 2:55 PM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I want to know about the code structure and zeppelin architecture? Is
>>> there any good post / article / wiki regarding the said.
>>> Also if there is any quick start guide regarding development of Zeppelin
>>> please share it with me.
>>>
>>> Thanks.
>>>
>>> On Mon, Mar 23, 2015 at 10:44 AM, madhuka udantha <
>>> madhukaudantha@gmail.com> wrote:
>>>
>>>> Hi, moon
>>>>
>>>> Yes, Since
>>>>
>>>>> "Moving computation is cheaper than moving data"
>>>>
>>>> We can do computation in computing framework.
>>>>
>>>> For simple pivot changing or filtering can be handle in local storage
>>>> with indexing databases depending on the current user level.
>>>> As you saw, computations will be handle in the back ends.
>>>>
>>>> Great to hear about the building rich GUI, I will give me chart
>>>> library ideas on there.
>>>>
>>>> Your ideas are always welcome, those will be helpful for my task and
>>>> draft proposal
>>>>
>>>> Thanks
>>>>
>>>> On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org> wrote:
>>>>
>>>>> Hi, madhuka udantha
>>>>>
>>>>> I think your idea about chart library and data transformation engine
>>>>> sounds
>>>>> cool. For the data transform modules, it's good idea to make this
>>>>> pluggable
>>>>> to data transform engine. But i'm not sure getting result locally and
>>>>> do
>>>>> transform for pivot or filtering to prevent run query again is good
>>>>> idea.
>>>>> Because of Zeppelin is (not limited but) trying to build analytical
>>>>> environment on top of distributed computing framework, like Spark,
>>>>> Flink,
>>>>> Ignite, etc. Most of distributed computing framework Zeppelin trying to
>>>>> integrate is following the same paradigm "Moving computation is cheaper
>>>>> than moving data". In this manner, size of data that transform engine
>>>>> need
>>>>> to handle can be easily multiple TB. Which will take long time to copy
>>>>> to
>>>>> local machine and process. So i think transform module should be run on
>>>>> underlying distributed computing framework.
>>>>>
>>>>> And about Chart library, we have started discussion thread about
>>>>> building
>>>>> rich GUI inside of notebook. it might be related.
>>>>>
>>>>> Thanks,
>>>>> moon
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <
>>>>> madhukaudantha@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <
>>>>> corneadoug@apache.org>
>>>>> > wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > Being able to aggregate on the query side is a great idea and
>>>>> would allow
>>>>> > > us to transfer less data as well as having a full query
>>>>> representation of
>>>>> > > the visualization.
>>>>> > >
>>>>> > > However creating a SQL query dynamically is a pretty difficult
>>>>> task, and
>>>>> > > might be too much for that scope.
>>>>> > >
>>>>> > > Also I see some possible problems with this method:
>>>>> > >  - Changing the pivot or simple filtering would mean running the
>>>>> query
>>>>> > > again
>>>>> > >
>>>>> > No, the query wont run again.
>>>>> > In the first run of the query data is collected and stored locally-
>>>>> local
>>>>> > storage [1](using indexing techniques to make retrieval faster) So
>>>>> changing
>>>>> > pivot or simple filtering will use the local storage.
>>>>> > If any attribute or data is missing in local storage then it will
>>>>> retrieve
>>>>> > only that and save the network bandwidth as well.
>>>>> > Does my explanation make sense.
>>>>> >
>>>>> >
>>>>> >
>>>>> > >  - Being able to make pivot style SQL query would be really hard,
>>>>> > >    we would need multiple sub-queries or even some times multiple
>>>>> queries
>>>>> > > (I tried a few times and could have the result wanted only with
>>>>> > > visualization side pivot).
>>>>> > >    It would end up with really bad SQL queries, especially with
>>>>> the Hive
>>>>> > > SQL or Spark SQL limitations and would take way more time to
>>>>> process.
>>>>> > >
>>>>> > Agreed. I'm not planing to use pivot style queries.
>>>>> >
>>>>> > Any suggestions?
>>>>> >
>>>>> >
>>>>> > Thanks.
>>>>> >
>>>>> >
>>>>> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com>
>>>>> wrote:
>>>>> > >
>>>>> > > > Hi,
>>>>> > > >
>>>>> > > > The Chart library features sounds promising.
>>>>> > > > As  for the data engine - one thing that I think is missing is
>>>>> the
>>>>> > > ability
>>>>> > > > to use the visualization to drive the aggregation in the SQL.
>>>>> today,
>>>>> > you
>>>>> > > > first write the SQL, you execute it, *limited by the number of
>>>>> results
>>>>> > > sent
>>>>> > > > to the client*, and then you use viz to understand the results.
>>>>> > > > Alternatively, if through the visualization I can generate a
>>>>> better SQL
>>>>> > > > which returns returns an aggregated data-set then I can analyze a
>>>>> > bigger
>>>>> > > > amount of data.
>>>>> > > >
>>>>> > > > I hope I was clear enough in my explanation :-)
>>>>> > > >
>>>>> > > > Eran
>>>>> > > >
>>>>> > > >
>>>>> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
>>>>> > > madhukaudantha@gmail.com
>>>>> > > > >
>>>>> > > > wrote:
>>>>> > > >
>>>>> > > > > Hi,
>>>>> > > > >
>>>>> > > > > Here is my proposing ideas.
>>>>> > > > > According to COMDEV-119 jira. Charts are hard coded until now
>>>>> and
>>>>> > data
>>>>> > > > > transformation issue was highlighted since different charts
>>>>> have
>>>>> > > > different
>>>>> > > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble
>>>>> charts,
>>>>> > > > Radar
>>>>> > > > > charts. etc..
>>>>> > > > >
>>>>> > > > > To solve this I am introducing a two major component one is
>>>>> called
>>>>> > > 'Chart
>>>>> > > > > library' and 'Data transformation engine'. Chart library is
>>>>> located
>>>>> > > where
>>>>> > > > > it shows the chats that are currently plugged. There we can
>>>>> plug
>>>>> > chart
>>>>> > > > > types and those can be reused.
>>>>> > > > >
>>>>> > > > > *Chart library features *
>>>>> > > > >
>>>>> > > > >    - Users can select the chart from library
>>>>> > > > >    - Those charts are pluggable to library
>>>>> > > > >    - Charts can be plugged by config(json)/UI with wizard
>>>>> > > > >    - Configuration/Meta file of the chart contains interface,
>>>>> libs,
>>>>> > > > themes
>>>>> > > > >    and a data transformation types/mappings
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > *Data Transformation Engine*
>>>>> > > > > 'Data transformation engine' contains data transformation
>>>>> modules.
>>>>> > > Those
>>>>> > > > > modules are also pluggable to engine. Those have connections to
>>>>> > charts.
>>>>> > > > > Data transformation engine sit between the data (sql) and
>>>>> chart. So
>>>>> > > this
>>>>> > > > > module  converts data and map them to each chart pivot field
>>>>> > > > >
>>>>> > > > >    - This module will look at pivot fields of the chart
>>>>> > > > >    - Selected attributes of the SQL query
>>>>> > > > >    - Attribute value operations improvement (string split,
>>>>> value
>>>>> > > > >    aggregation, round number round)
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > Another improvement that I notice is that
>>>>> > > > >
>>>>> > > > >    - Query Edit auto-completion support (with Ctrl+space)
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > Your ideas are welcome here
>>>>> > > > > Thanks
>>>>> > > > >
>>>>> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>>> > > > > madhukaudantha@gmail.com>
>>>>> > > > > wrote:
>>>>> > > > >
>>>>> > > > > > Hi All,
>>>>> > > > > >
>>>>> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This
>>>>> GSoC 2015
>>>>> > > > > > project, 0COMDEV-1190 captures my interest.
>>>>> > > > > >
>>>>> > > > > > I have abundant experiences of visualization techniques
>>>>> creating
>>>>> > > > numerous
>>>>> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3
>>>>> charting etc.
>>>>> > > > > >
>>>>> > > > > > My current research area comprises of big data where I have
>>>>> worked
>>>>> > > with
>>>>> > > > > > various types of data sets. Also I'm working with cluster
>>>>> > > > representation
>>>>> > > > > > and classification techniques where visualization amounts to
>>>>> a
>>>>> > > > > considerable
>>>>> > > > > > part. I was following COMDEV-119 (jira) with Alexander
>>>>> Bezzubov and
>>>>> > > > > CORNEAU
>>>>> > > > > > Damien for more than week.
>>>>> > > > > >
>>>>> > > > > > Thanks
>>>>> > > > > >
>>>>> > > > > > [1] http://wso2.com/products/user-engagement-server/
>>>>> > > > > > [2] https://github.com/wso2/jaggery
>>>>> > > > > > --
>>>>> > > > > > Cheers,
>>>>> > > > > > Madhuka Udantha
>>>>> > > > > > http://madhukaudantha.blogspot.com
>>>>> > > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > --
>>>>> > > > > Cheers,
>>>>> > > > > Madhuka Udantha
>>>>> > > > > http://madhukaudantha.blogspot.com
>>>>> > > > >
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > > Eran | CTO
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Cheers,
>>>>> > Madhuka Udantha
>>>>> > http://madhukaudantha.blogspot.com
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Madhuka Udantha
>>>> http://madhukaudantha.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi Alex,

This is the email thread which I used to collaborate with the Zeppelin
Community[1] .
Currently I'm writing the Proposal upon the points discussed here and on
JIRA. If I have missed anything regard to the task please feel free to
share it.
I'll share my proposal when I finish writing it.

Thanks.


[1] https://issues.apache.org/jira/browse/COMDEV-119

On Wed, Mar 25, 2015 at 11:09 PM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi,
>
> According to the discussion to get a clear understanding I just drew 2
> sequence diagrams that  will explain
> how chart will react to changing the pivot.
>
> Safe Level
>
> https://issues.apache.org/jira/secure/attachment/12707251/Changing%20the%20pivot%20-%20Safe%20Level.png
>
>
> In safe level (default level) only limited amount of data is
> retrieved(sufficient to draw the chart).
> At initial stage local storage don't contain data. But when you make a
> pivot change data will be there to draw the graph. If data is out-dated we
> will get it from back-end.
>
> Restricted Level
>
> https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png
> <https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png>
>
> User will reach Restricted Level after he successfully pass the Safe
> Level. Then in local storage we will have up-to-date data. But for this
> level it will be using all the data in the database. So Charting will grab
> the data from storage and back-end.
>
> Your ideas are mostly welcomed.
>
>
> On Wed, Mar 25, 2015 at 2:55 PM, madhuka udantha <madhukaudantha@gmail.com
> > wrote:
>
>> Hi,
>>
>> I want to know about the code structure and zeppelin architecture? Is
>> there any good post / article / wiki regarding the said.
>> Also if there is any quick start guide regarding development of Zeppelin
>> please share it with me.
>>
>> Thanks.
>>
>> On Mon, Mar 23, 2015 at 10:44 AM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi, moon
>>>
>>> Yes, Since
>>>
>>>> "Moving computation is cheaper than moving data"
>>>
>>> We can do computation in computing framework.
>>>
>>> For simple pivot changing or filtering can be handle in local storage
>>> with indexing databases depending on the current user level.
>>> As you saw, computations will be handle in the back ends.
>>>
>>> Great to hear about the building rich GUI, I will give me chart library ideas
>>> on there.
>>>
>>> Your ideas are always welcome, those will be helpful for my task and
>>> draft proposal
>>>
>>> Thanks
>>>
>>> On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org> wrote:
>>>
>>>> Hi, madhuka udantha
>>>>
>>>> I think your idea about chart library and data transformation engine
>>>> sounds
>>>> cool. For the data transform modules, it's good idea to make this
>>>> pluggable
>>>> to data transform engine. But i'm not sure getting result locally and do
>>>> transform for pivot or filtering to prevent run query again is good
>>>> idea.
>>>> Because of Zeppelin is (not limited but) trying to build analytical
>>>> environment on top of distributed computing framework, like Spark,
>>>> Flink,
>>>> Ignite, etc. Most of distributed computing framework Zeppelin trying to
>>>> integrate is following the same paradigm "Moving computation is cheaper
>>>> than moving data". In this manner, size of data that transform engine
>>>> need
>>>> to handle can be easily multiple TB. Which will take long time to copy
>>>> to
>>>> local machine and process. So i think transform module should be run on
>>>> underlying distributed computing framework.
>>>>
>>>> And about Chart library, we have started discussion thread about
>>>> building
>>>> rich GUI inside of notebook. it might be related.
>>>>
>>>> Thanks,
>>>> moon
>>>>
>>>>
>>>>
>>>> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <
>>>> madhukaudantha@gmail.com>
>>>> wrote:
>>>>
>>>> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <
>>>> corneadoug@apache.org>
>>>> > wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > Being able to aggregate on the query side is a great idea and would
>>>> allow
>>>> > > us to transfer less data as well as having a full query
>>>> representation of
>>>> > > the visualization.
>>>> > >
>>>> > > However creating a SQL query dynamically is a pretty difficult
>>>> task, and
>>>> > > might be too much for that scope.
>>>> > >
>>>> > > Also I see some possible problems with this method:
>>>> > >  - Changing the pivot or simple filtering would mean running the
>>>> query
>>>> > > again
>>>> > >
>>>> > No, the query wont run again.
>>>> > In the first run of the query data is collected and stored locally-
>>>> local
>>>> > storage [1](using indexing techniques to make retrieval faster) So
>>>> changing
>>>> > pivot or simple filtering will use the local storage.
>>>> > If any attribute or data is missing in local storage then it will
>>>> retrieve
>>>> > only that and save the network bandwidth as well.
>>>> > Does my explanation make sense.
>>>> >
>>>> >
>>>> >
>>>> > >  - Being able to make pivot style SQL query would be really hard,
>>>> > >    we would need multiple sub-queries or even some times multiple
>>>> queries
>>>> > > (I tried a few times and could have the result wanted only with
>>>> > > visualization side pivot).
>>>> > >    It would end up with really bad SQL queries, especially with the
>>>> Hive
>>>> > > SQL or Spark SQL limitations and would take way more time to
>>>> process.
>>>> > >
>>>> > Agreed. I'm not planing to use pivot style queries.
>>>> >
>>>> > Any suggestions?
>>>> >
>>>> >
>>>> > Thanks.
>>>> >
>>>> >
>>>> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:
>>>> > >
>>>> > > > Hi,
>>>> > > >
>>>> > > > The Chart library features sounds promising.
>>>> > > > As  for the data engine - one thing that I think is missing is the
>>>> > > ability
>>>> > > > to use the visualization to drive the aggregation in the SQL.
>>>> today,
>>>> > you
>>>> > > > first write the SQL, you execute it, *limited by the number of
>>>> results
>>>> > > sent
>>>> > > > to the client*, and then you use viz to understand the results.
>>>> > > > Alternatively, if through the visualization I can generate a
>>>> better SQL
>>>> > > > which returns returns an aggregated data-set then I can analyze a
>>>> > bigger
>>>> > > > amount of data.
>>>> > > >
>>>> > > > I hope I was clear enough in my explanation :-)
>>>> > > >
>>>> > > > Eran
>>>> > > >
>>>> > > >
>>>> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
>>>> > > madhukaudantha@gmail.com
>>>> > > > >
>>>> > > > wrote:
>>>> > > >
>>>> > > > > Hi,
>>>> > > > >
>>>> > > > > Here is my proposing ideas.
>>>> > > > > According to COMDEV-119 jira. Charts are hard coded until now
>>>> and
>>>> > data
>>>> > > > > transformation issue was highlighted since different charts have
>>>> > > > different
>>>> > > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble
>>>> charts,
>>>> > > > Radar
>>>> > > > > charts. etc..
>>>> > > > >
>>>> > > > > To solve this I am introducing a two major component one is
>>>> called
>>>> > > 'Chart
>>>> > > > > library' and 'Data transformation engine'. Chart library is
>>>> located
>>>> > > where
>>>> > > > > it shows the chats that are currently plugged. There we can plug
>>>> > chart
>>>> > > > > types and those can be reused.
>>>> > > > >
>>>> > > > > *Chart library features *
>>>> > > > >
>>>> > > > >    - Users can select the chart from library
>>>> > > > >    - Those charts are pluggable to library
>>>> > > > >    - Charts can be plugged by config(json)/UI with wizard
>>>> > > > >    - Configuration/Meta file of the chart contains interface,
>>>> libs,
>>>> > > > themes
>>>> > > > >    and a data transformation types/mappings
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > *Data Transformation Engine*
>>>> > > > > 'Data transformation engine' contains data transformation
>>>> modules.
>>>> > > Those
>>>> > > > > modules are also pluggable to engine. Those have connections to
>>>> > charts.
>>>> > > > > Data transformation engine sit between the data (sql) and
>>>> chart. So
>>>> > > this
>>>> > > > > module  converts data and map them to each chart pivot field
>>>> > > > >
>>>> > > > >    - This module will look at pivot fields of the chart
>>>> > > > >    - Selected attributes of the SQL query
>>>> > > > >    - Attribute value operations improvement (string split, value
>>>> > > > >    aggregation, round number round)
>>>> > > > >
>>>> > > > >
>>>> > > > > Another improvement that I notice is that
>>>> > > > >
>>>> > > > >    - Query Edit auto-completion support (with Ctrl+space)
>>>> > > > >
>>>> > > > >
>>>> > > > > Your ideas are welcome here
>>>> > > > > Thanks
>>>> > > > >
>>>> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>> > > > > madhukaudantha@gmail.com>
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > > > Hi All,
>>>> > > > > >
>>>> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This
>>>> GSoC 2015
>>>> > > > > > project, 0COMDEV-1190 captures my interest.
>>>> > > > > >
>>>> > > > > > I have abundant experiences of visualization techniques
>>>> creating
>>>> > > > numerous
>>>> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3
>>>> charting etc.
>>>> > > > > >
>>>> > > > > > My current research area comprises of big data where I have
>>>> worked
>>>> > > with
>>>> > > > > > various types of data sets. Also I'm working with cluster
>>>> > > > representation
>>>> > > > > > and classification techniques where visualization amounts to a
>>>> > > > > considerable
>>>> > > > > > part. I was following COMDEV-119 (jira) with Alexander
>>>> Bezzubov and
>>>> > > > > CORNEAU
>>>> > > > > > Damien for more than week.
>>>> > > > > >
>>>> > > > > > Thanks
>>>> > > > > >
>>>> > > > > > [1] http://wso2.com/products/user-engagement-server/
>>>> > > > > > [2] https://github.com/wso2/jaggery
>>>> > > > > > --
>>>> > > > > > Cheers,
>>>> > > > > > Madhuka Udantha
>>>> > > > > > http://madhukaudantha.blogspot.com
>>>> > > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > --
>>>> > > > > Cheers,
>>>> > > > > Madhuka Udantha
>>>> > > > > http://madhukaudantha.blogspot.com
>>>> > > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > Eran | CTO
>>>> > > >
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Cheers,
>>>> > Madhuka Udantha
>>>> > http://madhukaudantha.blogspot.com
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

According to the discussion to get a clear understanding I just drew 2
sequence diagrams that  will explain
how chart will react to changing the pivot.

Safe Level
https://issues.apache.org/jira/secure/attachment/12707251/Changing%20the%20pivot%20-%20Safe%20Level.png


In safe level (default level) only limited amount of data is
retrieved(sufficient to draw the chart).
At initial stage local storage don't contain data. But when you make a
pivot change data will be there to draw the graph. If data is out-dated we
will get it from back-end.

Restricted Level
https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png
<https://issues.apache.org/jira/secure/attachment/12707250/Changing%20the%20pivot%20-%20Restricted%20Level.png>

User will reach Restricted Level after he successfully pass the Safe Level.
Then in local storage we will have up-to-date data. But for this level it
will be using all the data in the database. So Charting will grab the data
from storage and back-end.

Your ideas are mostly welcomed.


On Wed, Mar 25, 2015 at 2:55 PM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi,
>
> I want to know about the code structure and zeppelin architecture? Is
> there any good post / article / wiki regarding the said.
> Also if there is any quick start guide regarding development of Zeppelin
> please share it with me.
>
> Thanks.
>
> On Mon, Mar 23, 2015 at 10:44 AM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi, moon
>>
>> Yes, Since
>>
>>> "Moving computation is cheaper than moving data"
>>
>> We can do computation in computing framework.
>>
>> For simple pivot changing or filtering can be handle in local storage
>> with indexing databases depending on the current user level.
>> As you saw, computations will be handle in the back ends.
>>
>> Great to hear about the building rich GUI, I will give me chart library ideas
>> on there.
>>
>> Your ideas are always welcome, those will be helpful for my task and
>> draft proposal
>>
>> Thanks
>>
>> On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org> wrote:
>>
>>> Hi, madhuka udantha
>>>
>>> I think your idea about chart library and data transformation engine
>>> sounds
>>> cool. For the data transform modules, it's good idea to make this
>>> pluggable
>>> to data transform engine. But i'm not sure getting result locally and do
>>> transform for pivot or filtering to prevent run query again is good idea.
>>> Because of Zeppelin is (not limited but) trying to build analytical
>>> environment on top of distributed computing framework, like Spark, Flink,
>>> Ignite, etc. Most of distributed computing framework Zeppelin trying to
>>> integrate is following the same paradigm "Moving computation is cheaper
>>> than moving data". In this manner, size of data that transform engine
>>> need
>>> to handle can be easily multiple TB. Which will take long time to copy to
>>> local machine and process. So i think transform module should be run on
>>> underlying distributed computing framework.
>>>
>>> And about Chart library, we have started discussion thread about building
>>> rich GUI inside of notebook. it might be related.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>>
>>> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <
>>> madhukaudantha@gmail.com>
>>> wrote:
>>>
>>> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <corneadoug@apache.org
>>> >
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Being able to aggregate on the query side is a great idea and would
>>> allow
>>> > > us to transfer less data as well as having a full query
>>> representation of
>>> > > the visualization.
>>> > >
>>> > > However creating a SQL query dynamically is a pretty difficult task,
>>> and
>>> > > might be too much for that scope.
>>> > >
>>> > > Also I see some possible problems with this method:
>>> > >  - Changing the pivot or simple filtering would mean running the
>>> query
>>> > > again
>>> > >
>>> > No, the query wont run again.
>>> > In the first run of the query data is collected and stored locally-
>>> local
>>> > storage [1](using indexing techniques to make retrieval faster) So
>>> changing
>>> > pivot or simple filtering will use the local storage.
>>> > If any attribute or data is missing in local storage then it will
>>> retrieve
>>> > only that and save the network bandwidth as well.
>>> > Does my explanation make sense.
>>> >
>>> >
>>> >
>>> > >  - Being able to make pivot style SQL query would be really hard,
>>> > >    we would need multiple sub-queries or even some times multiple
>>> queries
>>> > > (I tried a few times and could have the result wanted only with
>>> > > visualization side pivot).
>>> > >    It would end up with really bad SQL queries, especially with the
>>> Hive
>>> > > SQL or Spark SQL limitations and would take way more time to process.
>>> > >
>>> > Agreed. I'm not planing to use pivot style queries.
>>> >
>>> > Any suggestions?
>>> >
>>> >
>>> > Thanks.
>>> >
>>> >
>>> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:
>>> > >
>>> > > > Hi,
>>> > > >
>>> > > > The Chart library features sounds promising.
>>> > > > As  for the data engine - one thing that I think is missing is the
>>> > > ability
>>> > > > to use the visualization to drive the aggregation in the SQL.
>>> today,
>>> > you
>>> > > > first write the SQL, you execute it, *limited by the number of
>>> results
>>> > > sent
>>> > > > to the client*, and then you use viz to understand the results.
>>> > > > Alternatively, if through the visualization I can generate a
>>> better SQL
>>> > > > which returns returns an aggregated data-set then I can analyze a
>>> > bigger
>>> > > > amount of data.
>>> > > >
>>> > > > I hope I was clear enough in my explanation :-)
>>> > > >
>>> > > > Eran
>>> > > >
>>> > > >
>>> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
>>> > > madhukaudantha@gmail.com
>>> > > > >
>>> > > > wrote:
>>> > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > Here is my proposing ideas.
>>> > > > > According to COMDEV-119 jira. Charts are hard coded until now and
>>> > data
>>> > > > > transformation issue was highlighted since different charts have
>>> > > > different
>>> > > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble
>>> charts,
>>> > > > Radar
>>> > > > > charts. etc..
>>> > > > >
>>> > > > > To solve this I am introducing a two major component one is
>>> called
>>> > > 'Chart
>>> > > > > library' and 'Data transformation engine'. Chart library is
>>> located
>>> > > where
>>> > > > > it shows the chats that are currently plugged. There we can plug
>>> > chart
>>> > > > > types and those can be reused.
>>> > > > >
>>> > > > > *Chart library features *
>>> > > > >
>>> > > > >    - Users can select the chart from library
>>> > > > >    - Those charts are pluggable to library
>>> > > > >    - Charts can be plugged by config(json)/UI with wizard
>>> > > > >    - Configuration/Meta file of the chart contains interface,
>>> libs,
>>> > > > themes
>>> > > > >    and a data transformation types/mappings
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > *Data Transformation Engine*
>>> > > > > 'Data transformation engine' contains data transformation
>>> modules.
>>> > > Those
>>> > > > > modules are also pluggable to engine. Those have connections to
>>> > charts.
>>> > > > > Data transformation engine sit between the data (sql) and chart.
>>> So
>>> > > this
>>> > > > > module  converts data and map them to each chart pivot field
>>> > > > >
>>> > > > >    - This module will look at pivot fields of the chart
>>> > > > >    - Selected attributes of the SQL query
>>> > > > >    - Attribute value operations improvement (string split, value
>>> > > > >    aggregation, round number round)
>>> > > > >
>>> > > > >
>>> > > > > Another improvement that I notice is that
>>> > > > >
>>> > > > >    - Query Edit auto-completion support (with Ctrl+space)
>>> > > > >
>>> > > > >
>>> > > > > Your ideas are welcome here
>>> > > > > Thanks
>>> > > > >
>>> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>> > > > > madhukaudantha@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi All,
>>> > > > > >
>>> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
>>> 2015
>>> > > > > > project, 0COMDEV-1190 captures my interest.
>>> > > > > >
>>> > > > > > I have abundant experiences of visualization techniques
>>> creating
>>> > > > numerous
>>> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3 charting
>>> etc.
>>> > > > > >
>>> > > > > > My current research area comprises of big data where I have
>>> worked
>>> > > with
>>> > > > > > various types of data sets. Also I'm working with cluster
>>> > > > representation
>>> > > > > > and classification techniques where visualization amounts to a
>>> > > > > considerable
>>> > > > > > part. I was following COMDEV-119 (jira) with Alexander
>>> Bezzubov and
>>> > > > > CORNEAU
>>> > > > > > Damien for more than week.
>>> > > > > >
>>> > > > > > Thanks
>>> > > > > >
>>> > > > > > [1] http://wso2.com/products/user-engagement-server/
>>> > > > > > [2] https://github.com/wso2/jaggery
>>> > > > > > --
>>> > > > > > Cheers,
>>> > > > > > Madhuka Udantha
>>> > > > > > http://madhukaudantha.blogspot.com
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Cheers,
>>> > > > > Madhuka Udantha
>>> > > > > http://madhukaudantha.blogspot.com
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Eran | CTO
>>> > > >
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> > Madhuka Udantha
>>> > http://madhukaudantha.blogspot.com
>>> >
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

I want to know about the code structure and zeppelin architecture? Is there
any good post / article / wiki regarding the said.
Also if there is any quick start guide regarding development of Zeppelin
please share it with me.

Thanks.

On Mon, Mar 23, 2015 at 10:44 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi, moon
>
> Yes, Since
>
>> "Moving computation is cheaper than moving data"
>
> We can do computation in computing framework.
>
> For simple pivot changing or filtering can be handle in local storage with
> indexing databases depending on the current user level.
> As you saw, computations will be handle in the back ends.
>
> Great to hear about the building rich GUI, I will give me chart library ideas
> on there.
>
> Your ideas are always welcome, those will be helpful for my task and draft
> proposal
>
> Thanks
>
> On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hi, madhuka udantha
>>
>> I think your idea about chart library and data transformation engine
>> sounds
>> cool. For the data transform modules, it's good idea to make this
>> pluggable
>> to data transform engine. But i'm not sure getting result locally and do
>> transform for pivot or filtering to prevent run query again is good idea.
>> Because of Zeppelin is (not limited but) trying to build analytical
>> environment on top of distributed computing framework, like Spark, Flink,
>> Ignite, etc. Most of distributed computing framework Zeppelin trying to
>> integrate is following the same paradigm "Moving computation is cheaper
>> than moving data". In this manner, size of data that transform engine need
>> to handle can be easily multiple TB. Which will take long time to copy to
>> local machine and process. So i think transform module should be run on
>> underlying distributed computing framework.
>>
>> And about Chart library, we have started discussion thread about building
>> rich GUI inside of notebook. it might be related.
>>
>> Thanks,
>> moon
>>
>>
>>
>> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <madhukaudantha@gmail.com
>> >
>> wrote:
>>
>> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <co...@apache.org>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Being able to aggregate on the query side is a great idea and would
>> allow
>> > > us to transfer less data as well as having a full query
>> representation of
>> > > the visualization.
>> > >
>> > > However creating a SQL query dynamically is a pretty difficult task,
>> and
>> > > might be too much for that scope.
>> > >
>> > > Also I see some possible problems with this method:
>> > >  - Changing the pivot or simple filtering would mean running the query
>> > > again
>> > >
>> > No, the query wont run again.
>> > In the first run of the query data is collected and stored locally-
>> local
>> > storage [1](using indexing techniques to make retrieval faster) So
>> changing
>> > pivot or simple filtering will use the local storage.
>> > If any attribute or data is missing in local storage then it will
>> retrieve
>> > only that and save the network bandwidth as well.
>> > Does my explanation make sense.
>> >
>> >
>> >
>> > >  - Being able to make pivot style SQL query would be really hard,
>> > >    we would need multiple sub-queries or even some times multiple
>> queries
>> > > (I tried a few times and could have the result wanted only with
>> > > visualization side pivot).
>> > >    It would end up with really bad SQL queries, especially with the
>> Hive
>> > > SQL or Spark SQL limitations and would take way more time to process.
>> > >
>> > Agreed. I'm not planing to use pivot style queries.
>> >
>> > Any suggestions?
>> >
>> >
>> > Thanks.
>> >
>> >
>> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > The Chart library features sounds promising.
>> > > > As  for the data engine - one thing that I think is missing is the
>> > > ability
>> > > > to use the visualization to drive the aggregation in the SQL. today,
>> > you
>> > > > first write the SQL, you execute it, *limited by the number of
>> results
>> > > sent
>> > > > to the client*, and then you use viz to understand the results.
>> > > > Alternatively, if through the visualization I can generate a better
>> SQL
>> > > > which returns returns an aggregated data-set then I can analyze a
>> > bigger
>> > > > amount of data.
>> > > >
>> > > > I hope I was clear enough in my explanation :-)
>> > > >
>> > > > Eran
>> > > >
>> > > >
>> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
>> > > madhukaudantha@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > Here is my proposing ideas.
>> > > > > According to COMDEV-119 jira. Charts are hard coded until now and
>> > data
>> > > > > transformation issue was highlighted since different charts have
>> > > > different
>> > > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble
>> charts,
>> > > > Radar
>> > > > > charts. etc..
>> > > > >
>> > > > > To solve this I am introducing a two major component one is called
>> > > 'Chart
>> > > > > library' and 'Data transformation engine'. Chart library is
>> located
>> > > where
>> > > > > it shows the chats that are currently plugged. There we can plug
>> > chart
>> > > > > types and those can be reused.
>> > > > >
>> > > > > *Chart library features *
>> > > > >
>> > > > >    - Users can select the chart from library
>> > > > >    - Those charts are pluggable to library
>> > > > >    - Charts can be plugged by config(json)/UI with wizard
>> > > > >    - Configuration/Meta file of the chart contains interface,
>> libs,
>> > > > themes
>> > > > >    and a data transformation types/mappings
>> > > > >
>> > > > >
>> > > > >
>> > > > > *Data Transformation Engine*
>> > > > > 'Data transformation engine' contains data transformation modules.
>> > > Those
>> > > > > modules are also pluggable to engine. Those have connections to
>> > charts.
>> > > > > Data transformation engine sit between the data (sql) and chart.
>> So
>> > > this
>> > > > > module  converts data and map them to each chart pivot field
>> > > > >
>> > > > >    - This module will look at pivot fields of the chart
>> > > > >    - Selected attributes of the SQL query
>> > > > >    - Attribute value operations improvement (string split, value
>> > > > >    aggregation, round number round)
>> > > > >
>> > > > >
>> > > > > Another improvement that I notice is that
>> > > > >
>> > > > >    - Query Edit auto-completion support (with Ctrl+space)
>> > > > >
>> > > > >
>> > > > > Your ideas are welcome here
>> > > > > Thanks
>> > > > >
>> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>> > > > > madhukaudantha@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
>> 2015
>> > > > > > project, 0COMDEV-1190 captures my interest.
>> > > > > >
>> > > > > > I have abundant experiences of visualization techniques creating
>> > > > numerous
>> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3 charting
>> etc.
>> > > > > >
>> > > > > > My current research area comprises of big data where I have
>> worked
>> > > with
>> > > > > > various types of data sets. Also I'm working with cluster
>> > > > representation
>> > > > > > and classification techniques where visualization amounts to a
>> > > > > considerable
>> > > > > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov
>> and
>> > > > > CORNEAU
>> > > > > > Damien for more than week.
>> > > > > >
>> > > > > > Thanks
>> > > > > >
>> > > > > > [1] http://wso2.com/products/user-engagement-server/
>> > > > > > [2] https://github.com/wso2/jaggery
>> > > > > > --
>> > > > > > Cheers,
>> > > > > > Madhuka Udantha
>> > > > > > http://madhukaudantha.blogspot.com
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Cheers,
>> > > > > Madhuka Udantha
>> > > > > http://madhukaudantha.blogspot.com
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Eran | CTO
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Cheers,
>> > Madhuka Udantha
>> > http://madhukaudantha.blogspot.com
>> >
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi, moon

Yes, Since

> "Moving computation is cheaper than moving data"

We can do computation in computing framework.

For simple pivot changing or filtering can be handle in local storage with
indexing databases depending on the current user level.
As you saw, computations will be handle in the back ends.

Great to hear about the building rich GUI, I will give me chart library ideas
on there.

Your ideas are always welcome, those will be helpful for my task and draft
proposal

Thanks

On Mon, Mar 23, 2015 at 7:59 AM, moon soo Lee <mo...@apache.org> wrote:

> Hi, madhuka udantha
>
> I think your idea about chart library and data transformation engine sounds
> cool. For the data transform modules, it's good idea to make this pluggable
> to data transform engine. But i'm not sure getting result locally and do
> transform for pivot or filtering to prevent run query again is good idea.
> Because of Zeppelin is (not limited but) trying to build analytical
> environment on top of distributed computing framework, like Spark, Flink,
> Ignite, etc. Most of distributed computing framework Zeppelin trying to
> integrate is following the same paradigm "Moving computation is cheaper
> than moving data". In this manner, size of data that transform engine need
> to handle can be easily multiple TB. Which will take long time to copy to
> local machine and process. So i think transform module should be run on
> underlying distributed computing framework.
>
> And about Chart library, we have started discussion thread about building
> rich GUI inside of notebook. it might be related.
>
> Thanks,
> moon
>
>
>
> On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <ma...@gmail.com>
> wrote:
>
> > On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <co...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > Being able to aggregate on the query side is a great idea and would
> allow
> > > us to transfer less data as well as having a full query representation
> of
> > > the visualization.
> > >
> > > However creating a SQL query dynamically is a pretty difficult task,
> and
> > > might be too much for that scope.
> > >
> > > Also I see some possible problems with this method:
> > >  - Changing the pivot or simple filtering would mean running the query
> > > again
> > >
> > No, the query wont run again.
> > In the first run of the query data is collected and stored locally- local
> > storage [1](using indexing techniques to make retrieval faster) So
> changing
> > pivot or simple filtering will use the local storage.
> > If any attribute or data is missing in local storage then it will
> retrieve
> > only that and save the network bandwidth as well.
> > Does my explanation make sense.
> >
> >
> >
> > >  - Being able to make pivot style SQL query would be really hard,
> > >    we would need multiple sub-queries or even some times multiple
> queries
> > > (I tried a few times and could have the result wanted only with
> > > visualization side pivot).
> > >    It would end up with really bad SQL queries, especially with the
> Hive
> > > SQL or Spark SQL limitations and would take way more time to process.
> > >
> > Agreed. I'm not planing to use pivot style queries.
> >
> > Any suggestions?
> >
> >
> > Thanks.
> >
> >
> > > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > The Chart library features sounds promising.
> > > > As  for the data engine - one thing that I think is missing is the
> > > ability
> > > > to use the visualization to drive the aggregation in the SQL. today,
> > you
> > > > first write the SQL, you execute it, *limited by the number of
> results
> > > sent
> > > > to the client*, and then you use viz to understand the results.
> > > > Alternatively, if through the visualization I can generate a better
> SQL
> > > > which returns returns an aggregated data-set then I can analyze a
> > bigger
> > > > amount of data.
> > > >
> > > > I hope I was clear enough in my explanation :-)
> > > >
> > > > Eran
> > > >
> > > >
> > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
> > > madhukaudantha@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Here is my proposing ideas.
> > > > > According to COMDEV-119 jira. Charts are hard coded until now and
> > data
> > > > > transformation issue was highlighted since different charts have
> > > > different
> > > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble
> charts,
> > > > Radar
> > > > > charts. etc..
> > > > >
> > > > > To solve this I am introducing a two major component one is called
> > > 'Chart
> > > > > library' and 'Data transformation engine'. Chart library is located
> > > where
> > > > > it shows the chats that are currently plugged. There we can plug
> > chart
> > > > > types and those can be reused.
> > > > >
> > > > > *Chart library features *
> > > > >
> > > > >    - Users can select the chart from library
> > > > >    - Those charts are pluggable to library
> > > > >    - Charts can be plugged by config(json)/UI with wizard
> > > > >    - Configuration/Meta file of the chart contains interface, libs,
> > > > themes
> > > > >    and a data transformation types/mappings
> > > > >
> > > > >
> > > > >
> > > > > *Data Transformation Engine*
> > > > > 'Data transformation engine' contains data transformation modules.
> > > Those
> > > > > modules are also pluggable to engine. Those have connections to
> > charts.
> > > > > Data transformation engine sit between the data (sql) and chart. So
> > > this
> > > > > module  converts data and map them to each chart pivot field
> > > > >
> > > > >    - This module will look at pivot fields of the chart
> > > > >    - Selected attributes of the SQL query
> > > > >    - Attribute value operations improvement (string split, value
> > > > >    aggregation, round number round)
> > > > >
> > > > >
> > > > > Another improvement that I notice is that
> > > > >
> > > > >    - Query Edit auto-completion support (with Ctrl+space)
> > > > >
> > > > >
> > > > > Your ideas are welcome here
> > > > > Thanks
> > > > >
> > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > > > > madhukaudantha@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
> 2015
> > > > > > project, 0COMDEV-1190 captures my interest.
> > > > > >
> > > > > > I have abundant experiences of visualization techniques creating
> > > > numerous
> > > > > > dashboards[1,2] with javascript, html5, angularJS, d3 charting
> etc.
> > > > > >
> > > > > > My current research area comprises of big data where I have
> worked
> > > with
> > > > > > various types of data sets. Also I'm working with cluster
> > > > representation
> > > > > > and classification techniques where visualization amounts to a
> > > > > considerable
> > > > > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov
> and
> > > > > CORNEAU
> > > > > > Damien for more than week.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > [1] http://wso2.com/products/user-engagement-server/
> > > > > > [2] https://github.com/wso2/jaggery
> > > > > > --
> > > > > > Cheers,
> > > > > > Madhuka Udantha
> > > > > > http://madhukaudantha.blogspot.com
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers,
> > > > > Madhuka Udantha
> > > > > http://madhukaudantha.blogspot.com
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Eran | CTO
> > > >
> > >
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
> >
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by moon soo Lee <mo...@apache.org>.

Hi, madhuka udantha

I think your idea about chart library and data transformation engine sounds
cool. For the data transform modules, it's good idea to make this pluggable
to data transform engine. But i'm not sure getting result locally and do
transform for pivot or filtering to prevent run query again is good idea.
Because of Zeppelin is (not limited but) trying to build analytical
environment on top of distributed computing framework, like Spark, Flink,
Ignite, etc. Most of distributed computing framework Zeppelin trying to
integrate is following the same paradigm "Moving computation is cheaper
than moving data". In this manner, size of data that transform engine need
to handle can be easily multiple TB. Which will take long time to copy to
local machine and process. So i think transform module should be run on
underlying distributed computing framework.

And about Chart library, we have started discussion thread about building
rich GUI inside of notebook. it might be related.

Thanks,
moon



On Mon, Mar 23, 2015 at 2:27 AM madhuka udantha <ma...@gmail.com>
wrote:

> On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <co...@apache.org>
> wrote:
>
> > Hi,
> >
> > Being able to aggregate on the query side is a great idea and would allow
> > us to transfer less data as well as having a full query representation of
> > the visualization.
> >
> > However creating a SQL query dynamically is a pretty difficult task, and
> > might be too much for that scope.
> >
> > Also I see some possible problems with this method:
> >  - Changing the pivot or simple filtering would mean running the query
> > again
> >
> No, the query wont run again.
> In the first run of the query data is collected and stored locally- local
> storage [1](using indexing techniques to make retrieval faster) So changing
> pivot or simple filtering will use the local storage.
> If any attribute or data is missing in local storage then it will retrieve
> only that and save the network bandwidth as well.
> Does my explanation make sense.
>
>
>
> >  - Being able to make pivot style SQL query would be really hard,
> >    we would need multiple sub-queries or even some times multiple queries
> > (I tried a few times and could have the result wanted only with
> > visualization side pivot).
> >    It would end up with really bad SQL queries, especially with the Hive
> > SQL or Spark SQL limitations and would take way more time to process.
> >
> Agreed. I'm not planing to use pivot style queries.
>
> Any suggestions?
>
>
> Thanks.
>
>
> > On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > The Chart library features sounds promising.
> > > As  for the data engine - one thing that I think is missing is the
> > ability
> > > to use the visualization to drive the aggregation in the SQL. today,
> you
> > > first write the SQL, you execute it, *limited by the number of results
> > sent
> > > to the client*, and then you use viz to understand the results.
> > > Alternatively, if through the visualization I can generate a better SQL
> > > which returns returns an aggregated data-set then I can analyze a
> bigger
> > > amount of data.
> > >
> > > I hope I was clear enough in my explanation :-)
> > >
> > > Eran
> > >
> > >
> > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
> > madhukaudantha@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Here is my proposing ideas.
> > > > According to COMDEV-119 jira. Charts are hard coded until now and
> data
> > > > transformation issue was highlighted since different charts have
> > > different
> > > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts,
> > > Radar
> > > > charts. etc..
> > > >
> > > > To solve this I am introducing a two major component one is called
> > 'Chart
> > > > library' and 'Data transformation engine'. Chart library is located
> > where
> > > > it shows the chats that are currently plugged. There we can plug
> chart
> > > > types and those can be reused.
> > > >
> > > > *Chart library features *
> > > >
> > > >    - Users can select the chart from library
> > > >    - Those charts are pluggable to library
> > > >    - Charts can be plugged by config(json)/UI with wizard
> > > >    - Configuration/Meta file of the chart contains interface, libs,
> > > themes
> > > >    and a data transformation types/mappings
> > > >
> > > >
> > > >
> > > > *Data Transformation Engine*
> > > > 'Data transformation engine' contains data transformation modules.
> > Those
> > > > modules are also pluggable to engine. Those have connections to
> charts.
> > > > Data transformation engine sit between the data (sql) and chart. So
> > this
> > > > module  converts data and map them to each chart pivot field
> > > >
> > > >    - This module will look at pivot fields of the chart
> > > >    - Selected attributes of the SQL query
> > > >    - Attribute value operations improvement (string split, value
> > > >    aggregation, round number round)
> > > >
> > > >
> > > > Another improvement that I notice is that
> > > >
> > > >    - Query Edit auto-completion support (with Ctrl+space)
> > > >
> > > >
> > > > Your ideas are welcome here
> > > > Thanks
> > > >
> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > > > madhukaudantha@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > > > > project, 0COMDEV-1190 captures my interest.
> > > > >
> > > > > I have abundant experiences of visualization techniques creating
> > > numerous
> > > > > dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> > > > >
> > > > > My current research area comprises of big data where I have worked
> > with
> > > > > various types of data sets. Also I'm working with cluster
> > > representation
> > > > > and classification techniques where visualization amounts to a
> > > > considerable
> > > > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> > > > CORNEAU
> > > > > Damien for more than week.
> > > > >
> > > > > Thanks
> > > > >
> > > > > [1] http://wso2.com/products/user-engagement-server/
> > > > > [2] https://github.com/wso2/jaggery
> > > > > --
> > > > > Cheers,
> > > > > Madhuka Udantha
> > > > > http://madhukaudantha.blogspot.com
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers,
> > > > Madhuka Udantha
> > > > http://madhukaudantha.blogspot.com
> > > >
> > >
> > >
> > >
> > > --
> > > Eran | CTO
> > >
> >
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

On Sun, Mar 22, 2015 at 7:05 PM, Corneau Damien <co...@apache.org>
wrote:

> Hi,
>
> Being able to aggregate on the query side is a great idea and would allow
> us to transfer less data as well as having a full query representation of
> the visualization.
>
> However creating a SQL query dynamically is a pretty difficult task, and
> might be too much for that scope.
>
> Also I see some possible problems with this method:
>  - Changing the pivot or simple filtering would mean running the query
> again
>
No, the query wont run again.
In the first run of the query data is collected and stored locally- local
storage [1](using indexing techniques to make retrieval faster) So changing
pivot or simple filtering will use the local storage.
If any attribute or data is missing in local storage then it will retrieve
only that and save the network bandwidth as well.
Does my explanation make sense.



>  - Being able to make pivot style SQL query would be really hard,
>    we would need multiple sub-queries or even some times multiple queries
> (I tried a few times and could have the result wanted only with
> visualization side pivot).
>    It would end up with really bad SQL queries, especially with the Hive
> SQL or Spark SQL limitations and would take way more time to process.
>
Agreed. I'm not planing to use pivot style queries.

Any suggestions?


Thanks.


> On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:
>
> > Hi,
> >
> > The Chart library features sounds promising.
> > As  for the data engine - one thing that I think is missing is the
> ability
> > to use the visualization to drive the aggregation in the SQL. today, you
> > first write the SQL, you execute it, *limited by the number of results
> sent
> > to the client*, and then you use viz to understand the results.
> > Alternatively, if through the visualization I can generate a better SQL
> > which returns returns an aggregated data-set then I can analyze a bigger
> > amount of data.
> >
> > I hope I was clear enough in my explanation :-)
> >
> > Eran
> >
> >
> > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <
> madhukaudantha@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > Here is my proposing ideas.
> > > According to COMDEV-119 jira. Charts are hard coded until now and data
> > > transformation issue was highlighted since different charts have
> > different
> > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts,
> > Radar
> > > charts. etc..
> > >
> > > To solve this I am introducing a two major component one is called
> 'Chart
> > > library' and 'Data transformation engine'. Chart library is located
> where
> > > it shows the chats that are currently plugged. There we can plug chart
> > > types and those can be reused.
> > >
> > > *Chart library features *
> > >
> > >    - Users can select the chart from library
> > >    - Those charts are pluggable to library
> > >    - Charts can be plugged by config(json)/UI with wizard
> > >    - Configuration/Meta file of the chart contains interface, libs,
> > themes
> > >    and a data transformation types/mappings
> > >
> > >
> > >
> > > *Data Transformation Engine*
> > > 'Data transformation engine' contains data transformation modules.
> Those
> > > modules are also pluggable to engine. Those have connections to charts.
> > > Data transformation engine sit between the data (sql) and chart. So
> this
> > > module  converts data and map them to each chart pivot field
> > >
> > >    - This module will look at pivot fields of the chart
> > >    - Selected attributes of the SQL query
> > >    - Attribute value operations improvement (string split, value
> > >    aggregation, round number round)
> > >
> > >
> > > Another improvement that I notice is that
> > >
> > >    - Query Edit auto-completion support (with Ctrl+space)
> > >
> > >
> > > Your ideas are welcome here
> > > Thanks
> > >
> > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > > madhukaudantha@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > > > project, 0COMDEV-1190 captures my interest.
> > > >
> > > > I have abundant experiences of visualization techniques creating
> > numerous
> > > > dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> > > >
> > > > My current research area comprises of big data where I have worked
> with
> > > > various types of data sets. Also I'm working with cluster
> > representation
> > > > and classification techniques where visualization amounts to a
> > > considerable
> > > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> > > CORNEAU
> > > > Damien for more than week.
> > > >
> > > > Thanks
> > > >
> > > > [1] http://wso2.com/products/user-engagement-server/
> > > > [2] https://github.com/wso2/jaggery
> > > > --
> > > > Cheers,
> > > > Madhuka Udantha
> > > > http://madhukaudantha.blogspot.com
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > > Madhuka Udantha
> > > http://madhukaudantha.blogspot.com
> > >
> >
> >
> >
> > --
> > Eran | CTO
> >
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by Corneau Damien <co...@apache.org>.

Hi,

Being able to aggregate on the query side is a great idea and would allow
us to transfer less data as well as having a full query representation of
the visualization.

However creating a SQL query dynamically is a pretty difficult task, and
might be too much for that scope.

Also I see some possible problems with this method:
 - Changing the pivot or simple filtering would mean running the query again
 - Being able to make pivot style SQL query would be really hard,
   we would need multiple sub-queries or even some times multiple queries
(I tried a few times and could have the result wanted only with
visualization side pivot).
   It would end up with really bad SQL queries, especially with the Hive
SQL or Spark SQL limitations and would take way more time to process.

On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <go...@gmail.com> wrote:

> Hi,
>
> The Chart library features sounds promising.
> As  for the data engine - one thing that I think is missing is the ability
> to use the visualization to drive the aggregation in the SQL. today, you
> first write the SQL, you execute it, *limited by the number of results sent
> to the client*, and then you use viz to understand the results.
> Alternatively, if through the visualization I can generate a better SQL
> which returns returns an aggregated data-set then I can analyze a bigger
> amount of data.
>
> I hope I was clear enough in my explanation :-)
>
> Eran
>
>
> On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <madhukaudantha@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > Here is my proposing ideas.
> > According to COMDEV-119 jira. Charts are hard coded until now and data
> > transformation issue was highlighted since different charts have
> different
> > pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts,
> Radar
> > charts. etc..
> >
> > To solve this I am introducing a two major component one is called 'Chart
> > library' and 'Data transformation engine'. Chart library is located where
> > it shows the chats that are currently plugged. There we can plug chart
> > types and those can be reused.
> >
> > *Chart library features *
> >
> >    - Users can select the chart from library
> >    - Those charts are pluggable to library
> >    - Charts can be plugged by config(json)/UI with wizard
> >    - Configuration/Meta file of the chart contains interface, libs,
> themes
> >    and a data transformation types/mappings
> >
> >
> >
> > *Data Transformation Engine*
> > 'Data transformation engine' contains data transformation modules. Those
> > modules are also pluggable to engine. Those have connections to charts.
> > Data transformation engine sit between the data (sql) and chart. So this
> > module  converts data and map them to each chart pivot field
> >
> >    - This module will look at pivot fields of the chart
> >    - Selected attributes of the SQL query
> >    - Attribute value operations improvement (string split, value
> >    aggregation, round number round)
> >
> >
> > Another improvement that I notice is that
> >
> >    - Query Edit auto-completion support (with Ctrl+space)
> >
> >
> > Your ideas are welcome here
> > Thanks
> >
> > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > madhukaudantha@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > > project, 0COMDEV-1190 captures my interest.
> > >
> > > I have abundant experiences of visualization techniques creating
> numerous
> > > dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> > >
> > > My current research area comprises of big data where I have worked with
> > > various types of data sets. Also I'm working with cluster
> representation
> > > and classification techniques where visualization amounts to a
> > considerable
> > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> > CORNEAU
> > > Damien for more than week.
> > >
> > > Thanks
> > >
> > > [1] http://wso2.com/products/user-engagement-server/
> > > [2] https://github.com/wso2/jaggery
> > > --
> > > Cheers,
> > > Madhuka Udantha
> > > http://madhukaudantha.blogspot.com
> > >
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
> >
>
>
>
> --
> Eran | CTO
>

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi, Eran

Yes I agree on your point. And it's a worthwhile idea you shared with us.

I also thought that we could have different levels such as safe, medium,
restricted and highly restricted so that the user can reach higher levels
by acceptance of the lower levels after successful retrieval or
visualization.

I hope my point it clear enough. If you have any other ideas you are
welcomed to share with us.

Thanks.


On Sun, Mar 22, 2015 at 6:38 PM, IT CTO <go...@gmail.com> wrote:

> Hi,
>
> The Chart library features sounds promising.
> As  for the data engine - one thing that I think is missing is the ability
> to use the visualization to drive the aggregation in the SQL. today, you
> first write the SQL, you execute it, *limited by the number of results sent
> to the client*, and then you use viz to understand the results.
> Alternatively, if through the visualization I can generate a better SQL
> which returns returns an aggregated data-set then I can analyze a bigger
> amount of data.
>
> I hope I was clear enough in my explanation :-)
>
> Eran
>
>
> On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <madhukaudantha@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > Here is my proposing ideas.
> > According to COMDEV-119 jira. Charts are hard coded until now and data
> > transformation issue was highlighted since different charts have
> different
> > pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts,
> Radar
> > charts. etc..
> >
> > To solve this I am introducing a two major component one is called 'Chart
> > library' and 'Data transformation engine'. Chart library is located where
> > it shows the chats that are currently plugged. There we can plug chart
> > types and those can be reused.
> >
> > *Chart library features *
> >
> >    - Users can select the chart from library
> >    - Those charts are pluggable to library
> >    - Charts can be plugged by config(json)/UI with wizard
> >    - Configuration/Meta file of the chart contains interface, libs,
> themes
> >    and a data transformation types/mappings
> >
> >
> >
> > *Data Transformation Engine*
> > 'Data transformation engine' contains data transformation modules. Those
> > modules are also pluggable to engine. Those have connections to charts.
> > Data transformation engine sit between the data (sql) and chart. So this
> > module  converts data and map them to each chart pivot field
> >
> >    - This module will look at pivot fields of the chart
> >    - Selected attributes of the SQL query
> >    - Attribute value operations improvement (string split, value
> >    aggregation, round number round)
> >
> >
> > Another improvement that I notice is that
> >
> >    - Query Edit auto-completion support (with Ctrl+space)
> >
> >
> > Your ideas are welcome here
> > Thanks
> >
> > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > madhukaudantha@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > > project, 0COMDEV-1190 captures my interest.
> > >
> > > I have abundant experiences of visualization techniques creating
> numerous
> > > dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> > >
> > > My current research area comprises of big data where I have worked with
> > > various types of data sets. Also I'm working with cluster
> representation
> > > and classification techniques where visualization amounts to a
> > considerable
> > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> > CORNEAU
> > > Damien for more than week.
> > >
> > > Thanks
> > >
> > > [1] http://wso2.com/products/user-engagement-server/
> > > [2] https://github.com/wso2/jaggery
> > > --
> > > Cheers,
> > > Madhuka Udantha
> > > http://madhukaudantha.blogspot.com
> > >
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
> >
>
>
>
> --
> Eran | CTO
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by IT CTO <go...@gmail.com>.

Hi,

The Chart library features sounds promising.
As  for the data engine - one thing that I think is missing is the ability
to use the visualization to drive the aggregation in the SQL. today, you
first write the SQL, you execute it, *limited by the number of results sent
to the client*, and then you use viz to understand the results.
Alternatively, if through the visualization I can generate a better SQL
which returns returns an aggregated data-set then I can analyze a bigger
amount of data.

I hope I was clear enough in my explanation :-)

Eran


On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi,
>
> Here is my proposing ideas.
> According to COMDEV-119 jira. Charts are hard coded until now and data
> transformation issue was highlighted since different charts have different
> pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts, Radar
> charts. etc..
>
> To solve this I am introducing a two major component one is called 'Chart
> library' and 'Data transformation engine'. Chart library is located where
> it shows the chats that are currently plugged. There we can plug chart
> types and those can be reused.
>
> *Chart library features *
>
>    - Users can select the chart from library
>    - Those charts are pluggable to library
>    - Charts can be plugged by config(json)/UI with wizard
>    - Configuration/Meta file of the chart contains interface, libs, themes
>    and a data transformation types/mappings
>
>
>
> *Data Transformation Engine*
> 'Data transformation engine' contains data transformation modules. Those
> modules are also pluggable to engine. Those have connections to charts.
> Data transformation engine sit between the data (sql) and chart. So this
> module  converts data and map them to each chart pivot field
>
>    - This module will look at pivot fields of the chart
>    - Selected attributes of the SQL query
>    - Attribute value operations improvement (string split, value
>    aggregation, round number round)
>
>
> Another improvement that I notice is that
>
>    - Query Edit auto-completion support (with Ctrl+space)
>
>
> Your ideas are welcome here
> Thanks
>
> On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> madhukaudantha@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > project, 0COMDEV-1190 captures my interest.
> >
> > I have abundant experiences of visualization techniques creating numerous
> > dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> >
> > My current research area comprises of big data where I have worked with
> > various types of data sets. Also I'm working with cluster representation
> > and classification techniques where visualization amounts to a
> considerable
> > part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> CORNEAU
> > Damien for more than week.
> >
> > Thanks
> >
> > [1] http://wso2.com/products/user-engagement-server/
> > [2] https://github.com/wso2/jaggery
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
> >
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Eran | CTO

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

Here is my proposing ideas.
According to COMDEV-119 jira. Charts are hard coded until now and data
transformation issue was highlighted since different charts have different
pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts, Radar
charts. etc..

To solve this I am introducing a two major component one is called 'Chart
library' and 'Data transformation engine'. Chart library is located where
it shows the chats that are currently plugged. There we can plug chart
types and those can be reused.

*Chart library features *

   - Users can select the chart from library
   - Those charts are pluggable to library
   - Charts can be plugged by config(json)/UI with wizard
   - Configuration/Meta file of the chart contains interface, libs, themes
   and a data transformation types/mappings

*Data Transformation Engine*
'Data transformation engine' contains data transformation modules. Those
modules are also pluggable to engine. Those have connections to charts.
Data transformation engine sit between the data (sql) and chart. So this
module  converts data and map them to each chart pivot field

   - This module will look at pivot fields of the chart
   - Selected attributes of the SQL query
   - Attribute value operations improvement (string split, value
   aggregation, round number round)

Another improvement that I notice is that

   - Query Edit auto-completion support (with Ctrl+space)

Your ideas are welcome here
Thanks

On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi All,
>
> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> project, 0COMDEV-1190 captures my interest.
>
> I have abundant experiences of visualization techniques creating numerous
> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
>
> My current research area comprises of big data where I have worked with
> various types of data sets. Also I'm working with cluster representation
> and classification techniques where visualization amounts to a considerable
> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and CORNEAU
> Damien for more than week.
>
> Thanks
>
> [1] http://wso2.com/products/user-engagement-server/
> [2] https://github.com/wso2/jaggery
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>

-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi, Damien

Thanks for sharing information.
I will look in to 'Spring Cleaning PR' and start work in there.

On Thu, May 21, 2015 at 7:26 AM, Corneau Damien <co...@apache.org>
wrote:

> Hi madhuka,
>
> The Spring Cleaning PR will change the code base a lot, the way the code
> work will be the same (for 80% of the case), so you can familiarize
> yourself with it however keep in mind that the Spring Cleaning PR will
> change:
> * folder structure
> * Separate some code into multiple files
> * Refactor some code to be simpler/optimized
>
> The folder structure part is kinda set right now (just a few tiny changes
> more), and we are working on making some refactoring ready as fast as
> possible.
>
> I think that on Monday, you can get familiar with that PR new structure
> and work from there, it should help you understand better all the part of
> the zeppelin-web.
>
>
> On Thu, May 21, 2015 at 1:46 AM, madhuka udantha <madhukaudantha@gmail.com
> > wrote:
>
>> Hi All,
>>
>>
>> I went through google charts / d3 (chart-options, chart-types, chart-data
>> and  chart-format)  with  angularJS. And I have written few blog
>> articles regard to the topics[1,2].
>> To Familiarize myself with Zepplin I wrote angular controller and new tab
>> for  Zepplin-Web app.
>>
>> As 'spring cleaning of Zeppelin-web'[3] is in-progress, will this change
>> existing code base? I'm currently working on zepplin web app. Will the
>> change be significant? Shall I familiarize with the existing code base or
>> the resulting one of the 'spring cleaning of Zeppelin-web'
>>
>>
>>
>> [1] http://madhukaudantha.blogspot.com/search/label/Google%20Chart
>> [2] https://github.com/Madhuka/angularjs-tutorial
>> [3] https://github.com/apache/incubator-zeppelin/pull/56
>>
>> On Fri, May 1, 2015 at 9:40 AM, Alex B. <ab...@nflabs.com> wrote:
>>
>>> Congratulations, Madhuka!
>>>
>>> Glad that you can make it and thank you for your contributions. So we
>>> will go on with the initial plan of making more visualizations for Zeppelin
>>> then.
>>>
>>> Meanwhile it is community bonding period [1] I would suggest you go on
>>> participating at the dev@ and user@ mailing lists, helping people,
>>> sharing your experience with Zeppelin and meanwhile you can also try doing
>>> a simple PoC as Ilya suggested, by picking two different libraries or
>>> chart types (1 existing, 1 new) and writing an implementation of the
>>> transformation function to fit them together.
>>>
>>> Please keep in touch and let us know who it goes every week or two in a
>>> form of short email with your progress summary. We can also arrange
>>> something like a regular call\IRC hours if you will.
>>>
>>> Thanks again, please feel free to reach out in case of any further
>>> questions!
>>>
>>> [1]
>>> http://googlesummerofcode.blogspot.kr/2007/04/so-what-is-this-community-bonding-all.html
>>>
>>> On Thu, Apr 30, 2015 at 12:57 PM, madhuka udantha <
>>> madhukaudantha@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thank you all for your participation in this thread and the support I
>>>> got. My GSoC proposal for Zeppelin had been accepted. Hope you will
>>>> continue to give the same support you have been giving me to see to the
>>>> successful finishing of the project.
>>>>
>>>> Also Thanks to Corneau for spring cleaning zeppelin. I can too support
>>>> this if needed. Please share any thread which will be discussing [1] so I
>>>> can go through the code changes happening.
>>>>
>>>> Currently I'm using zeppelin/master on github. Is it alright? Do you
>>>> have any ideas to what my first deliverable would need to contain?
>>>>
>>>> Thanks.
>>>>
>>>> [1] https://github.com/apache/incubator-zeppelin/pull/56
>>>>
>>>> On Wed, Apr 29, 2015 at 2:41 PM, Corneau Damien <co...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Madhuka and dev mailing list,
>>>>>
>>>>> Since the code became quite a mess (didn't supervised it enough I
>>>>> guess :p
>>>>> ), and GSOC 2015 will start soon,
>>>>> I'm currently working on making things easier for people who would
>>>>> like to
>>>>> participate in the front-end of Zeppelin.
>>>>> Work in Progress in that Pull Request :
>>>>> https://github.com/apache/incubator-zeppelin/pull/56
>>>>>
>>>>> It should be finished before the GSOC 2015 coding time start :)
>>>>>
>>>>> On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi Madhuka,
>>>>> >
>>>>> > I’m not aware of any efforts in bridging different chart libraries.
>>>>> But my
>>>>> > guess is, we’ll have to do it ourselves. You can start by picking two
>>>>> > different libraries or chart types and writing an implementation of
>>>>> the
>>>>> > transformation function. It might help you to assess the scope of
>>>>> work.
>>>>> >
>>>>> > Regarding paragraph.js.
>>>>> > I don’t think there is documentation for it.
>>>>> > So the best way to understand the data flow it is to read the code.
>>>>> > I know it’s not very helpful, but I don’t think anyone can help you
>>>>> in
>>>>> > this situation.
>>>>> >
>>>>> > Regards,
>>>>> > Ilya
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
>>>>> > madhukaudantha@gmail.com> wrote:
>>>>> >
>>>>> >> Hi,
>>>>> >>
>>>>> >> In different charting libraries the data template take different
>>>>> formats.
>>>>> >> Therefore to convert between these formats we use a data
>>>>> transformation
>>>>> >> module. It converts the input data (JSON format) to different data
>>>>> >> templates based on the chart library/ chart type selected. If chart
>>>>> library
>>>>> >> A and chart library B both use the same data template it will reuse
>>>>> the
>>>>> >> data transformation algorithm.
>>>>> >>
>>>>> >> Please share your valuable ideas on this matter. Has any
>>>>> implementation
>>>>> >> been done similar to this before?
>>>>> >>
>>>>> >> I also need to know the data flow within Paragraph.js. Please be
>>>>> kind
>>>>> >> enough to share the required resources. What is the logging
>>>>> mechanism that
>>>>> >> is being used in zeppelin?
>>>>> >>
>>>>> >> I have written few articles[1] on zeppelin
>>>>> >>
>>>>> >> [1]
>>>>> http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
>>>>> >>
>>>>> >> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
>>>>> >> madhukaudantha@gmail.com> wrote:
>>>>> >>
>>>>> >>> Hi,
>>>>> >>>
>>>>> >>> Here are some blog posts[1] on AngularJS that I wrote last week and
>>>>> >>> git-hub[2] repo on them
>>>>> >>>
>>>>> >>> [1]
>>>>> >>>
>>>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>>>>> >>> <
>>>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic
>>>>> >
>>>>> >>> [2] https://github.com/Madhuka/angularjs-tutorial
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>>>>> >>> madhukaudantha@gmail.com> wrote:
>>>>> >>>
>>>>> >>>> Hi Alex,
>>>>> >>>>
>>>>> >>>> I ran the tests and they were fine. Yes I'm familiar with the code
>>>>> >>>> layout/architecture. Basically I'm looking at AngularJS and
>>>>> Typescript. I
>>>>> >>>> have played around with those two technologies and ran some
>>>>> example code
>>>>> >>>> which worked fine. I'm writing few blog posts on them as well.
>>>>> >>>>
>>>>> >>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\
>>>>> and I'm
>>>>> >>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping
>>>>> to write
>>>>> >>>> some blog posts on it.
>>>>> >>>>
>>>>> >>>> I'm also following paragraph.js functions mainly the two
>>>>> >>>> functions pivot() and pivotDataToD3ChartFormat() for constructing
>>>>> d3 data.
>>>>> >>>>
>>>>> >>>> Further I'm finding what are the mainly used charting types by
>>>>> Research
>>>>> >>>> Teams of Universities all over the world. Also I'm researching on
>>>>> the main
>>>>> >>>> features that would be used in such graphs.
>>>>> >>>>
>>>>> >>>> Yes it will be great to have a call/meeting. This week or
>>>>> following
>>>>> >>>> will be fine for me. IRC meeting or google hangout will be fine.
>>>>> >>>>
>>>>> >>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> Hi Madhuka,
>>>>> >>>>>
>>>>> >>>>> how Zeppelin code-base exploration it going for you? Do you have
>>>>> any
>>>>> >>>>> updates?
>>>>> >>>>> I can see from
>>>>> >>>>>
>>>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>>>> >>>>> that you were able to build it well.
>>>>> >>>>>
>>>>> >>>>> Were you able to run some tests? Are you familiar by now with
>>>>> code
>>>>> >>>>> layout\client-server architecture?
>>>>> >>>>>
>>>>> >>>>> Asking because it is time for us now to rate the GSoC proposals
>>>>> that
>>>>> >>>>> we'v
>>>>> >>>>> got, so please let us know about the progress that you have made
>>>>> so
>>>>> >>>>> fas, so
>>>>> >>>>> we can use it as foundation.
>>>>> >>>>>
>>>>> >>>>> Will you be interested in making a quick call this early\next
>>>>> week and
>>>>> >>>>> also
>>>>> >>>>> talk about your GSoC proposal a bit?
>>>>> >>>>>
>>>>> >>>>> That would be awesome and could help us a lot!
>>>>> >>>>> Thanks in advance.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>>>> >>>>> madhukaudantha@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> > Hi Damien,
>>>>> >>>>> >
>>>>> >>>>> > Thanks for the quick and descriptive reply. I will go through
>>>>> each
>>>>> >>>>> of them
>>>>> >>>>> > and update you on those and I will share my experiences in my
>>>>> blog
>>>>> >>>>> too. I
>>>>> >>>>> > will go through the code-base with consideration for the above
>>>>> files.
>>>>> >>>>> >
>>>>> >>>>> > Thanks
>>>>> >>>>> >
>>>>> >>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
>>>>> >>>>> corneadoug@apache.org>
>>>>> >>>>> > wrote:
>>>>> >>>>> >
>>>>> >>>>> > > Hi,
>>>>> >>>>> > >
>>>>> >>>>> > > First of all we use angularjs so you might want to
>>>>> familiarize
>>>>> >>>>> yourself
>>>>> >>>>> > > with it.
>>>>> >>>>> > > Currently the code is not well organized (big files), and not
>>>>> >>>>> really
>>>>> >>>>> > > angular friendly either, but we are working on fixing that.
>>>>> >>>>> > >
>>>>> >>>>> > > We also were preparing a switch to Typescript, so you might
>>>>> want
>>>>> >>>>> to read
>>>>> >>>>> > > about it too.
>>>>> >>>>> > >
>>>>> >>>>> > > You will find most of the web logic inside of
>>>>> >>>>> > > zeppelin-web/app/scripts/controllers/
>>>>> >>>>> > >
>>>>> >>>>> > > Main controllers are: notebook.js and paragraph.js
>>>>> >>>>> > >
>>>>> >>>>> > > notebook.js take care of the full note, while paragraph.js
>>>>> is for
>>>>> >>>>> each
>>>>> >>>>> > > paragraph of the notebook.
>>>>> >>>>> > >
>>>>> >>>>> > > In notebook.js will have functions that distributes actions
>>>>> to all
>>>>> >>>>> > > paragraphs, interpreter binding, note style etc...
>>>>> >>>>> > > In paragraph.js you shall find everything about text-editor,
>>>>> >>>>> running
>>>>> >>>>> > > queries, pivot and visualization.
>>>>> >>>>> > >
>>>>> >>>>> > > Since the file is somewhere around 1600 lines in
>>>>> paragraph.js, I
>>>>> >>>>> can help
>>>>> >>>>> > > you find what you want with more precise questions.
>>>>> >>>>> > >
>>>>> >>>>> > > Some functions you might be interested in: 'pivot =
>>>>> function(' , '
>>>>> >>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart =
>>>>> function(',
>>>>> >>>>> '$scope.
>>>>> >>>>> > > loadTableData = function('
>>>>> >>>>> > >
>>>>> >>>>> > > This Pull request (typescript) wasn't ported to the incubator
>>>>> >>>>> repository
>>>>> >>>>> > > yet, but there was some work on splitting the paragraph into
>>>>> >>>>> smaller
>>>>> >>>>> > parts.
>>>>> >>>>> > > It might help you understand or find what you want.
>>>>> >>>>> > >
>>>>> >>>>> > >
>>>>> >>>>> > >
>>>>> >>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>>>> >>>>> > madhukaudantha@gmail.com>
>>>>> >>>>> > > wrote:
>>>>> >>>>> > >
>>>>> >>>>> > > > Hi All,
>>>>> >>>>> > > >
>>>>> >>>>> > > > Can you point out some resources  /  tutorials to
>>>>> understand the
>>>>> >>>>> code
>>>>> >>>>> > > base
>>>>> >>>>> > > > of zeppelin?
>>>>> >>>>> > > > Since I am trying to familiarize myself  with the zeppelin
>>>>> code
>>>>> >>>>> > structure
>>>>> >>>>> > > > and architecture
>>>>> >>>>> > > >
>>>>> >>>>> > > > Thanks
>>>>> >>>>> > > >
>>>>> >>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>>> >>>>> > > > madhukaudantha@gmail.com> wrote:
>>>>> >>>>> > > >
>>>>> >>>>> > > >> Hi All,
>>>>> >>>>> > > >>
>>>>> >>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This
>>>>> GSoC
>>>>> >>>>> 2015
>>>>> >>>>> > > >> project, 0COMDEV-1190 captures my interest.
>>>>> >>>>> > > >>
>>>>> >>>>> > > >> I have abundant experiences of visualization techniques
>>>>> creating
>>>>> >>>>> > > numerous
>>>>> >>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3
>>>>> charting
>>>>> >>>>> etc.
>>>>> >>>>> > > >>
>>>>> >>>>> > > >> My current research area comprises of big data where I
>>>>> have
>>>>> >>>>> worked
>>>>> >>>>> > with
>>>>> >>>>> > > >> various types of data sets. Also I'm working with cluster
>>>>> >>>>> > representation
>>>>> >>>>> > > >> and classification techniques where visualization amounts
>>>>> to a
>>>>> >>>>> > > considerable
>>>>> >>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander
>>>>> Bezzubov
>>>>> >>>>> and
>>>>> >>>>> > > CORNEAU
>>>>> >>>>> > > >> Damien for more than week.
>>>>> >>>>> > > >>
>>>>> >>>>> > > >> Thanks
>>>>> >>>>> > > >>
>>>>> >>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>>>> >>>>> > > >> [2] https://github.com/wso2/jaggery
>>>>> >>>>> > > >> --
>>>>> >>>>> > > >> Cheers,
>>>>> >>>>> > > >> Madhuka Udantha
>>>>> >>>>> > > >> http://madhukaudantha.blogspot.com
>>>>> >>>>> > > >>
>>>>> >>>>> > > >
>>>>> >>>>> > > >
>>>>> >>>>> > > >
>>>>> >>>>> > > > --
>>>>> >>>>> > > > Cheers,
>>>>> >>>>> > > > Madhuka Udantha
>>>>> >>>>> > > > http://madhukaudantha.blogspot.com
>>>>> >>>>> > > >
>>>>> >>>>> > >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > --
>>>>> >>>>> > Cheers,
>>>>> >>>>> > Madhuka Udantha
>>>>> >>>>> > http://madhukaudantha.blogspot.com
>>>>> >>>>> >
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> --
>>>>> >>>>> Kind regards,
>>>>> >>>>> Alexander.
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Cheers,
>>>>> >>>> Madhuka Udantha
>>>>> >>>> http://madhukaudantha.blogspot.com
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Cheers,
>>>>> >>> Madhuka Udantha
>>>>> >>> http://madhukaudantha.blogspot.com
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Cheers,
>>>>> >> Madhuka Udantha
>>>>> >> http://madhukaudantha.blogspot.com
>>>>> >>
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Madhuka Udantha
>>>> http://madhukaudantha.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Kind regards,
>>> Alexander.
>>>
>>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>


-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by Corneau Damien <co...@apache.org>.

Hi madhuka,

The Spring Cleaning PR will change the code base a lot, the way the code
work will be the same (for 80% of the case), so you can familiarize
yourself with it however keep in mind that the Spring Cleaning PR will
change:
* folder structure
* Separate some code into multiple files
* Refactor some code to be simpler/optimized

The folder structure part is kinda set right now (just a few tiny changes
more), and we are working on making some refactoring ready as fast as
possible.

I think that on Monday, you can get familiar with that PR new structure and
work from there, it should help you understand better all the part of the
zeppelin-web.


On Thu, May 21, 2015 at 1:46 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi All,
>
>
> I went through google charts / d3 (chart-options, chart-types, chart-data
> and  chart-format)  with  angularJS. And I have written few blog articles
> regard to the topics[1,2].
> To Familiarize myself with Zepplin I wrote angular controller and new tab
> for  Zepplin-Web app.
>
> As 'spring cleaning of Zeppelin-web'[3] is in-progress, will this change
> existing code base? I'm currently working on zepplin web app. Will the
> change be significant? Shall I familiarize with the existing code base or
> the resulting one of the 'spring cleaning of Zeppelin-web'
>
>
>
> [1] http://madhukaudantha.blogspot.com/search/label/Google%20Chart
> [2] https://github.com/Madhuka/angularjs-tutorial
> [3] https://github.com/apache/incubator-zeppelin/pull/56
>
> On Fri, May 1, 2015 at 9:40 AM, Alex B. <ab...@nflabs.com> wrote:
>
>> Congratulations, Madhuka!
>>
>> Glad that you can make it and thank you for your contributions. So we
>> will go on with the initial plan of making more visualizations for Zeppelin
>> then.
>>
>> Meanwhile it is community bonding period [1] I would suggest you go on
>> participating at the dev@ and user@ mailing lists, helping people,
>> sharing your experience with Zeppelin and meanwhile you can also try doing
>> a simple PoC as Ilya suggested, by picking two different libraries or
>> chart types (1 existing, 1 new) and writing an implementation of the
>> transformation function to fit them together.
>>
>> Please keep in touch and let us know who it goes every week or two in a
>> form of short email with your progress summary. We can also arrange
>> something like a regular call\IRC hours if you will.
>>
>> Thanks again, please feel free to reach out in case of any further
>> questions!
>>
>> [1]
>> http://googlesummerofcode.blogspot.kr/2007/04/so-what-is-this-community-bonding-all.html
>>
>> On Thu, Apr 30, 2015 at 12:57 PM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Thank you all for your participation in this thread and the support I
>>> got. My GSoC proposal for Zeppelin had been accepted. Hope you will
>>> continue to give the same support you have been giving me to see to the
>>> successful finishing of the project.
>>>
>>> Also Thanks to Corneau for spring cleaning zeppelin. I can too support
>>> this if needed. Please share any thread which will be discussing [1] so I
>>> can go through the code changes happening.
>>>
>>> Currently I'm using zeppelin/master on github. Is it alright? Do you
>>> have any ideas to what my first deliverable would need to contain?
>>>
>>> Thanks.
>>>
>>> [1] https://github.com/apache/incubator-zeppelin/pull/56
>>>
>>> On Wed, Apr 29, 2015 at 2:41 PM, Corneau Damien <co...@apache.org>
>>> wrote:
>>>
>>>> Hi Madhuka and dev mailing list,
>>>>
>>>> Since the code became quite a mess (didn't supervised it enough I guess
>>>> :p
>>>> ), and GSOC 2015 will start soon,
>>>> I'm currently working on making things easier for people who would like
>>>> to
>>>> participate in the front-end of Zeppelin.
>>>> Work in Progress in that Pull Request :
>>>> https://github.com/apache/incubator-zeppelin/pull/56
>>>>
>>>> It should be finished before the GSOC 2015 coding time start :)
>>>>
>>>> On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Madhuka,
>>>> >
>>>> > I’m not aware of any efforts in bridging different chart libraries.
>>>> But my
>>>> > guess is, we’ll have to do it ourselves. You can start by picking two
>>>> > different libraries or chart types and writing an implementation of
>>>> the
>>>> > transformation function. It might help you to assess the scope of
>>>> work.
>>>> >
>>>> > Regarding paragraph.js.
>>>> > I don’t think there is documentation for it.
>>>> > So the best way to understand the data flow it is to read the code.
>>>> > I know it’s not very helpful, but I don’t think anyone can help you in
>>>> > this situation.
>>>> >
>>>> > Regards,
>>>> > Ilya
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
>>>> > madhukaudantha@gmail.com> wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> In different charting libraries the data template take different
>>>> formats.
>>>> >> Therefore to convert between these formats we use a data
>>>> transformation
>>>> >> module. It converts the input data (JSON format) to different data
>>>> >> templates based on the chart library/ chart type selected. If chart
>>>> library
>>>> >> A and chart library B both use the same data template it will reuse
>>>> the
>>>> >> data transformation algorithm.
>>>> >>
>>>> >> Please share your valuable ideas on this matter. Has any
>>>> implementation
>>>> >> been done similar to this before?
>>>> >>
>>>> >> I also need to know the data flow within Paragraph.js. Please be kind
>>>> >> enough to share the required resources. What is the logging
>>>> mechanism that
>>>> >> is being used in zeppelin?
>>>> >>
>>>> >> I have written few articles[1] on zeppelin
>>>> >>
>>>> >> [1]
>>>> http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
>>>> >>
>>>> >> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
>>>> >> madhukaudantha@gmail.com> wrote:
>>>> >>
>>>> >>> Hi,
>>>> >>>
>>>> >>> Here are some blog posts[1] on AngularJS that I wrote last week and
>>>> >>> git-hub[2] repo on them
>>>> >>>
>>>> >>> [1]
>>>> >>>
>>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>>>> >>> <
>>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
>>>> >>> [2] https://github.com/Madhuka/angularjs-tutorial
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>>>> >>> madhukaudantha@gmail.com> wrote:
>>>> >>>
>>>> >>>> Hi Alex,
>>>> >>>>
>>>> >>>> I ran the tests and they were fine. Yes I'm familiar with the code
>>>> >>>> layout/architecture. Basically I'm looking at AngularJS and
>>>> Typescript. I
>>>> >>>> have played around with those two technologies and ran some
>>>> example code
>>>> >>>> which worked fine. I'm writing few blog posts on them as well.
>>>> >>>>
>>>> >>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\
>>>> and I'm
>>>> >>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping
>>>> to write
>>>> >>>> some blog posts on it.
>>>> >>>>
>>>> >>>> I'm also following paragraph.js functions mainly the two
>>>> >>>> functions pivot() and pivotDataToD3ChartFormat() for constructing
>>>> d3 data.
>>>> >>>>
>>>> >>>> Further I'm finding what are the mainly used charting types by
>>>> Research
>>>> >>>> Teams of Universities all over the world. Also I'm researching on
>>>> the main
>>>> >>>> features that would be used in such graphs.
>>>> >>>>
>>>> >>>> Yes it will be great to have a call/meeting. This week or following
>>>> >>>> will be fine for me. IRC meeting or google hangout will be fine.
>>>> >>>>
>>>> >>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com>
>>>> wrote:
>>>> >>>>
>>>> >>>>> Hi Madhuka,
>>>> >>>>>
>>>> >>>>> how Zeppelin code-base exploration it going for you? Do you have
>>>> any
>>>> >>>>> updates?
>>>> >>>>> I can see from
>>>> >>>>>
>>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>>> >>>>> that you were able to build it well.
>>>> >>>>>
>>>> >>>>> Were you able to run some tests? Are you familiar by now with code
>>>> >>>>> layout\client-server architecture?
>>>> >>>>>
>>>> >>>>> Asking because it is time for us now to rate the GSoC proposals
>>>> that
>>>> >>>>> we'v
>>>> >>>>> got, so please let us know about the progress that you have made
>>>> so
>>>> >>>>> fas, so
>>>> >>>>> we can use it as foundation.
>>>> >>>>>
>>>> >>>>> Will you be interested in making a quick call this early\next
>>>> week and
>>>> >>>>> also
>>>> >>>>> talk about your GSoC proposal a bit?
>>>> >>>>>
>>>> >>>>> That would be awesome and could help us a lot!
>>>> >>>>> Thanks in advance.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>>> >>>>> madhukaudantha@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>>
>>>> >>>>> > Hi Damien,
>>>> >>>>> >
>>>> >>>>> > Thanks for the quick and descriptive reply. I will go through
>>>> each
>>>> >>>>> of them
>>>> >>>>> > and update you on those and I will share my experiences in my
>>>> blog
>>>> >>>>> too. I
>>>> >>>>> > will go through the code-base with consideration for the above
>>>> files.
>>>> >>>>> >
>>>> >>>>> > Thanks
>>>> >>>>> >
>>>> >>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
>>>> >>>>> corneadoug@apache.org>
>>>> >>>>> > wrote:
>>>> >>>>> >
>>>> >>>>> > > Hi,
>>>> >>>>> > >
>>>> >>>>> > > First of all we use angularjs so you might want to familiarize
>>>> >>>>> yourself
>>>> >>>>> > > with it.
>>>> >>>>> > > Currently the code is not well organized (big files), and not
>>>> >>>>> really
>>>> >>>>> > > angular friendly either, but we are working on fixing that.
>>>> >>>>> > >
>>>> >>>>> > > We also were preparing a switch to Typescript, so you might
>>>> want
>>>> >>>>> to read
>>>> >>>>> > > about it too.
>>>> >>>>> > >
>>>> >>>>> > > You will find most of the web logic inside of
>>>> >>>>> > > zeppelin-web/app/scripts/controllers/
>>>> >>>>> > >
>>>> >>>>> > > Main controllers are: notebook.js and paragraph.js
>>>> >>>>> > >
>>>> >>>>> > > notebook.js take care of the full note, while paragraph.js is
>>>> for
>>>> >>>>> each
>>>> >>>>> > > paragraph of the notebook.
>>>> >>>>> > >
>>>> >>>>> > > In notebook.js will have functions that distributes actions
>>>> to all
>>>> >>>>> > > paragraphs, interpreter binding, note style etc...
>>>> >>>>> > > In paragraph.js you shall find everything about text-editor,
>>>> >>>>> running
>>>> >>>>> > > queries, pivot and visualization.
>>>> >>>>> > >
>>>> >>>>> > > Since the file is somewhere around 1600 lines in
>>>> paragraph.js, I
>>>> >>>>> can help
>>>> >>>>> > > you find what you want with more precise questions.
>>>> >>>>> > >
>>>> >>>>> > > Some functions you might be interested in: 'pivot =
>>>> function(' , '
>>>> >>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart =
>>>> function(',
>>>> >>>>> '$scope.
>>>> >>>>> > > loadTableData = function('
>>>> >>>>> > >
>>>> >>>>> > > This Pull request (typescript) wasn't ported to the incubator
>>>> >>>>> repository
>>>> >>>>> > > yet, but there was some work on splitting the paragraph into
>>>> >>>>> smaller
>>>> >>>>> > parts.
>>>> >>>>> > > It might help you understand or find what you want.
>>>> >>>>> > >
>>>> >>>>> > >
>>>> >>>>> > >
>>>> >>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>>> >>>>> > madhukaudantha@gmail.com>
>>>> >>>>> > > wrote:
>>>> >>>>> > >
>>>> >>>>> > > > Hi All,
>>>> >>>>> > > >
>>>> >>>>> > > > Can you point out some resources  /  tutorials to
>>>> understand the
>>>> >>>>> code
>>>> >>>>> > > base
>>>> >>>>> > > > of zeppelin?
>>>> >>>>> > > > Since I am trying to familiarize myself  with the zeppelin
>>>> code
>>>> >>>>> > structure
>>>> >>>>> > > > and architecture
>>>> >>>>> > > >
>>>> >>>>> > > > Thanks
>>>> >>>>> > > >
>>>> >>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>> >>>>> > > > madhukaudantha@gmail.com> wrote:
>>>> >>>>> > > >
>>>> >>>>> > > >> Hi All,
>>>> >>>>> > > >>
>>>> >>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This
>>>> GSoC
>>>> >>>>> 2015
>>>> >>>>> > > >> project, 0COMDEV-1190 captures my interest.
>>>> >>>>> > > >>
>>>> >>>>> > > >> I have abundant experiences of visualization techniques
>>>> creating
>>>> >>>>> > > numerous
>>>> >>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3
>>>> charting
>>>> >>>>> etc.
>>>> >>>>> > > >>
>>>> >>>>> > > >> My current research area comprises of big data where I have
>>>> >>>>> worked
>>>> >>>>> > with
>>>> >>>>> > > >> various types of data sets. Also I'm working with cluster
>>>> >>>>> > representation
>>>> >>>>> > > >> and classification techniques where visualization amounts
>>>> to a
>>>> >>>>> > > considerable
>>>> >>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander
>>>> Bezzubov
>>>> >>>>> and
>>>> >>>>> > > CORNEAU
>>>> >>>>> > > >> Damien for more than week.
>>>> >>>>> > > >>
>>>> >>>>> > > >> Thanks
>>>> >>>>> > > >>
>>>> >>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>>> >>>>> > > >> [2] https://github.com/wso2/jaggery
>>>> >>>>> > > >> --
>>>> >>>>> > > >> Cheers,
>>>> >>>>> > > >> Madhuka Udantha
>>>> >>>>> > > >> http://madhukaudantha.blogspot.com
>>>> >>>>> > > >>
>>>> >>>>> > > >
>>>> >>>>> > > >
>>>> >>>>> > > >
>>>> >>>>> > > > --
>>>> >>>>> > > > Cheers,
>>>> >>>>> > > > Madhuka Udantha
>>>> >>>>> > > > http://madhukaudantha.blogspot.com
>>>> >>>>> > > >
>>>> >>>>> > >
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> > --
>>>> >>>>> > Cheers,
>>>> >>>>> > Madhuka Udantha
>>>> >>>>> > http://madhukaudantha.blogspot.com
>>>> >>>>> >
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> --
>>>> >>>>> Kind regards,
>>>> >>>>> Alexander.
>>>> >>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Cheers,
>>>> >>>> Madhuka Udantha
>>>> >>>> http://madhukaudantha.blogspot.com
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Cheers,
>>>> >>> Madhuka Udantha
>>>> >>> http://madhukaudantha.blogspot.com
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Cheers,
>>>> >> Madhuka Udantha
>>>> >> http://madhukaudantha.blogspot.com
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> --
>> Kind regards,
>> Alexander.
>>
>>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi All,


I went through google charts / d3 (chart-options, chart-types, chart-data
and  chart-format)  with  angularJS. And I have written few blog articles
regard to the topics[1,2].
To Familiarize myself with Zepplin I wrote angular controller and new tab
for  Zepplin-Web app.

As 'spring cleaning of Zeppelin-web'[3] is in-progress, will this change
existing code base? I'm currently working on zepplin web app. Will the
change be significant? Shall I familiarize with the existing code base or
the resulting one of the 'spring cleaning of Zeppelin-web'



[1] http://madhukaudantha.blogspot.com/search/label/Google%20Chart
[2] https://github.com/Madhuka/angularjs-tutorial
[3] https://github.com/apache/incubator-zeppelin/pull/56

On Fri, May 1, 2015 at 9:40 AM, Alex B. <ab...@nflabs.com> wrote:

> Congratulations, Madhuka!
>
> Glad that you can make it and thank you for your contributions. So we will
> go on with the initial plan of making more visualizations for Zeppelin then.
>
> Meanwhile it is community bonding period [1] I would suggest you go on
> participating at the dev@ and user@ mailing lists, helping people,
> sharing your experience with Zeppelin and meanwhile you can also try doing
> a simple PoC as Ilya suggested, by picking two different libraries or
> chart types (1 existing, 1 new) and writing an implementation of the
> transformation function to fit them together.
>
> Please keep in touch and let us know who it goes every week or two in a
> form of short email with your progress summary. We can also arrange
> something like a regular call\IRC hours if you will.
>
> Thanks again, please feel free to reach out in case of any further
> questions!
>
> [1]
> http://googlesummerofcode.blogspot.kr/2007/04/so-what-is-this-community-bonding-all.html
>
> On Thu, Apr 30, 2015 at 12:57 PM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi,
>>
>> Thank you all for your participation in this thread and the support I
>> got. My GSoC proposal for Zeppelin had been accepted. Hope you will
>> continue to give the same support you have been giving me to see to the
>> successful finishing of the project.
>>
>> Also Thanks to Corneau for spring cleaning zeppelin. I can too support
>> this if needed. Please share any thread which will be discussing [1] so I
>> can go through the code changes happening.
>>
>> Currently I'm using zeppelin/master on github. Is it alright? Do you have
>> any ideas to what my first deliverable would need to contain?
>>
>> Thanks.
>>
>> [1] https://github.com/apache/incubator-zeppelin/pull/56
>>
>> On Wed, Apr 29, 2015 at 2:41 PM, Corneau Damien <co...@apache.org>
>> wrote:
>>
>>> Hi Madhuka and dev mailing list,
>>>
>>> Since the code became quite a mess (didn't supervised it enough I guess
>>> :p
>>> ), and GSOC 2015 will start soon,
>>> I'm currently working on making things easier for people who would like
>>> to
>>> participate in the front-end of Zeppelin.
>>> Work in Progress in that Pull Request :
>>> https://github.com/apache/incubator-zeppelin/pull/56
>>>
>>> It should be finished before the GSOC 2015 coding time start :)
>>>
>>> On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com>
>>> wrote:
>>>
>>> > Hi Madhuka,
>>> >
>>> > I’m not aware of any efforts in bridging different chart libraries.
>>> But my
>>> > guess is, we’ll have to do it ourselves. You can start by picking two
>>> > different libraries or chart types and writing an implementation of the
>>> > transformation function. It might help you to assess the scope of work.
>>> >
>>> > Regarding paragraph.js.
>>> > I don’t think there is documentation for it.
>>> > So the best way to understand the data flow it is to read the code.
>>> > I know it’s not very helpful, but I don’t think anyone can help you in
>>> > this situation.
>>> >
>>> > Regards,
>>> > Ilya
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
>>> > madhukaudantha@gmail.com> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> In different charting libraries the data template take different
>>> formats.
>>> >> Therefore to convert between these formats we use a data
>>> transformation
>>> >> module. It converts the input data (JSON format) to different data
>>> >> templates based on the chart library/ chart type selected. If chart
>>> library
>>> >> A and chart library B both use the same data template it will reuse
>>> the
>>> >> data transformation algorithm.
>>> >>
>>> >> Please share your valuable ideas on this matter. Has any
>>> implementation
>>> >> been done similar to this before?
>>> >>
>>> >> I also need to know the data flow within Paragraph.js. Please be kind
>>> >> enough to share the required resources. What is the logging mechanism
>>> that
>>> >> is being used in zeppelin?
>>> >>
>>> >> I have written few articles[1] on zeppelin
>>> >>
>>> >> [1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
>>> >>
>>> >> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
>>> >> madhukaudantha@gmail.com> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> Here are some blog posts[1] on AngularJS that I wrote last week and
>>> >>> git-hub[2] repo on them
>>> >>>
>>> >>> [1]
>>> >>>
>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>>> >>> <
>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
>>> >>> [2] https://github.com/Madhuka/angularjs-tutorial
>>> >>>
>>> >>>
>>> >>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>>> >>> madhukaudantha@gmail.com> wrote:
>>> >>>
>>> >>>> Hi Alex,
>>> >>>>
>>> >>>> I ran the tests and they were fine. Yes I'm familiar with the code
>>> >>>> layout/architecture. Basically I'm looking at AngularJS and
>>> Typescript. I
>>> >>>> have played around with those two technologies and ran some example
>>> code
>>> >>>> which worked fine. I'm writing few blog posts on them as well.
>>> >>>>
>>> >>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and
>>> I'm
>>> >>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping
>>> to write
>>> >>>> some blog posts on it.
>>> >>>>
>>> >>>> I'm also following paragraph.js functions mainly the two
>>> >>>> functions pivot() and pivotDataToD3ChartFormat() for constructing
>>> d3 data.
>>> >>>>
>>> >>>> Further I'm finding what are the mainly used charting types by
>>> Research
>>> >>>> Teams of Universities all over the world. Also I'm researching on
>>> the main
>>> >>>> features that would be used in such graphs.
>>> >>>>
>>> >>>> Yes it will be great to have a call/meeting. This week or following
>>> >>>> will be fine for me. IRC meeting or google hangout will be fine.
>>> >>>>
>>> >>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com>
>>> wrote:
>>> >>>>
>>> >>>>> Hi Madhuka,
>>> >>>>>
>>> >>>>> how Zeppelin code-base exploration it going for you? Do you have
>>> any
>>> >>>>> updates?
>>> >>>>> I can see from
>>> >>>>>
>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>> >>>>> that you were able to build it well.
>>> >>>>>
>>> >>>>> Were you able to run some tests? Are you familiar by now with code
>>> >>>>> layout\client-server architecture?
>>> >>>>>
>>> >>>>> Asking because it is time for us now to rate the GSoC proposals
>>> that
>>> >>>>> we'v
>>> >>>>> got, so please let us know about the progress that you have made so
>>> >>>>> fas, so
>>> >>>>> we can use it as foundation.
>>> >>>>>
>>> >>>>> Will you be interested in making a quick call this early\next week
>>> and
>>> >>>>> also
>>> >>>>> talk about your GSoC proposal a bit?
>>> >>>>>
>>> >>>>> That would be awesome and could help us a lot!
>>> >>>>> Thanks in advance.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>> >>>>> madhukaudantha@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>> > Hi Damien,
>>> >>>>> >
>>> >>>>> > Thanks for the quick and descriptive reply. I will go through
>>> each
>>> >>>>> of them
>>> >>>>> > and update you on those and I will share my experiences in my
>>> blog
>>> >>>>> too. I
>>> >>>>> > will go through the code-base with consideration for the above
>>> files.
>>> >>>>> >
>>> >>>>> > Thanks
>>> >>>>> >
>>> >>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
>>> >>>>> corneadoug@apache.org>
>>> >>>>> > wrote:
>>> >>>>> >
>>> >>>>> > > Hi,
>>> >>>>> > >
>>> >>>>> > > First of all we use angularjs so you might want to familiarize
>>> >>>>> yourself
>>> >>>>> > > with it.
>>> >>>>> > > Currently the code is not well organized (big files), and not
>>> >>>>> really
>>> >>>>> > > angular friendly either, but we are working on fixing that.
>>> >>>>> > >
>>> >>>>> > > We also were preparing a switch to Typescript, so you might
>>> want
>>> >>>>> to read
>>> >>>>> > > about it too.
>>> >>>>> > >
>>> >>>>> > > You will find most of the web logic inside of
>>> >>>>> > > zeppelin-web/app/scripts/controllers/
>>> >>>>> > >
>>> >>>>> > > Main controllers are: notebook.js and paragraph.js
>>> >>>>> > >
>>> >>>>> > > notebook.js take care of the full note, while paragraph.js is
>>> for
>>> >>>>> each
>>> >>>>> > > paragraph of the notebook.
>>> >>>>> > >
>>> >>>>> > > In notebook.js will have functions that distributes actions to
>>> all
>>> >>>>> > > paragraphs, interpreter binding, note style etc...
>>> >>>>> > > In paragraph.js you shall find everything about text-editor,
>>> >>>>> running
>>> >>>>> > > queries, pivot and visualization.
>>> >>>>> > >
>>> >>>>> > > Since the file is somewhere around 1600 lines in paragraph.js,
>>> I
>>> >>>>> can help
>>> >>>>> > > you find what you want with more precise questions.
>>> >>>>> > >
>>> >>>>> > > Some functions you might be interested in: 'pivot = function('
>>> , '
>>> >>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart =
>>> function(',
>>> >>>>> '$scope.
>>> >>>>> > > loadTableData = function('
>>> >>>>> > >
>>> >>>>> > > This Pull request (typescript) wasn't ported to the incubator
>>> >>>>> repository
>>> >>>>> > > yet, but there was some work on splitting the paragraph into
>>> >>>>> smaller
>>> >>>>> > parts.
>>> >>>>> > > It might help you understand or find what you want.
>>> >>>>> > >
>>> >>>>> > >
>>> >>>>> > >
>>> >>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>> >>>>> > madhukaudantha@gmail.com>
>>> >>>>> > > wrote:
>>> >>>>> > >
>>> >>>>> > > > Hi All,
>>> >>>>> > > >
>>> >>>>> > > > Can you point out some resources  /  tutorials to understand
>>> the
>>> >>>>> code
>>> >>>>> > > base
>>> >>>>> > > > of zeppelin?
>>> >>>>> > > > Since I am trying to familiarize myself  with the zeppelin
>>> code
>>> >>>>> > structure
>>> >>>>> > > > and architecture
>>> >>>>> > > >
>>> >>>>> > > > Thanks
>>> >>>>> > > >
>>> >>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>> >>>>> > > > madhukaudantha@gmail.com> wrote:
>>> >>>>> > > >
>>> >>>>> > > >> Hi All,
>>> >>>>> > > >>
>>> >>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This
>>> GSoC
>>> >>>>> 2015
>>> >>>>> > > >> project, 0COMDEV-1190 captures my interest.
>>> >>>>> > > >>
>>> >>>>> > > >> I have abundant experiences of visualization techniques
>>> creating
>>> >>>>> > > numerous
>>> >>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3
>>> charting
>>> >>>>> etc.
>>> >>>>> > > >>
>>> >>>>> > > >> My current research area comprises of big data where I have
>>> >>>>> worked
>>> >>>>> > with
>>> >>>>> > > >> various types of data sets. Also I'm working with cluster
>>> >>>>> > representation
>>> >>>>> > > >> and classification techniques where visualization amounts
>>> to a
>>> >>>>> > > considerable
>>> >>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander
>>> Bezzubov
>>> >>>>> and
>>> >>>>> > > CORNEAU
>>> >>>>> > > >> Damien for more than week.
>>> >>>>> > > >>
>>> >>>>> > > >> Thanks
>>> >>>>> > > >>
>>> >>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>> >>>>> > > >> [2] https://github.com/wso2/jaggery
>>> >>>>> > > >> --
>>> >>>>> > > >> Cheers,
>>> >>>>> > > >> Madhuka Udantha
>>> >>>>> > > >> http://madhukaudantha.blogspot.com
>>> >>>>> > > >>
>>> >>>>> > > >
>>> >>>>> > > >
>>> >>>>> > > >
>>> >>>>> > > > --
>>> >>>>> > > > Cheers,
>>> >>>>> > > > Madhuka Udantha
>>> >>>>> > > > http://madhukaudantha.blogspot.com
>>> >>>>> > > >
>>> >>>>> > >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > --
>>> >>>>> > Cheers,
>>> >>>>> > Madhuka Udantha
>>> >>>>> > http://madhukaudantha.blogspot.com
>>> >>>>> >
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> --
>>> >>>>> Kind regards,
>>> >>>>> Alexander.
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Cheers,
>>> >>>> Madhuka Udantha
>>> >>>> http://madhukaudantha.blogspot.com
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Cheers,
>>> >>> Madhuka Udantha
>>> >>> http://madhukaudantha.blogspot.com
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Cheers,
>>> >> Madhuka Udantha
>>> >> http://madhukaudantha.blogspot.com
>>> >>
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>


-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Thanks Alex

On Fri, May 1, 2015 at 9:40 AM, Alex B. <ab...@nflabs.com> wrote:

> Congratulations, Madhuka!
>
> Glad that you can make it and thank you for your contributions. So we will
> go on with the initial plan of making more visualizations for Zeppelin then.
>
> Meanwhile it is community bonding period [1] I would suggest you go on
> participating at the dev@ and user@ mailing lists, helping people,
> sharing your experience with Zeppelin
>
Yes sure. As it is community bonding period I will participate on them. I
will share my experiences in ML and my blog.

and meanwhile you can also try doing a simple PoC as Ilya suggested, by
> picking two different libraries or chart types (1 existing, 1 new) and
> writing an implementation of the transformation function to fit them
> together.
>
Yes, I will do the simple POC as llya suggested and I think it will be good
starting point for my task.


>
> Please keep in touch and let us know who it goes every week or two in a
> form of short email with your progress summary. We can also arrange
> something like a regular call\IRC hours if you will.
>
I will update my progress in ML and even call\IRC

>
> Thanks again, please feel free to reach out in case of any further
> questions!
>
> [1]
> http://googlesummerofcode.blogspot.kr/2007/04/so-what-is-this-community-bonding-all.html
>
> On Thu, Apr 30, 2015 at 12:57 PM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi,
>>
>> Thank you all for your participation in this thread and the support I
>> got. My GSoC proposal for Zeppelin had been accepted. Hope you will
>> continue to give the same support you have been giving me to see to the
>> successful finishing of the project.
>>
>> Also Thanks to Corneau for spring cleaning zeppelin. I can too support
>> this if needed. Please share any thread which will be discussing [1] so I
>> can go through the code changes happening.
>>
>> Currently I'm using zeppelin/master on github. Is it alright? Do you have
>> any ideas to what my first deliverable would need to contain?
>>
>> Thanks.
>>
>> [1] https://github.com/apache/incubator-zeppelin/pull/56
>>
>> On Wed, Apr 29, 2015 at 2:41 PM, Corneau Damien <co...@apache.org>
>> wrote:
>>
>>> Hi Madhuka and dev mailing list,
>>>
>>> Since the code became quite a mess (didn't supervised it enough I guess
>>> :p
>>> ), and GSOC 2015 will start soon,
>>> I'm currently working on making things easier for people who would like
>>> to
>>> participate in the front-end of Zeppelin.
>>> Work in Progress in that Pull Request :
>>> https://github.com/apache/incubator-zeppelin/pull/56
>>>
>>> It should be finished before the GSOC 2015 coding time start :)
>>>
>>> On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com>
>>> wrote:
>>>
>>> > Hi Madhuka,
>>> >
>>> > I’m not aware of any efforts in bridging different chart libraries.
>>> But my
>>> > guess is, we’ll have to do it ourselves. You can start by picking two
>>> > different libraries or chart types and writing an implementation of the
>>> > transformation function. It might help you to assess the scope of work.
>>> >
>>> > Regarding paragraph.js.
>>> > I don’t think there is documentation for it.
>>> > So the best way to understand the data flow it is to read the code.
>>> > I know it’s not very helpful, but I don’t think anyone can help you in
>>> > this situation.
>>> >
>>> > Regards,
>>> > Ilya
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
>>> > madhukaudantha@gmail.com> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> In different charting libraries the data template take different
>>> formats.
>>> >> Therefore to convert between these formats we use a data
>>> transformation
>>> >> module. It converts the input data (JSON format) to different data
>>> >> templates based on the chart library/ chart type selected. If chart
>>> library
>>> >> A and chart library B both use the same data template it will reuse
>>> the
>>> >> data transformation algorithm.
>>> >>
>>> >> Please share your valuable ideas on this matter. Has any
>>> implementation
>>> >> been done similar to this before?
>>> >>
>>> >> I also need to know the data flow within Paragraph.js. Please be kind
>>> >> enough to share the required resources. What is the logging mechanism
>>> that
>>> >> is being used in zeppelin?
>>> >>
>>> >> I have written few articles[1] on zeppelin
>>> >>
>>> >> [1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
>>> >>
>>> >> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
>>> >> madhukaudantha@gmail.com> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> Here are some blog posts[1] on AngularJS that I wrote last week and
>>> >>> git-hub[2] repo on them
>>> >>>
>>> >>> [1]
>>> >>>
>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>>> >>> <
>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
>>> >>> [2] https://github.com/Madhuka/angularjs-tutorial
>>> >>>
>>> >>>
>>> >>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>>> >>> madhukaudantha@gmail.com> wrote:
>>> >>>
>>> >>>> Hi Alex,
>>> >>>>
>>> >>>> I ran the tests and they were fine. Yes I'm familiar with the code
>>> >>>> layout/architecture. Basically I'm looking at AngularJS and
>>> Typescript. I
>>> >>>> have played around with those two technologies and ran some example
>>> code
>>> >>>> which worked fine. I'm writing few blog posts on them as well.
>>> >>>>
>>> >>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and
>>> I'm
>>> >>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping
>>> to write
>>> >>>> some blog posts on it.
>>> >>>>
>>> >>>> I'm also following paragraph.js functions mainly the two
>>> >>>> functions pivot() and pivotDataToD3ChartFormat() for constructing
>>> d3 data.
>>> >>>>
>>> >>>> Further I'm finding what are the mainly used charting types by
>>> Research
>>> >>>> Teams of Universities all over the world. Also I'm researching on
>>> the main
>>> >>>> features that would be used in such graphs.
>>> >>>>
>>> >>>> Yes it will be great to have a call/meeting. This week or following
>>> >>>> will be fine for me. IRC meeting or google hangout will be fine.
>>> >>>>
>>> >>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com>
>>> wrote:
>>> >>>>
>>> >>>>> Hi Madhuka,
>>> >>>>>
>>> >>>>> how Zeppelin code-base exploration it going for you? Do you have
>>> any
>>> >>>>> updates?
>>> >>>>> I can see from
>>> >>>>>
>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>> >>>>> that you were able to build it well.
>>> >>>>>
>>> >>>>> Were you able to run some tests? Are you familiar by now with code
>>> >>>>> layout\client-server architecture?
>>> >>>>>
>>> >>>>> Asking because it is time for us now to rate the GSoC proposals
>>> that
>>> >>>>> we'v
>>> >>>>> got, so please let us know about the progress that you have made so
>>> >>>>> fas, so
>>> >>>>> we can use it as foundation.
>>> >>>>>
>>> >>>>> Will you be interested in making a quick call this early\next week
>>> and
>>> >>>>> also
>>> >>>>> talk about your GSoC proposal a bit?
>>> >>>>>
>>> >>>>> That would be awesome and could help us a lot!
>>> >>>>> Thanks in advance.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>> >>>>> madhukaudantha@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>> > Hi Damien,
>>> >>>>> >
>>> >>>>> > Thanks for the quick and descriptive reply. I will go through
>>> each
>>> >>>>> of them
>>> >>>>> > and update you on those and I will share my experiences in my
>>> blog
>>> >>>>> too. I
>>> >>>>> > will go through the code-base with consideration for the above
>>> files.
>>> >>>>> >
>>> >>>>> > Thanks
>>> >>>>> >
>>> >>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
>>> >>>>> corneadoug@apache.org>
>>> >>>>> > wrote:
>>> >>>>> >
>>> >>>>> > > Hi,
>>> >>>>> > >
>>> >>>>> > > First of all we use angularjs so you might want to familiarize
>>> >>>>> yourself
>>> >>>>> > > with it.
>>> >>>>> > > Currently the code is not well organized (big files), and not
>>> >>>>> really
>>> >>>>> > > angular friendly either, but we are working on fixing that.
>>> >>>>> > >
>>> >>>>> > > We also were preparing a switch to Typescript, so you might
>>> want
>>> >>>>> to read
>>> >>>>> > > about it too.
>>> >>>>> > >
>>> >>>>> > > You will find most of the web logic inside of
>>> >>>>> > > zeppelin-web/app/scripts/controllers/
>>> >>>>> > >
>>> >>>>> > > Main controllers are: notebook.js and paragraph.js
>>> >>>>> > >
>>> >>>>> > > notebook.js take care of the full note, while paragraph.js is
>>> for
>>> >>>>> each
>>> >>>>> > > paragraph of the notebook.
>>> >>>>> > >
>>> >>>>> > > In notebook.js will have functions that distributes actions to
>>> all
>>> >>>>> > > paragraphs, interpreter binding, note style etc...
>>> >>>>> > > In paragraph.js you shall find everything about text-editor,
>>> >>>>> running
>>> >>>>> > > queries, pivot and visualization.
>>> >>>>> > >
>>> >>>>> > > Since the file is somewhere around 1600 lines in paragraph.js,
>>> I
>>> >>>>> can help
>>> >>>>> > > you find what you want with more precise questions.
>>> >>>>> > >
>>> >>>>> > > Some functions you might be interested in: 'pivot = function('
>>> , '
>>> >>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart =
>>> function(',
>>> >>>>> '$scope.
>>> >>>>> > > loadTableData = function('
>>> >>>>> > >
>>> >>>>> > > This Pull request (typescript) wasn't ported to the incubator
>>> >>>>> repository
>>> >>>>> > > yet, but there was some work on splitting the paragraph into
>>> >>>>> smaller
>>> >>>>> > parts.
>>> >>>>> > > It might help you understand or find what you want.
>>> >>>>> > >
>>> >>>>> > >
>>> >>>>> > >
>>> >>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>> >>>>> > madhukaudantha@gmail.com>
>>> >>>>> > > wrote:
>>> >>>>> > >
>>> >>>>> > > > Hi All,
>>> >>>>> > > >
>>> >>>>> > > > Can you point out some resources  /  tutorials to understand
>>> the
>>> >>>>> code
>>> >>>>> > > base
>>> >>>>> > > > of zeppelin?
>>> >>>>> > > > Since I am trying to familiarize myself  with the zeppelin
>>> code
>>> >>>>> > structure
>>> >>>>> > > > and architecture
>>> >>>>> > > >
>>> >>>>> > > > Thanks
>>> >>>>> > > >
>>> >>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>> >>>>> > > > madhukaudantha@gmail.com> wrote:
>>> >>>>> > > >
>>> >>>>> > > >> Hi All,
>>> >>>>> > > >>
>>> >>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This
>>> GSoC
>>> >>>>> 2015
>>> >>>>> > > >> project, 0COMDEV-1190 captures my interest.
>>> >>>>> > > >>
>>> >>>>> > > >> I have abundant experiences of visualization techniques
>>> creating
>>> >>>>> > > numerous
>>> >>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3
>>> charting
>>> >>>>> etc.
>>> >>>>> > > >>
>>> >>>>> > > >> My current research area comprises of big data where I have
>>> >>>>> worked
>>> >>>>> > with
>>> >>>>> > > >> various types of data sets. Also I'm working with cluster
>>> >>>>> > representation
>>> >>>>> > > >> and classification techniques where visualization amounts
>>> to a
>>> >>>>> > > considerable
>>> >>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander
>>> Bezzubov
>>> >>>>> and
>>> >>>>> > > CORNEAU
>>> >>>>> > > >> Damien for more than week.
>>> >>>>> > > >>
>>> >>>>> > > >> Thanks
>>> >>>>> > > >>
>>> >>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>> >>>>> > > >> [2] https://github.com/wso2/jaggery
>>> >>>>> > > >> --
>>> >>>>> > > >> Cheers,
>>> >>>>> > > >> Madhuka Udantha
>>> >>>>> > > >> http://madhukaudantha.blogspot.com
>>> >>>>> > > >>
>>> >>>>> > > >
>>> >>>>> > > >
>>> >>>>> > > >
>>> >>>>> > > > --
>>> >>>>> > > > Cheers,
>>> >>>>> > > > Madhuka Udantha
>>> >>>>> > > > http://madhukaudantha.blogspot.com
>>> >>>>> > > >
>>> >>>>> > >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > --
>>> >>>>> > Cheers,
>>> >>>>> > Madhuka Udantha
>>> >>>>> > http://madhukaudantha.blogspot.com
>>> >>>>> >
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> --
>>> >>>>> Kind regards,
>>> >>>>> Alexander.
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Cheers,
>>> >>>> Madhuka Udantha
>>> >>>> http://madhukaudantha.blogspot.com
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Cheers,
>>> >>> Madhuka Udantha
>>> >>> http://madhukaudantha.blogspot.com
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Cheers,
>>> >> Madhuka Udantha
>>> >> http://madhukaudantha.blogspot.com
>>> >>
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>


-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by "Alex B." <ab...@nflabs.com>.

Congratulations, Madhuka!

Glad that you can make it and thank you for your contributions. So we will
go on with the initial plan of making more visualizations for Zeppelin then.

Meanwhile it is community bonding period [1] I would suggest you go on
participating at the dev@ and user@ mailing lists, helping people, sharing
your experience with Zeppelin and meanwhile you can also try doing a simple
PoC as Ilya suggested, by picking two different libraries or chart types
(1 existing, 1 new) and writing an implementation of the transformation
function to fit them together.

Please keep in touch and let us know who it goes every week or two in a
form of short email with your progress summary. We can also arrange
something like a regular call\IRC hours if you will.

Thanks again, please feel free to reach out in case of any further
questions!

[1]
http://googlesummerofcode.blogspot.kr/2007/04/so-what-is-this-community-bonding-all.html

On Thu, Apr 30, 2015 at 12:57 PM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi,
>
> Thank you all for your participation in this thread and the support I got.
> My GSoC proposal for Zeppelin had been accepted. Hope you will continue to
> give the same support you have been giving me to see to the successful
> finishing of the project.
>
> Also Thanks to Corneau for spring cleaning zeppelin. I can too support
> this if needed. Please share any thread which will be discussing [1] so I
> can go through the code changes happening.
>
> Currently I'm using zeppelin/master on github. Is it alright? Do you have
> any ideas to what my first deliverable would need to contain?
>
> Thanks.
>
> [1] https://github.com/apache/incubator-zeppelin/pull/56
>
> On Wed, Apr 29, 2015 at 2:41 PM, Corneau Damien <co...@apache.org>
> wrote:
>
>> Hi Madhuka and dev mailing list,
>>
>> Since the code became quite a mess (didn't supervised it enough I guess :p
>> ), and GSOC 2015 will start soon,
>> I'm currently working on making things easier for people who would like to
>> participate in the front-end of Zeppelin.
>> Work in Progress in that Pull Request :
>> https://github.com/apache/incubator-zeppelin/pull/56
>>
>> It should be finished before the GSOC 2015 coding time start :)
>>
>> On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com>
>> wrote:
>>
>> > Hi Madhuka,
>> >
>> > I’m not aware of any efforts in bridging different chart libraries. But
>> my
>> > guess is, we’ll have to do it ourselves. You can start by picking two
>> > different libraries or chart types and writing an implementation of the
>> > transformation function. It might help you to assess the scope of work.
>> >
>> > Regarding paragraph.js.
>> > I don’t think there is documentation for it.
>> > So the best way to understand the data flow it is to read the code.
>> > I know it’s not very helpful, but I don’t think anyone can help you in
>> > this situation.
>> >
>> > Regards,
>> > Ilya
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
>> > madhukaudantha@gmail.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> In different charting libraries the data template take different
>> formats.
>> >> Therefore to convert between these formats we use a data transformation
>> >> module. It converts the input data (JSON format) to different data
>> >> templates based on the chart library/ chart type selected. If chart
>> library
>> >> A and chart library B both use the same data template it will reuse the
>> >> data transformation algorithm.
>> >>
>> >> Please share your valuable ideas on this matter. Has any implementation
>> >> been done similar to this before?
>> >>
>> >> I also need to know the data flow within Paragraph.js. Please be kind
>> >> enough to share the required resources. What is the logging mechanism
>> that
>> >> is being used in zeppelin?
>> >>
>> >> I have written few articles[1] on zeppelin
>> >>
>> >> [1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
>> >>
>> >> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
>> >> madhukaudantha@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Here are some blog posts[1] on AngularJS that I wrote last week and
>> >>> git-hub[2] repo on them
>> >>>
>> >>> [1]
>> >>>
>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>> >>> <
>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
>> >>> [2] https://github.com/Madhuka/angularjs-tutorial
>> >>>
>> >>>
>> >>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>> >>> madhukaudantha@gmail.com> wrote:
>> >>>
>> >>>> Hi Alex,
>> >>>>
>> >>>> I ran the tests and they were fine. Yes I'm familiar with the code
>> >>>> layout/architecture. Basically I'm looking at AngularJS and
>> Typescript. I
>> >>>> have played around with those two technologies and ran some example
>> code
>> >>>> which worked fine. I'm writing few blog posts on them as well.
>> >>>>
>> >>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and
>> I'm
>> >>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping to
>> write
>> >>>> some blog posts on it.
>> >>>>
>> >>>> I'm also following paragraph.js functions mainly the two
>> >>>> functions pivot() and pivotDataToD3ChartFormat() for constructing d3
>> data.
>> >>>>
>> >>>> Further I'm finding what are the mainly used charting types by
>> Research
>> >>>> Teams of Universities all over the world. Also I'm researching on
>> the main
>> >>>> features that would be used in such graphs.
>> >>>>
>> >>>> Yes it will be great to have a call/meeting. This week or following
>> >>>> will be fine for me. IRC meeting or google hangout will be fine.
>> >>>>
>> >>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com>
>> wrote:
>> >>>>
>> >>>>> Hi Madhuka,
>> >>>>>
>> >>>>> how Zeppelin code-base exploration it going for you? Do you have any
>> >>>>> updates?
>> >>>>> I can see from
>> >>>>>
>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>> >>>>> that you were able to build it well.
>> >>>>>
>> >>>>> Were you able to run some tests? Are you familiar by now with code
>> >>>>> layout\client-server architecture?
>> >>>>>
>> >>>>> Asking because it is time for us now to rate the GSoC proposals that
>> >>>>> we'v
>> >>>>> got, so please let us know about the progress that you have made so
>> >>>>> fas, so
>> >>>>> we can use it as foundation.
>> >>>>>
>> >>>>> Will you be interested in making a quick call this early\next week
>> and
>> >>>>> also
>> >>>>> talk about your GSoC proposal a bit?
>> >>>>>
>> >>>>> That would be awesome and could help us a lot!
>> >>>>> Thanks in advance.
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>> >>>>> madhukaudantha@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> > Hi Damien,
>> >>>>> >
>> >>>>> > Thanks for the quick and descriptive reply. I will go through each
>> >>>>> of them
>> >>>>> > and update you on those and I will share my experiences in my blog
>> >>>>> too. I
>> >>>>> > will go through the code-base with consideration for the above
>> files.
>> >>>>> >
>> >>>>> > Thanks
>> >>>>> >
>> >>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
>> >>>>> corneadoug@apache.org>
>> >>>>> > wrote:
>> >>>>> >
>> >>>>> > > Hi,
>> >>>>> > >
>> >>>>> > > First of all we use angularjs so you might want to familiarize
>> >>>>> yourself
>> >>>>> > > with it.
>> >>>>> > > Currently the code is not well organized (big files), and not
>> >>>>> really
>> >>>>> > > angular friendly either, but we are working on fixing that.
>> >>>>> > >
>> >>>>> > > We also were preparing a switch to Typescript, so you might want
>> >>>>> to read
>> >>>>> > > about it too.
>> >>>>> > >
>> >>>>> > > You will find most of the web logic inside of
>> >>>>> > > zeppelin-web/app/scripts/controllers/
>> >>>>> > >
>> >>>>> > > Main controllers are: notebook.js and paragraph.js
>> >>>>> > >
>> >>>>> > > notebook.js take care of the full note, while paragraph.js is
>> for
>> >>>>> each
>> >>>>> > > paragraph of the notebook.
>> >>>>> > >
>> >>>>> > > In notebook.js will have functions that distributes actions to
>> all
>> >>>>> > > paragraphs, interpreter binding, note style etc...
>> >>>>> > > In paragraph.js you shall find everything about text-editor,
>> >>>>> running
>> >>>>> > > queries, pivot and visualization.
>> >>>>> > >
>> >>>>> > > Since the file is somewhere around 1600 lines in paragraph.js, I
>> >>>>> can help
>> >>>>> > > you find what you want with more precise questions.
>> >>>>> > >
>> >>>>> > > Some functions you might be interested in: 'pivot = function('
>> , '
>> >>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
>> >>>>> '$scope.
>> >>>>> > > loadTableData = function('
>> >>>>> > >
>> >>>>> > > This Pull request (typescript) wasn't ported to the incubator
>> >>>>> repository
>> >>>>> > > yet, but there was some work on splitting the paragraph into
>> >>>>> smaller
>> >>>>> > parts.
>> >>>>> > > It might help you understand or find what you want.
>> >>>>> > >
>> >>>>> > >
>> >>>>> > >
>> >>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>> >>>>> > madhukaudantha@gmail.com>
>> >>>>> > > wrote:
>> >>>>> > >
>> >>>>> > > > Hi All,
>> >>>>> > > >
>> >>>>> > > > Can you point out some resources  /  tutorials to understand
>> the
>> >>>>> code
>> >>>>> > > base
>> >>>>> > > > of zeppelin?
>> >>>>> > > > Since I am trying to familiarize myself  with the zeppelin
>> code
>> >>>>> > structure
>> >>>>> > > > and architecture
>> >>>>> > > >
>> >>>>> > > > Thanks
>> >>>>> > > >
>> >>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>> >>>>> > > > madhukaudantha@gmail.com> wrote:
>> >>>>> > > >
>> >>>>> > > >> Hi All,
>> >>>>> > > >>
>> >>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This
>> GSoC
>> >>>>> 2015
>> >>>>> > > >> project, 0COMDEV-1190 captures my interest.
>> >>>>> > > >>
>> >>>>> > > >> I have abundant experiences of visualization techniques
>> creating
>> >>>>> > > numerous
>> >>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3
>> charting
>> >>>>> etc.
>> >>>>> > > >>
>> >>>>> > > >> My current research area comprises of big data where I have
>> >>>>> worked
>> >>>>> > with
>> >>>>> > > >> various types of data sets. Also I'm working with cluster
>> >>>>> > representation
>> >>>>> > > >> and classification techniques where visualization amounts to
>> a
>> >>>>> > > considerable
>> >>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander
>> Bezzubov
>> >>>>> and
>> >>>>> > > CORNEAU
>> >>>>> > > >> Damien for more than week.
>> >>>>> > > >>
>> >>>>> > > >> Thanks
>> >>>>> > > >>
>> >>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>> >>>>> > > >> [2] https://github.com/wso2/jaggery
>> >>>>> > > >> --
>> >>>>> > > >> Cheers,
>> >>>>> > > >> Madhuka Udantha
>> >>>>> > > >> http://madhukaudantha.blogspot.com
>> >>>>> > > >>
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> > > > --
>> >>>>> > > > Cheers,
>> >>>>> > > > Madhuka Udantha
>> >>>>> > > > http://madhukaudantha.blogspot.com
>> >>>>> > > >
>> >>>>> > >
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > Cheers,
>> >>>>> > Madhuka Udantha
>> >>>>> > http://madhukaudantha.blogspot.com
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> --
>> >>>>> Kind regards,
>> >>>>> Alexander.
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Cheers,
>> >>>> Madhuka Udantha
>> >>>> http://madhukaudantha.blogspot.com
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Cheers,
>> >>> Madhuka Udantha
>> >>> http://madhukaudantha.blogspot.com
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Cheers,
>> >> Madhuka Udantha
>> >> http://madhukaudantha.blogspot.com
>> >>
>> >
>> >
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
--
Kind regards,
Alexander.

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

Thank you all for your participation in this thread and the support I got.
My GSoC proposal for Zeppelin had been accepted. Hope you will continue to
give the same support you have been giving me to see to the successful
finishing of the project.

Also Thanks to Corneau for spring cleaning zeppelin. I can too support this
if needed. Please share any thread which will be discussing [1] so I can go
through the code changes happening.

Currently I'm using zeppelin/master on github. Is it alright? Do you have
any ideas to what my first deliverable would need to contain?

Thanks.

[1] https://github.com/apache/incubator-zeppelin/pull/56

On Wed, Apr 29, 2015 at 2:41 PM, Corneau Damien <co...@apache.org>
wrote:

> Hi Madhuka and dev mailing list,
>
> Since the code became quite a mess (didn't supervised it enough I guess :p
> ), and GSOC 2015 will start soon,
> I'm currently working on making things easier for people who would like to
> participate in the front-end of Zeppelin.
> Work in Progress in that Pull Request :
> https://github.com/apache/incubator-zeppelin/pull/56
>
> It should be finished before the GSOC 2015 coding time start :)
>
> On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com>
> wrote:
>
> > Hi Madhuka,
> >
> > I’m not aware of any efforts in bridging different chart libraries. But
> my
> > guess is, we’ll have to do it ourselves. You can start by picking two
> > different libraries or chart types and writing an implementation of the
> > transformation function. It might help you to assess the scope of work.
> >
> > Regarding paragraph.js.
> > I don’t think there is documentation for it.
> > So the best way to understand the data flow it is to read the code.
> > I know it’s not very helpful, but I don’t think anyone can help you in
> > this situation.
> >
> > Regards,
> > Ilya
> >
> >
> >
> >
> >
> >
> > On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
> > madhukaudantha@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> In different charting libraries the data template take different
> formats.
> >> Therefore to convert between these formats we use a data transformation
> >> module. It converts the input data (JSON format) to different data
> >> templates based on the chart library/ chart type selected. If chart
> library
> >> A and chart library B both use the same data template it will reuse the
> >> data transformation algorithm.
> >>
> >> Please share your valuable ideas on this matter. Has any implementation
> >> been done similar to this before?
> >>
> >> I also need to know the data flow within Paragraph.js. Please be kind
> >> enough to share the required resources. What is the logging mechanism
> that
> >> is being used in zeppelin?
> >>
> >> I have written few articles[1] on zeppelin
> >>
> >> [1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
> >>
> >> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
> >> madhukaudantha@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Here are some blog posts[1] on AngularJS that I wrote last week and
> >>> git-hub[2] repo on them
> >>>
> >>> [1]
> >>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
> >>> <
> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
> >>> [2] https://github.com/Madhuka/angularjs-tutorial
> >>>
> >>>
> >>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
> >>> madhukaudantha@gmail.com> wrote:
> >>>
> >>>> Hi Alex,
> >>>>
> >>>> I ran the tests and they were fine. Yes I'm familiar with the code
> >>>> layout/architecture. Basically I'm looking at AngularJS and
> Typescript. I
> >>>> have played around with those two technologies and ran some example
> code
> >>>> which worked fine. I'm writing few blog posts on them as well.
> >>>>
> >>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and
> I'm
> >>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping to
> write
> >>>> some blog posts on it.
> >>>>
> >>>> I'm also following paragraph.js functions mainly the two
> >>>> functions pivot() and pivotDataToD3ChartFormat() for constructing d3
> data.
> >>>>
> >>>> Further I'm finding what are the mainly used charting types by
> Research
> >>>> Teams of Universities all over the world. Also I'm researching on the
> main
> >>>> features that would be used in such graphs.
> >>>>
> >>>> Yes it will be great to have a call/meeting. This week or following
> >>>> will be fine for me. IRC meeting or google hangout will be fine.
> >>>>
> >>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com>
> wrote:
> >>>>
> >>>>> Hi Madhuka,
> >>>>>
> >>>>> how Zeppelin code-base exploration it going for you? Do you have any
> >>>>> updates?
> >>>>> I can see from
> >>>>>
> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
> >>>>> that you were able to build it well.
> >>>>>
> >>>>> Were you able to run some tests? Are you familiar by now with code
> >>>>> layout\client-server architecture?
> >>>>>
> >>>>> Asking because it is time for us now to rate the GSoC proposals that
> >>>>> we'v
> >>>>> got, so please let us know about the progress that you have made so
> >>>>> fas, so
> >>>>> we can use it as foundation.
> >>>>>
> >>>>> Will you be interested in making a quick call this early\next week
> and
> >>>>> also
> >>>>> talk about your GSoC proposal a bit?
> >>>>>
> >>>>> That would be awesome and could help us a lot!
> >>>>> Thanks in advance.
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
> >>>>> madhukaudantha@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> > Hi Damien,
> >>>>> >
> >>>>> > Thanks for the quick and descriptive reply. I will go through each
> >>>>> of them
> >>>>> > and update you on those and I will share my experiences in my blog
> >>>>> too. I
> >>>>> > will go through the code-base with consideration for the above
> files.
> >>>>> >
> >>>>> > Thanks
> >>>>> >
> >>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
> >>>>> corneadoug@apache.org>
> >>>>> > wrote:
> >>>>> >
> >>>>> > > Hi,
> >>>>> > >
> >>>>> > > First of all we use angularjs so you might want to familiarize
> >>>>> yourself
> >>>>> > > with it.
> >>>>> > > Currently the code is not well organized (big files), and not
> >>>>> really
> >>>>> > > angular friendly either, but we are working on fixing that.
> >>>>> > >
> >>>>> > > We also were preparing a switch to Typescript, so you might want
> >>>>> to read
> >>>>> > > about it too.
> >>>>> > >
> >>>>> > > You will find most of the web logic inside of
> >>>>> > > zeppelin-web/app/scripts/controllers/
> >>>>> > >
> >>>>> > > Main controllers are: notebook.js and paragraph.js
> >>>>> > >
> >>>>> > > notebook.js take care of the full note, while paragraph.js is for
> >>>>> each
> >>>>> > > paragraph of the notebook.
> >>>>> > >
> >>>>> > > In notebook.js will have functions that distributes actions to
> all
> >>>>> > > paragraphs, interpreter binding, note style etc...
> >>>>> > > In paragraph.js you shall find everything about text-editor,
> >>>>> running
> >>>>> > > queries, pivot and visualization.
> >>>>> > >
> >>>>> > > Since the file is somewhere around 1600 lines in paragraph.js, I
> >>>>> can help
> >>>>> > > you find what you want with more precise questions.
> >>>>> > >
> >>>>> > > Some functions you might be interested in: 'pivot = function(' ,
> '
> >>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
> >>>>> '$scope.
> >>>>> > > loadTableData = function('
> >>>>> > >
> >>>>> > > This Pull request (typescript) wasn't ported to the incubator
> >>>>> repository
> >>>>> > > yet, but there was some work on splitting the paragraph into
> >>>>> smaller
> >>>>> > parts.
> >>>>> > > It might help you understand or find what you want.
> >>>>> > >
> >>>>> > >
> >>>>> > >
> >>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
> >>>>> > madhukaudantha@gmail.com>
> >>>>> > > wrote:
> >>>>> > >
> >>>>> > > > Hi All,
> >>>>> > > >
> >>>>> > > > Can you point out some resources  /  tutorials to understand
> the
> >>>>> code
> >>>>> > > base
> >>>>> > > > of zeppelin?
> >>>>> > > > Since I am trying to familiarize myself  with the zeppelin code
> >>>>> > structure
> >>>>> > > > and architecture
> >>>>> > > >
> >>>>> > > > Thanks
> >>>>> > > >
> >>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> >>>>> > > > madhukaudantha@gmail.com> wrote:
> >>>>> > > >
> >>>>> > > >> Hi All,
> >>>>> > > >>
> >>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
> >>>>> 2015
> >>>>> > > >> project, 0COMDEV-1190 captures my interest.
> >>>>> > > >>
> >>>>> > > >> I have abundant experiences of visualization techniques
> creating
> >>>>> > > numerous
> >>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting
> >>>>> etc.
> >>>>> > > >>
> >>>>> > > >> My current research area comprises of big data where I have
> >>>>> worked
> >>>>> > with
> >>>>> > > >> various types of data sets. Also I'm working with cluster
> >>>>> > representation
> >>>>> > > >> and classification techniques where visualization amounts to a
> >>>>> > > considerable
> >>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander
> Bezzubov
> >>>>> and
> >>>>> > > CORNEAU
> >>>>> > > >> Damien for more than week.
> >>>>> > > >>
> >>>>> > > >> Thanks
> >>>>> > > >>
> >>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
> >>>>> > > >> [2] https://github.com/wso2/jaggery
> >>>>> > > >> --
> >>>>> > > >> Cheers,
> >>>>> > > >> Madhuka Udantha
> >>>>> > > >> http://madhukaudantha.blogspot.com
> >>>>> > > >>
> >>>>> > > >
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > --
> >>>>> > > > Cheers,
> >>>>> > > > Madhuka Udantha
> >>>>> > > > http://madhukaudantha.blogspot.com
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > Cheers,
> >>>>> > Madhuka Udantha
> >>>>> > http://madhukaudantha.blogspot.com
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> --
> >>>>> Kind regards,
> >>>>> Alexander.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Cheers,
> >>>> Madhuka Udantha
> >>>> http://madhukaudantha.blogspot.com
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Cheers,
> >>> Madhuka Udantha
> >>> http://madhukaudantha.blogspot.com
> >>>
> >>
> >>
> >>
> >> --
> >> Cheers,
> >> Madhuka Udantha
> >> http://madhukaudantha.blogspot.com
> >>
> >
> >
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by Corneau Damien <co...@apache.org>.

Hi Madhuka and dev mailing list,

Since the code became quite a mess (didn't supervised it enough I guess :p
), and GSOC 2015 will start soon,
I'm currently working on making things easier for people who would like to
participate in the front-end of Zeppelin.
Work in Progress in that Pull Request :
https://github.com/apache/incubator-zeppelin/pull/56

It should be finished before the GSOC 2015 coding time start :)

On Mon, Apr 27, 2015 at 2:00 PM, Ilya Malyanov <im...@gmail.com> wrote:

> Hi Madhuka,
>
> I’m not aware of any efforts in bridging different chart libraries. But my
> guess is, we’ll have to do it ourselves. You can start by picking two
> different libraries or chart types and writing an implementation of the
> transformation function. It might help you to assess the scope of work.
>
> Regarding paragraph.js.
> I don’t think there is documentation for it.
> So the best way to understand the data flow it is to read the code.
> I know it’s not very helpful, but I don’t think anyone can help you in
> this situation.
>
> Regards,
> Ilya
>
>
>
>
>
>
> On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi,
>>
>> In different charting libraries the data template take different formats.
>> Therefore to convert between these formats we use a data transformation
>> module. It converts the input data (JSON format) to different data
>> templates based on the chart library/ chart type selected. If chart library
>> A and chart library B both use the same data template it will reuse the
>> data transformation algorithm.
>>
>> Please share your valuable ideas on this matter. Has any implementation
>> been done similar to this before?
>>
>> I also need to know the data flow within Paragraph.js. Please be kind
>> enough to share the required resources. What is the logging mechanism that
>> is being used in zeppelin?
>>
>> I have written few articles[1] on zeppelin
>>
>> [1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
>>
>> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Here are some blog posts[1] on AngularJS that I wrote last week and
>>> git-hub[2] repo on them
>>>
>>> [1]
>>> http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>>> <http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
>>> [2] https://github.com/Madhuka/angularjs-tutorial
>>>
>>>
>>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>>> madhukaudantha@gmail.com> wrote:
>>>
>>>> Hi Alex,
>>>>
>>>> I ran the tests and they were fine. Yes I'm familiar with the code
>>>> layout/architecture. Basically I'm looking at AngularJS and Typescript. I
>>>> have played around with those two technologies and ran some example code
>>>> which worked fine. I'm writing few blog posts on them as well.
>>>>
>>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
>>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping to write
>>>> some blog posts on it.
>>>>
>>>> I'm also following paragraph.js functions mainly the two
>>>> functions pivot() and pivotDataToD3ChartFormat() for constructing d3 data.
>>>>
>>>> Further I'm finding what are the mainly used charting types by Research
>>>> Teams of Universities all over the world. Also I'm researching on the main
>>>> features that would be used in such graphs.
>>>>
>>>> Yes it will be great to have a call/meeting. This week or following
>>>> will be fine for me. IRC meeting or google hangout will be fine.
>>>>
>>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:
>>>>
>>>>> Hi Madhuka,
>>>>>
>>>>> how Zeppelin code-base exploration it going for you? Do you have any
>>>>> updates?
>>>>> I can see from
>>>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>>>> that you were able to build it well.
>>>>>
>>>>> Were you able to run some tests? Are you familiar by now with code
>>>>> layout\client-server architecture?
>>>>>
>>>>> Asking because it is time for us now to rate the GSoC proposals that
>>>>> we'v
>>>>> got, so please let us know about the progress that you have made so
>>>>> fas, so
>>>>> we can use it as foundation.
>>>>>
>>>>> Will you be interested in making a quick call this early\next week and
>>>>> also
>>>>> talk about your GSoC proposal a bit?
>>>>>
>>>>> That would be awesome and could help us a lot!
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>>>> madhukaudantha@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi Damien,
>>>>> >
>>>>> > Thanks for the quick and descriptive reply. I will go through each
>>>>> of them
>>>>> > and update you on those and I will share my experiences in my blog
>>>>> too. I
>>>>> > will go through the code-base with consideration for the above files.
>>>>> >
>>>>> > Thanks
>>>>> >
>>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <
>>>>> corneadoug@apache.org>
>>>>> > wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > First of all we use angularjs so you might want to familiarize
>>>>> yourself
>>>>> > > with it.
>>>>> > > Currently the code is not well organized (big files), and not
>>>>> really
>>>>> > > angular friendly either, but we are working on fixing that.
>>>>> > >
>>>>> > > We also were preparing a switch to Typescript, so you might want
>>>>> to read
>>>>> > > about it too.
>>>>> > >
>>>>> > > You will find most of the web logic inside of
>>>>> > > zeppelin-web/app/scripts/controllers/
>>>>> > >
>>>>> > > Main controllers are: notebook.js and paragraph.js
>>>>> > >
>>>>> > > notebook.js take care of the full note, while paragraph.js is for
>>>>> each
>>>>> > > paragraph of the notebook.
>>>>> > >
>>>>> > > In notebook.js will have functions that distributes actions to all
>>>>> > > paragraphs, interpreter binding, note style etc...
>>>>> > > In paragraph.js you shall find everything about text-editor,
>>>>> running
>>>>> > > queries, pivot and visualization.
>>>>> > >
>>>>> > > Since the file is somewhere around 1600 lines in paragraph.js, I
>>>>> can help
>>>>> > > you find what you want with more precise questions.
>>>>> > >
>>>>> > > Some functions you might be interested in: 'pivot = function(' , '
>>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
>>>>> '$scope.
>>>>> > > loadTableData = function('
>>>>> > >
>>>>> > > This Pull request (typescript) wasn't ported to the incubator
>>>>> repository
>>>>> > > yet, but there was some work on splitting the paragraph into
>>>>> smaller
>>>>> > parts.
>>>>> > > It might help you understand or find what you want.
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>>>> > madhukaudantha@gmail.com>
>>>>> > > wrote:
>>>>> > >
>>>>> > > > Hi All,
>>>>> > > >
>>>>> > > > Can you point out some resources  /  tutorials to understand the
>>>>> code
>>>>> > > base
>>>>> > > > of zeppelin?
>>>>> > > > Since I am trying to familiarize myself  with the zeppelin code
>>>>> > structure
>>>>> > > > and architecture
>>>>> > > >
>>>>> > > > Thanks
>>>>> > > >
>>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>>> > > > madhukaudantha@gmail.com> wrote:
>>>>> > > >
>>>>> > > >> Hi All,
>>>>> > > >>
>>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
>>>>> 2015
>>>>> > > >> project, 0COMDEV-1190 captures my interest.
>>>>> > > >>
>>>>> > > >> I have abundant experiences of visualization techniques creating
>>>>> > > numerous
>>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting
>>>>> etc.
>>>>> > > >>
>>>>> > > >> My current research area comprises of big data where I have
>>>>> worked
>>>>> > with
>>>>> > > >> various types of data sets. Also I'm working with cluster
>>>>> > representation
>>>>> > > >> and classification techniques where visualization amounts to a
>>>>> > > considerable
>>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov
>>>>> and
>>>>> > > CORNEAU
>>>>> > > >> Damien for more than week.
>>>>> > > >>
>>>>> > > >> Thanks
>>>>> > > >>
>>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>>>> > > >> [2] https://github.com/wso2/jaggery
>>>>> > > >> --
>>>>> > > >> Cheers,
>>>>> > > >> Madhuka Udantha
>>>>> > > >> http://madhukaudantha.blogspot.com
>>>>> > > >>
>>>>> > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > > Cheers,
>>>>> > > > Madhuka Udantha
>>>>> > > > http://madhukaudantha.blogspot.com
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Cheers,
>>>>> > Madhuka Udantha
>>>>> > http://madhukaudantha.blogspot.com
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Kind regards,
>>>>> Alexander.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Madhuka Udantha
>>>> http://madhukaudantha.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by Ilya Malyanov <im...@gmail.com>.

Hi Madhuka,




I’m not aware of any efforts in bridging different chart libraries. But my guess is, we’ll have to do it ourselves. You can start by picking two different libraries or chart types and writing an implementation of the transformation function. It might help you to assess the scope of work.




Regarding paragraph.js. 

I don’t think there is documentation for it.

So the best way to understand the data flow it is to read the code.

I know it’s not very helpful, but I don’t think anyone can help you in this situation.




Regards,

Ilya

On Thu, Apr 23, 2015 at 10:43 PM, madhuka udantha
<ma...@gmail.com> wrote:

> Hi,
> In different charting libraries the data template take different formats.
> Therefore to convert between these formats we use a data transformation
> module. It converts the input data (JSON format) to different data
> templates based on the chart library/ chart type selected. If chart library
> A and chart library B both use the same data template it will reuse the
> data transformation algorithm.
> Please share your valuable ideas on this matter. Has any implementation
> been done similar to this before?
> I also need to know the data flow within Paragraph.js. Please be kind
> enough to share the required resources. What is the logging mechanism that
> is being used in zeppelin?
> I have written few articles[1] on zeppelin
> [1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin
> On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <ma...@gmail.com>
> wrote:
>> Hi,
>>
>> Here are some blog posts[1] on AngularJS that I wrote last week and
>> git-hub[2] repo on them
>>
>> [1] http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
>> <http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
>> [2] https://github.com/Madhuka/angularjs-tutorial
>>
>>
>> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
>> madhukaudantha@gmail.com> wrote:
>>
>>> Hi Alex,
>>>
>>> I ran the tests and they were fine. Yes I'm familiar with the code
>>> layout/architecture. Basically I'm looking at AngularJS and Typescript. I
>>> have played around with those two technologies and ran some example code
>>> which worked fine. I'm writing few blog posts on them as well.
>>>
>>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
>>> getting familiarized with the Zeppelin Code Base. Also I'm hoping to write
>>> some blog posts on it.
>>>
>>> I'm also following paragraph.js functions mainly the two
>>> functions pivot() and pivotDataToD3ChartFormat() for constructing d3 data.
>>>
>>> Further I'm finding what are the mainly used charting types by Research
>>> Teams of Universities all over the world. Also I'm researching on the main
>>> features that would be used in such graphs.
>>>
>>> Yes it will be great to have a call/meeting. This week or following will
>>> be fine for me. IRC meeting or google hangout will be fine.
>>>
>>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:
>>>
>>>> Hi Madhuka,
>>>>
>>>> how Zeppelin code-base exploration it going for you? Do you have any
>>>> updates?
>>>> I can see from
>>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>>> that you were able to build it well.
>>>>
>>>> Were you able to run some tests? Are you familiar by now with code
>>>> layout\client-server architecture?
>>>>
>>>> Asking because it is time for us now to rate the GSoC proposals that we'v
>>>> got, so please let us know about the progress that you have made so fas,
>>>> so
>>>> we can use it as foundation.
>>>>
>>>> Will you be interested in making a quick call this early\next week and
>>>> also
>>>> talk about your GSoC proposal a bit?
>>>>
>>>> That would be awesome and could help us a lot!
>>>> Thanks in advance.
>>>>
>>>>
>>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>>> madhukaudantha@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Damien,
>>>> >
>>>> > Thanks for the quick and descriptive reply. I will go through each of
>>>> them
>>>> > and update you on those and I will share my experiences in my blog
>>>> too. I
>>>> > will go through the code-base with consideration for the above files.
>>>> >
>>>> > Thanks
>>>> >
>>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
>>>> > wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > First of all we use angularjs so you might want to familiarize
>>>> yourself
>>>> > > with it.
>>>> > > Currently the code is not well organized (big files), and not really
>>>> > > angular friendly either, but we are working on fixing that.
>>>> > >
>>>> > > We also were preparing a switch to Typescript, so you might want to
>>>> read
>>>> > > about it too.
>>>> > >
>>>> > > You will find most of the web logic inside of
>>>> > > zeppelin-web/app/scripts/controllers/
>>>> > >
>>>> > > Main controllers are: notebook.js and paragraph.js
>>>> > >
>>>> > > notebook.js take care of the full note, while paragraph.js is for
>>>> each
>>>> > > paragraph of the notebook.
>>>> > >
>>>> > > In notebook.js will have functions that distributes actions to all
>>>> > > paragraphs, interpreter binding, note style etc...
>>>> > > In paragraph.js you shall find everything about text-editor, running
>>>> > > queries, pivot and visualization.
>>>> > >
>>>> > > Since the file is somewhere around 1600 lines in paragraph.js, I can
>>>> help
>>>> > > you find what you want with more precise questions.
>>>> > >
>>>> > > Some functions you might be interested in: 'pivot = function(' , '
>>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
>>>> '$scope.
>>>> > > loadTableData = function('
>>>> > >
>>>> > > This Pull request (typescript) wasn't ported to the incubator
>>>> repository
>>>> > > yet, but there was some work on splitting the paragraph into smaller
>>>> > parts.
>>>> > > It might help you understand or find what you want.
>>>> > >
>>>> > >
>>>> > >
>>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>>> > madhukaudantha@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Hi All,
>>>> > > >
>>>> > > > Can you point out some resources  /  tutorials to understand the
>>>> code
>>>> > > base
>>>> > > > of zeppelin?
>>>> > > > Since I am trying to familiarize myself  with the zeppelin code
>>>> > structure
>>>> > > > and architecture
>>>> > > >
>>>> > > > Thanks
>>>> > > >
>>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>> > > > madhukaudantha@gmail.com> wrote:
>>>> > > >
>>>> > > >> Hi All,
>>>> > > >>
>>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
>>>> 2015
>>>> > > >> project, 0COMDEV-1190 captures my interest.
>>>> > > >>
>>>> > > >> I have abundant experiences of visualization techniques creating
>>>> > > numerous
>>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting
>>>> etc.
>>>> > > >>
>>>> > > >> My current research area comprises of big data where I have worked
>>>> > with
>>>> > > >> various types of data sets. Also I'm working with cluster
>>>> > representation
>>>> > > >> and classification techniques where visualization amounts to a
>>>> > > considerable
>>>> > > >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov
>>>> and
>>>> > > CORNEAU
>>>> > > >> Damien for more than week.
>>>> > > >>
>>>> > > >> Thanks
>>>> > > >>
>>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>>> > > >> [2] https://github.com/wso2/jaggery
>>>> > > >> --
>>>> > > >> Cheers,
>>>> > > >> Madhuka Udantha
>>>> > > >> http://madhukaudantha.blogspot.com
>>>> > > >>
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > Cheers,
>>>> > > > Madhuka Udantha
>>>> > > > http://madhukaudantha.blogspot.com
>>>> > > >
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Cheers,
>>>> > Madhuka Udantha
>>>> > http://madhukaudantha.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Kind regards,
>>>> Alexander.
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
> -- 
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

In different charting libraries the data template take different formats.
Therefore to convert between these formats we use a data transformation
module. It converts the input data (JSON format) to different data
templates based on the chart library/ chart type selected. If chart library
A and chart library B both use the same data template it will reuse the
data transformation algorithm.

Please share your valuable ideas on this matter. Has any implementation
been done similar to this before?

I also need to know the data flow within Paragraph.js. Please be kind
enough to share the required resources. What is the logging mechanism that
is being used in zeppelin?

I have written few articles[1] on zeppelin

[1] http://madhukaudantha.blogspot.com/search/label/Apache%20Zeppelin

On Mon, Apr 20, 2015 at 11:44 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi,
>
> Here are some blog posts[1] on AngularJS that I wrote last week and
> git-hub[2] repo on them
>
> [1] http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
> <http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
> [2] https://github.com/Madhuka/angularjs-tutorial
>
>
> On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi Alex,
>>
>> I ran the tests and they were fine. Yes I'm familiar with the code
>> layout/architecture. Basically I'm looking at AngularJS and Typescript. I
>> have played around with those two technologies and ran some example code
>> which worked fine. I'm writing few blog posts on them as well.
>>
>> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
>> getting familiarized with the Zeppelin Code Base. Also I'm hoping to write
>> some blog posts on it.
>>
>> I'm also following paragraph.js functions mainly the two
>> functions pivot() and pivotDataToD3ChartFormat() for constructing d3 data.
>>
>> Further I'm finding what are the mainly used charting types by Research
>> Teams of Universities all over the world. Also I'm researching on the main
>> features that would be used in such graphs.
>>
>> Yes it will be great to have a call/meeting. This week or following will
>> be fine for me. IRC meeting or google hangout will be fine.
>>
>> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:
>>
>>> Hi Madhuka,
>>>
>>> how Zeppelin code-base exploration it going for you? Do you have any
>>> updates?
>>> I can see from
>>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>>> that you were able to build it well.
>>>
>>> Were you able to run some tests? Are you familiar by now with code
>>> layout\client-server architecture?
>>>
>>> Asking because it is time for us now to rate the GSoC proposals that we'v
>>> got, so please let us know about the progress that you have made so fas,
>>> so
>>> we can use it as foundation.
>>>
>>> Will you be interested in making a quick call this early\next week and
>>> also
>>> talk about your GSoC proposal a bit?
>>>
>>> That would be awesome and could help us a lot!
>>> Thanks in advance.
>>>
>>>
>>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
>>> madhukaudantha@gmail.com>
>>> wrote:
>>>
>>> > Hi Damien,
>>> >
>>> > Thanks for the quick and descriptive reply. I will go through each of
>>> them
>>> > and update you on those and I will share my experiences in my blog
>>> too. I
>>> > will go through the code-base with consideration for the above files.
>>> >
>>> > Thanks
>>> >
>>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > First of all we use angularjs so you might want to familiarize
>>> yourself
>>> > > with it.
>>> > > Currently the code is not well organized (big files), and not really
>>> > > angular friendly either, but we are working on fixing that.
>>> > >
>>> > > We also were preparing a switch to Typescript, so you might want to
>>> read
>>> > > about it too.
>>> > >
>>> > > You will find most of the web logic inside of
>>> > > zeppelin-web/app/scripts/controllers/
>>> > >
>>> > > Main controllers are: notebook.js and paragraph.js
>>> > >
>>> > > notebook.js take care of the full note, while paragraph.js is for
>>> each
>>> > > paragraph of the notebook.
>>> > >
>>> > > In notebook.js will have functions that distributes actions to all
>>> > > paragraphs, interpreter binding, note style etc...
>>> > > In paragraph.js you shall find everything about text-editor, running
>>> > > queries, pivot and visualization.
>>> > >
>>> > > Since the file is somewhere around 1600 lines in paragraph.js, I can
>>> help
>>> > > you find what you want with more precise questions.
>>> > >
>>> > > Some functions you might be interested in: 'pivot = function(' , '
>>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
>>> '$scope.
>>> > > loadTableData = function('
>>> > >
>>> > > This Pull request (typescript) wasn't ported to the incubator
>>> repository
>>> > > yet, but there was some work on splitting the paragraph into smaller
>>> > parts.
>>> > > It might help you understand or find what you want.
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>> > madhukaudantha@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Hi All,
>>> > > >
>>> > > > Can you point out some resources  /  tutorials to understand the
>>> code
>>> > > base
>>> > > > of zeppelin?
>>> > > > Since I am trying to familiarize myself  with the zeppelin code
>>> > structure
>>> > > > and architecture
>>> > > >
>>> > > > Thanks
>>> > > >
>>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>> > > > madhukaudantha@gmail.com> wrote:
>>> > > >
>>> > > >> Hi All,
>>> > > >>
>>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC
>>> 2015
>>> > > >> project, 0COMDEV-1190 captures my interest.
>>> > > >>
>>> > > >> I have abundant experiences of visualization techniques creating
>>> > > numerous
>>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting
>>> etc.
>>> > > >>
>>> > > >> My current research area comprises of big data where I have worked
>>> > with
>>> > > >> various types of data sets. Also I'm working with cluster
>>> > representation
>>> > > >> and classification techniques where visualization amounts to a
>>> > > considerable
>>> > > >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov
>>> and
>>> > > CORNEAU
>>> > > >> Damien for more than week.
>>> > > >>
>>> > > >> Thanks
>>> > > >>
>>> > > >> [1] http://wso2.com/products/user-engagement-server/
>>> > > >> [2] https://github.com/wso2/jaggery
>>> > > >> --
>>> > > >> Cheers,
>>> > > >> Madhuka Udantha
>>> > > >> http://madhukaudantha.blogspot.com
>>> > > >>
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Cheers,
>>> > > > Madhuka Udantha
>>> > > > http://madhukaudantha.blogspot.com
>>> > > >
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> > Madhuka Udantha
>>> > http://madhukaudantha.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> --
>>> Kind regards,
>>> Alexander.
>>>
>>
>>
>>
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi,

Here are some blog posts[1] on AngularJS that I wrote last week and
git-hub[2] repo on them

[1] http://madhukaudantha.blogspot.com/search/label/AngularJS?view=sidebar
<http://madhukaudantha.blogspot.com/search/label/AngularJS?view=classic>
[2] https://github.com/Madhuka/angularjs-tutorial


On Wed, Apr 15, 2015 at 12:20 PM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi Alex,
>
> I ran the tests and they were fine. Yes I'm familiar with the code
> layout/architecture. Basically I'm looking at AngularJS and Typescript. I
> have played around with those two technologies and ran some example code
> which worked fine. I'm writing few blog posts on them as well.
>
> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
> getting familiarized with the Zeppelin Code Base. Also I'm hoping to write
> some blog posts on it.
>
> I'm also following paragraph.js functions mainly the two
> functions pivot() and pivotDataToD3ChartFormat() for constructing d3 data.
>
> Further I'm finding what are the mainly used charting types by Research
> Teams of Universities all over the world. Also I'm researching on the main
> features that would be used in such graphs.
>
> Yes it will be great to have a call/meeting. This week or following will
> be fine for me. IRC meeting or google hangout will be fine.
>
> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:
>
>> Hi Madhuka,
>>
>> how Zeppelin code-base exploration it going for you? Do you have any
>> updates?
>> I can see from
>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>> that you were able to build it well.
>>
>> Were you able to run some tests? Are you familiar by now with code
>> layout\client-server architecture?
>>
>> Asking because it is time for us now to rate the GSoC proposals that we'v
>> got, so please let us know about the progress that you have made so fas,
>> so
>> we can use it as foundation.
>>
>> Will you be interested in making a quick call this early\next week and
>> also
>> talk about your GSoC proposal a bit?
>>
>> That would be awesome and could help us a lot!
>> Thanks in advance.
>>
>>
>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <madhukaudantha@gmail.com
>> >
>> wrote:
>>
>> > Hi Damien,
>> >
>> > Thanks for the quick and descriptive reply. I will go through each of
>> them
>> > and update you on those and I will share my experiences in my blog too.
>> I
>> > will go through the code-base with consideration for the above files.
>> >
>> > Thanks
>> >
>> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > First of all we use angularjs so you might want to familiarize
>> yourself
>> > > with it.
>> > > Currently the code is not well organized (big files), and not really
>> > > angular friendly either, but we are working on fixing that.
>> > >
>> > > We also were preparing a switch to Typescript, so you might want to
>> read
>> > > about it too.
>> > >
>> > > You will find most of the web logic inside of
>> > > zeppelin-web/app/scripts/controllers/
>> > >
>> > > Main controllers are: notebook.js and paragraph.js
>> > >
>> > > notebook.js take care of the full note, while paragraph.js is for each
>> > > paragraph of the notebook.
>> > >
>> > > In notebook.js will have functions that distributes actions to all
>> > > paragraphs, interpreter binding, note style etc...
>> > > In paragraph.js you shall find everything about text-editor, running
>> > > queries, pivot and visualization.
>> > >
>> > > Since the file is somewhere around 1600 lines in paragraph.js, I can
>> help
>> > > you find what you want with more precise questions.
>> > >
>> > > Some functions you might be interested in: 'pivot = function(' , '
>> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
>> '$scope.
>> > > loadTableData = function('
>> > >
>> > > This Pull request (typescript) wasn't ported to the incubator
>> repository
>> > > yet, but there was some work on splitting the paragraph into smaller
>> > parts.
>> > > It might help you understand or find what you want.
>> > >
>> > >
>> > >
>> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>> > madhukaudantha@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > Can you point out some resources  /  tutorials to understand the
>> code
>> > > base
>> > > > of zeppelin?
>> > > > Since I am trying to familiarize myself  with the zeppelin code
>> > structure
>> > > > and architecture
>> > > >
>> > > > Thanks
>> > > >
>> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>> > > > madhukaudantha@gmail.com> wrote:
>> > > >
>> > > >> Hi All,
>> > > >>
>> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
>> > > >> project, 0COMDEV-1190 captures my interest.
>> > > >>
>> > > >> I have abundant experiences of visualization techniques creating
>> > > numerous
>> > > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
>> > > >>
>> > > >> My current research area comprises of big data where I have worked
>> > with
>> > > >> various types of data sets. Also I'm working with cluster
>> > representation
>> > > >> and classification techniques where visualization amounts to a
>> > > considerable
>> > > >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
>> > > CORNEAU
>> > > >> Damien for more than week.
>> > > >>
>> > > >> Thanks
>> > > >>
>> > > >> [1] http://wso2.com/products/user-engagement-server/
>> > > >> [2] https://github.com/wso2/jaggery
>> > > >> --
>> > > >> Cheers,
>> > > >> Madhuka Udantha
>> > > >> http://madhukaudantha.blogspot.com
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Cheers,
>> > > > Madhuka Udantha
>> > > > http://madhukaudantha.blogspot.com
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Cheers,
>> > Madhuka Udantha
>> > http://madhukaudantha.blogspot.com
>> >
>>
>>
>>
>> --
>> --
>> Kind regards,
>> Alexander.
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi Robin,

Please do join with us. I am fine with that.

On Wed, Apr 15, 2015 at 9:14 PM, Robin Hillyard <ro...@gmail.com>
wrote:

> Hi Madhuka,
>
>         I’d be interested in joining your conversation if you would permit
> it. Just say when.
>
>         Robin Hillyard
>
> > On Apr 15, 2015, at 2:50 AM, madhuka udantha <ma...@gmail.com>
> wrote:
> >
> > Hi Alex,
> >
> > I ran the tests and they were fine. Yes I'm familiar with the code
> > layout/architecture. Basically I'm looking at AngularJS and Typescript. I
> > have played around with those two technologies and ran some example code
> > which worked fine. I'm writing few blog posts on them as well.
> >
> > Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
> > getting familiarized with the Zeppelin Code Base. Also I'm hoping to
> write
> > some blog posts on it.
> >
> > I'm also following paragraph.js functions mainly the two
> > functions pivot() and pivotDataToD3ChartFormat() for constructing d3
> data.
> >
> > Further I'm finding what are the mainly used charting types by Research
> > Teams of Universities all over the world. Also I'm researching on the
> main
> > features that would be used in such graphs.
> >
> > Yes it will be great to have a call/meeting. This week or following will
> be
> > fine for me. IRC meeting or google hangout will be fine.
> >
> > On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:
> >
> >> Hi Madhuka,
> >>
> >> how Zeppelin code-base exploration it going for you? Do you have any
> >> updates?
> >> I can see from
> >> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
> >> that you were able to build it well.
> >>
> >> Were you able to run some tests? Are you familiar by now with code
> >> layout\client-server architecture?
> >>
> >> Asking because it is time for us now to rate the GSoC proposals that
> we'v
> >> got, so please let us know about the progress that you have made so
> fas, so
> >> we can use it as foundation.
> >>
> >> Will you be interested in making a quick call this early\next week and
> also
> >> talk about your GSoC proposal a bit?
> >>
> >> That would be awesome and could help us a lot!
> >> Thanks in advance.
> >>
> >>
> >> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <
> madhukaudantha@gmail.com>
> >> wrote:
> >>
> >>> Hi Damien,
> >>>
> >>> Thanks for the quick and descriptive reply. I will go through each of
> >> them
> >>> and update you on those and I will share my experiences in my blog
> too. I
> >>> will go through the code-base with consideration for the above files.
> >>>
> >>> Thanks
> >>>
> >>> On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> First of all we use angularjs so you might want to familiarize
> yourself
> >>>> with it.
> >>>> Currently the code is not well organized (big files), and not really
> >>>> angular friendly either, but we are working on fixing that.
> >>>>
> >>>> We also were preparing a switch to Typescript, so you might want to
> >> read
> >>>> about it too.
> >>>>
> >>>> You will find most of the web logic inside of
> >>>> zeppelin-web/app/scripts/controllers/
> >>>>
> >>>> Main controllers are: notebook.js and paragraph.js
> >>>>
> >>>> notebook.js take care of the full note, while paragraph.js is for each
> >>>> paragraph of the notebook.
> >>>>
> >>>> In notebook.js will have functions that distributes actions to all
> >>>> paragraphs, interpreter binding, note style etc...
> >>>> In paragraph.js you shall find everything about text-editor, running
> >>>> queries, pivot and visualization.
> >>>>
> >>>> Since the file is somewhere around 1600 lines in paragraph.js, I can
> >> help
> >>>> you find what you want with more precise questions.
> >>>>
> >>>> Some functions you might be interested in: 'pivot = function(' , '
> >>>> pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
> >> '$scope.
> >>>> loadTableData = function('
> >>>>
> >>>> This Pull request (typescript) wasn't ported to the incubator
> >> repository
> >>>> yet, but there was some work on splitting the paragraph into smaller
> >>> parts.
> >>>> It might help you understand or find what you want.
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
> >>> madhukaudantha@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Can you point out some resources  /  tutorials to understand the code
> >>>> base
> >>>>> of zeppelin?
> >>>>> Since I am trying to familiarize myself  with the zeppelin code
> >>> structure
> >>>>> and architecture
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> >>>>> madhukaudantha@gmail.com> wrote:
> >>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> >>>>>> project, 0COMDEV-1190 captures my interest.
> >>>>>>
> >>>>>> I have abundant experiences of visualization techniques creating
> >>>> numerous
> >>>>>> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> >>>>>>
> >>>>>> My current research area comprises of big data where I have worked
> >>> with
> >>>>>> various types of data sets. Also I'm working with cluster
> >>> representation
> >>>>>> and classification techniques where visualization amounts to a
> >>>> considerable
> >>>>>> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> >>>> CORNEAU
> >>>>>> Damien for more than week.
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>> [1] http://wso2.com/products/user-engagement-server/
> >>>>>> [2] https://github.com/wso2/jaggery
> >>>>>> --
> >>>>>> Cheers,
> >>>>>> Madhuka Udantha
> >>>>>> http://madhukaudantha.blogspot.com
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Cheers,
> >>>>> Madhuka Udantha
> >>>>> http://madhukaudantha.blogspot.com
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Cheers,
> >>> Madhuka Udantha
> >>> http://madhukaudantha.blogspot.com
> >>>
> >>
> >>
> >>
> >> --
> >> --
> >> Kind regards,
> >> Alexander.
> >>
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
>
>


-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by Robin Hillyard <ro...@gmail.com>.

Hi Madhuka,

	I’d be interested in joining your conversation if you would permit it. Just say when.

	Robin Hillyard

> On Apr 15, 2015, at 2:50 AM, madhuka udantha <ma...@gmail.com> wrote:
> 
> Hi Alex,
> 
> I ran the tests and they were fine. Yes I'm familiar with the code
> layout/architecture. Basically I'm looking at AngularJS and Typescript. I
> have played around with those two technologies and ran some example code
> which worked fine. I'm writing few blog posts on them as well.
> 
> Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
> getting familiarized with the Zeppelin Code Base. Also I'm hoping to write
> some blog posts on it.
> 
> I'm also following paragraph.js functions mainly the two
> functions pivot() and pivotDataToD3ChartFormat() for constructing d3 data.
> 
> Further I'm finding what are the mainly used charting types by Research
> Teams of Universities all over the world. Also I'm researching on the main
> features that would be used in such graphs.
> 
> Yes it will be great to have a call/meeting. This week or following will be
> fine for me. IRC meeting or google hangout will be fine.
> 
> On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:
> 
>> Hi Madhuka,
>> 
>> how Zeppelin code-base exploration it going for you? Do you have any
>> updates?
>> I can see from
>> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
>> that you were able to build it well.
>> 
>> Were you able to run some tests? Are you familiar by now with code
>> layout\client-server architecture?
>> 
>> Asking because it is time for us now to rate the GSoC proposals that we'v
>> got, so please let us know about the progress that you have made so fas, so
>> we can use it as foundation.
>> 
>> Will you be interested in making a quick call this early\next week and also
>> talk about your GSoC proposal a bit?
>> 
>> That would be awesome and could help us a lot!
>> Thanks in advance.
>> 
>> 
>> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <ma...@gmail.com>
>> wrote:
>> 
>>> Hi Damien,
>>> 
>>> Thanks for the quick and descriptive reply. I will go through each of
>> them
>>> and update you on those and I will share my experiences in my blog too. I
>>> will go through the code-base with consideration for the above files.
>>> 
>>> Thanks
>>> 
>>> On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> First of all we use angularjs so you might want to familiarize yourself
>>>> with it.
>>>> Currently the code is not well organized (big files), and not really
>>>> angular friendly either, but we are working on fixing that.
>>>> 
>>>> We also were preparing a switch to Typescript, so you might want to
>> read
>>>> about it too.
>>>> 
>>>> You will find most of the web logic inside of
>>>> zeppelin-web/app/scripts/controllers/
>>>> 
>>>> Main controllers are: notebook.js and paragraph.js
>>>> 
>>>> notebook.js take care of the full note, while paragraph.js is for each
>>>> paragraph of the notebook.
>>>> 
>>>> In notebook.js will have functions that distributes actions to all
>>>> paragraphs, interpreter binding, note style etc...
>>>> In paragraph.js you shall find everything about text-editor, running
>>>> queries, pivot and visualization.
>>>> 
>>>> Since the file is somewhere around 1600 lines in paragraph.js, I can
>> help
>>>> you find what you want with more precise questions.
>>>> 
>>>> Some functions you might be interested in: 'pivot = function(' , '
>>>> pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
>> '$scope.
>>>> loadTableData = function('
>>>> 
>>>> This Pull request (typescript) wasn't ported to the incubator
>> repository
>>>> yet, but there was some work on splitting the paragraph into smaller
>>> parts.
>>>> It might help you understand or find what you want.
>>>> 
>>>> 
>>>> 
>>>> On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
>>> madhukaudantha@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi All,
>>>>> 
>>>>> Can you point out some resources  /  tutorials to understand the code
>>>> base
>>>>> of zeppelin?
>>>>> Since I am trying to familiarize myself  with the zeppelin code
>>> structure
>>>>> and architecture
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
>>>>> madhukaudantha@gmail.com> wrote:
>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
>>>>>> project, 0COMDEV-1190 captures my interest.
>>>>>> 
>>>>>> I have abundant experiences of visualization techniques creating
>>>> numerous
>>>>>> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
>>>>>> 
>>>>>> My current research area comprises of big data where I have worked
>>> with
>>>>>> various types of data sets. Also I'm working with cluster
>>> representation
>>>>>> and classification techniques where visualization amounts to a
>>>> considerable
>>>>>> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
>>>> CORNEAU
>>>>>> Damien for more than week.
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> [1] http://wso2.com/products/user-engagement-server/
>>>>>> [2] https://github.com/wso2/jaggery
>>>>>> --
>>>>>> Cheers,
>>>>>> Madhuka Udantha
>>>>>> http://madhukaudantha.blogspot.com
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Cheers,
>>>>> Madhuka Udantha
>>>>> http://madhukaudantha.blogspot.com
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Cheers,
>>> Madhuka Udantha
>>> http://madhukaudantha.blogspot.com
>>> 
>> 
>> 
>> 
>> --
>> --
>> Kind regards,
>> Alexander.
>> 
> 
> 
> 
> -- 
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi Alex,

I ran the tests and they were fine. Yes I'm familiar with the code
layout/architecture. Basically I'm looking at AngularJS and Typescript. I
have played around with those two technologies and ran some example code
which worked fine. I'm writing few blog posts on them as well.

Currently I'm  looking at zeppelin-web\app\scripts\controllers\ and I'm
getting familiarized with the Zeppelin Code Base. Also I'm hoping to write
some blog posts on it.

I'm also following paragraph.js functions mainly the two
functions pivot() and pivotDataToD3ChartFormat() for constructing d3 data.

Further I'm finding what are the mainly used charting types by Research
Teams of Universities all over the world. Also I'm researching on the main
features that would be used in such graphs.

Yes it will be great to have a call/meeting. This week or following will be
fine for me. IRC meeting or google hangout will be fine.

On Tue, Apr 14, 2015 at 12:33 PM, Alex B. <ab...@nflabs.com> wrote:

> Hi Madhuka,
>
> how Zeppelin code-base exploration it going for you? Do you have any
> updates?
> I can see from
> http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
> that you were able to build it well.
>
> Were you able to run some tests? Are you familiar by now with code
> layout\client-server architecture?
>
> Asking because it is time for us now to rate the GSoC proposals that we'v
> got, so please let us know about the progress that you have made so fas, so
> we can use it as foundation.
>
> Will you be interested in making a quick call this early\next week and also
> talk about your GSoC proposal a bit?
>
> That would be awesome and could help us a lot!
> Thanks in advance.
>
>
> On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <ma...@gmail.com>
> wrote:
>
> > Hi Damien,
> >
> > Thanks for the quick and descriptive reply. I will go through each of
> them
> > and update you on those and I will share my experiences in my blog too. I
> > will go through the code-base with consideration for the above files.
> >
> > Thanks
> >
> > On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > First of all we use angularjs so you might want to familiarize yourself
> > > with it.
> > > Currently the code is not well organized (big files), and not really
> > > angular friendly either, but we are working on fixing that.
> > >
> > > We also were preparing a switch to Typescript, so you might want to
> read
> > > about it too.
> > >
> > > You will find most of the web logic inside of
> > > zeppelin-web/app/scripts/controllers/
> > >
> > > Main controllers are: notebook.js and paragraph.js
> > >
> > > notebook.js take care of the full note, while paragraph.js is for each
> > > paragraph of the notebook.
> > >
> > > In notebook.js will have functions that distributes actions to all
> > > paragraphs, interpreter binding, note style etc...
> > > In paragraph.js you shall find everything about text-editor, running
> > > queries, pivot and visualization.
> > >
> > > Since the file is somewhere around 1600 lines in paragraph.js, I can
> help
> > > you find what you want with more precise questions.
> > >
> > > Some functions you might be interested in: 'pivot = function(' , '
> > > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(',
> '$scope.
> > > loadTableData = function('
> > >
> > > This Pull request (typescript) wasn't ported to the incubator
> repository
> > > yet, but there was some work on splitting the paragraph into smaller
> > parts.
> > > It might help you understand or find what you want.
> > >
> > >
> > >
> > > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
> > madhukaudantha@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Can you point out some resources  /  tutorials to understand the code
> > > base
> > > > of zeppelin?
> > > > Since I am trying to familiarize myself  with the zeppelin code
> > structure
> > > > and architecture
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > > > madhukaudantha@gmail.com> wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > > >> project, 0COMDEV-1190 captures my interest.
> > > >>
> > > >> I have abundant experiences of visualization techniques creating
> > > numerous
> > > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> > > >>
> > > >> My current research area comprises of big data where I have worked
> > with
> > > >> various types of data sets. Also I'm working with cluster
> > representation
> > > >> and classification techniques where visualization amounts to a
> > > considerable
> > > >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> > > CORNEAU
> > > >> Damien for more than week.
> > > >>
> > > >> Thanks
> > > >>
> > > >> [1] http://wso2.com/products/user-engagement-server/
> > > >> [2] https://github.com/wso2/jaggery
> > > >> --
> > > >> Cheers,
> > > >> Madhuka Udantha
> > > >> http://madhukaudantha.blogspot.com
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers,
> > > > Madhuka Udantha
> > > > http://madhukaudantha.blogspot.com
> > > >
> > >
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
> >
>
>
>
> --
> --
> Kind regards,
> Alexander.
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by "Alex B." <ab...@nflabs.com>.

Hi Madhuka,

how Zeppelin code-base exploration it going for you? Do you have any
updates?
I can see from
http://madhukaudantha.blogspot.kr/2015/03/building-apache-zeppelin.html
that you were able to build it well.

Were you able to run some tests? Are you familiar by now with code
layout\client-server architecture?

Asking because it is time for us now to rate the GSoC proposals that we'v
got, so please let us know about the progress that you have made so fas, so
we can use it as foundation.

Will you be interested in making a quick call this early\next week and also
talk about your GSoC proposal a bit?

That would be awesome and could help us a lot!
Thanks in advance.


On Wed, Apr 1, 2015 at 2:27 PM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi Damien,
>
> Thanks for the quick and descriptive reply. I will go through each of them
> and update you on those and I will share my experiences in my blog too. I
> will go through the code-base with consideration for the above files.
>
> Thanks
>
> On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
> wrote:
>
> > Hi,
> >
> > First of all we use angularjs so you might want to familiarize yourself
> > with it.
> > Currently the code is not well organized (big files), and not really
> > angular friendly either, but we are working on fixing that.
> >
> > We also were preparing a switch to Typescript, so you might want to read
> > about it too.
> >
> > You will find most of the web logic inside of
> > zeppelin-web/app/scripts/controllers/
> >
> > Main controllers are: notebook.js and paragraph.js
> >
> > notebook.js take care of the full note, while paragraph.js is for each
> > paragraph of the notebook.
> >
> > In notebook.js will have functions that distributes actions to all
> > paragraphs, interpreter binding, note style etc...
> > In paragraph.js you shall find everything about text-editor, running
> > queries, pivot and visualization.
> >
> > Since the file is somewhere around 1600 lines in paragraph.js, I can help
> > you find what you want with more precise questions.
> >
> > Some functions you might be interested in: 'pivot = function(' , '
> > pivotDataToD3ChartFormat = function(', 'setD3Chart = function(', '$scope.
> > loadTableData = function('
> >
> > This Pull request (typescript) wasn't ported to the incubator repository
> > yet, but there was some work on splitting the paragraph into smaller
> parts.
> > It might help you understand or find what you want.
> >
> >
> >
> > On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <
> madhukaudantha@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Can you point out some resources  /  tutorials to understand the code
> > base
> > > of zeppelin?
> > > Since I am trying to familiarize myself  with the zeppelin code
> structure
> > > and architecture
> > >
> > > Thanks
> > >
> > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > > madhukaudantha@gmail.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> > >> project, 0COMDEV-1190 captures my interest.
> > >>
> > >> I have abundant experiences of visualization techniques creating
> > numerous
> > >> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> > >>
> > >> My current research area comprises of big data where I have worked
> with
> > >> various types of data sets. Also I'm working with cluster
> representation
> > >> and classification techniques where visualization amounts to a
> > considerable
> > >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> > CORNEAU
> > >> Damien for more than week.
> > >>
> > >> Thanks
> > >>
> > >> [1] http://wso2.com/products/user-engagement-server/
> > >> [2] https://github.com/wso2/jaggery
> > >> --
> > >> Cheers,
> > >> Madhuka Udantha
> > >> http://madhukaudantha.blogspot.com
> > >>
> > >
> > >
> > >
> > > --
> > > Cheers,
> > > Madhuka Udantha
> > > http://madhukaudantha.blogspot.com
> > >
> >
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
--
Kind regards,
Alexander.

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi Damien,

Thanks for the quick and descriptive reply. I will go through each of them
and update you on those and I will share my experiences in my blog too. I
will go through the code-base with consideration for the above files.

Thanks

On Wed, Apr 1, 2015 at 7:36 AM, Corneau Damien <co...@apache.org>
wrote:

> Hi,
>
> First of all we use angularjs so you might want to familiarize yourself
> with it.
> Currently the code is not well organized (big files), and not really
> angular friendly either, but we are working on fixing that.
>
> We also were preparing a switch to Typescript, so you might want to read
> about it too.
>
> You will find most of the web logic inside of
> zeppelin-web/app/scripts/controllers/
>
> Main controllers are: notebook.js and paragraph.js
>
> notebook.js take care of the full note, while paragraph.js is for each
> paragraph of the notebook.
>
> In notebook.js will have functions that distributes actions to all
> paragraphs, interpreter binding, note style etc...
> In paragraph.js you shall find everything about text-editor, running
> queries, pivot and visualization.
>
> Since the file is somewhere around 1600 lines in paragraph.js, I can help
> you find what you want with more precise questions.
>
> Some functions you might be interested in: 'pivot = function(' , '
> pivotDataToD3ChartFormat = function(', 'setD3Chart = function(', '$scope.
> loadTableData = function('
>
> This Pull request (typescript) wasn't ported to the incubator repository
> yet, but there was some work on splitting the paragraph into smaller parts.
> It might help you understand or find what you want.
>
>
>
> On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <ma...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Can you point out some resources  /  tutorials to understand the code
> base
> > of zeppelin?
> > Since I am trying to familiarize myself  with the zeppelin code structure
> > and architecture
> >
> > Thanks
> >
> > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> > madhukaudantha@gmail.com> wrote:
> >
> >> Hi All,
> >>
> >> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> >> project, 0COMDEV-1190 captures my interest.
> >>
> >> I have abundant experiences of visualization techniques creating
> numerous
> >> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
> >>
> >> My current research area comprises of big data where I have worked with
> >> various types of data sets. Also I'm working with cluster representation
> >> and classification techniques where visualization amounts to a
> considerable
> >> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and
> CORNEAU
> >> Damien for more than week.
> >>
> >> Thanks
> >>
> >> [1] http://wso2.com/products/user-engagement-server/
> >> [2] https://github.com/wso2/jaggery
> >> --
> >> Cheers,
> >> Madhuka Udantha
> >> http://madhukaudantha.blogspot.com
> >>
> >
> >
> >
> > --
> > Cheers,
> > Madhuka Udantha
> > http://madhukaudantha.blogspot.com
> >
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by Corneau Damien <co...@apache.org>.

Hi,

First of all we use angularjs so you might want to familiarize yourself
with it.
Currently the code is not well organized (big files), and not really
angular friendly either, but we are working on fixing that.

We also were preparing a switch to Typescript, so you might want to read
about it too.

You will find most of the web logic inside of
zeppelin-web/app/scripts/controllers/

Main controllers are: notebook.js and paragraph.js

notebook.js take care of the full note, while paragraph.js is for each
paragraph of the notebook.

In notebook.js will have functions that distributes actions to all
paragraphs, interpreter binding, note style etc...
In paragraph.js you shall find everything about text-editor, running
queries, pivot and visualization.

Since the file is somewhere around 1600 lines in paragraph.js, I can help
you find what you want with more precise questions.

Some functions you might be interested in: 'pivot = function(' , '
pivotDataToD3ChartFormat = function(', 'setD3Chart = function(', '$scope.
loadTableData = function('

This Pull request (typescript) wasn't ported to the incubator repository
yet, but there was some work on splitting the paragraph into smaller parts.
It might help you understand or find what you want.

On Wed, Apr 1, 2015 at 2:23 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi All,
>
> Can you point out some resources  /  tutorials to understand the code base
> of zeppelin?
> Since I am trying to familiarize myself  with the zeppelin code structure
> and architecture
>
> Thanks
>
> On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <
> madhukaudantha@gmail.com> wrote:
>
>> Hi All,
>>
>> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
>> project, 0COMDEV-1190 captures my interest.
>>
>> I have abundant experiences of visualization techniques creating numerous
>> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
>>
>> My current research area comprises of big data where I have worked with
>> various types of data sets. Also I'm working with cluster representation
>> and classification techniques where visualization amounts to a considerable
>> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and CORNEAU
>> Damien for more than week.
>>
>> Thanks
>>
>> [1] http://wso2.com/products/user-engagement-server/
>> [2] https://github.com/wso2/jaggery
>> --
>> Cheers,
>> Madhuka Udantha
>> http://madhukaudantha.blogspot.com
>>
>
>
>
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>

Re: [GSoC 2015][COMDEV-119] Zeppelin GSoC Project: add more D3 visualization

Posted by madhuka udantha <ma...@gmail.com>.

Hi All,

Can you point out some resources  /  tutorials to understand the code base
of zeppelin?
Since I am trying to familiarize myself  with the zeppelin code structure
and architecture

Thanks

On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha <ma...@gmail.com>
wrote:

> Hi All,
>
> I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015
> project, 0COMDEV-1190 captures my interest.
>
> I have abundant experiences of visualization techniques creating numerous
> dashboards[1,2] with javascript, html5, angularJS, d3 charting etc.
>
> My current research area comprises of big data where I have worked with
> various types of data sets. Also I'm working with cluster representation
> and classification techniques where visualization amounts to a considerable
> part. I was following COMDEV-119 (jira) with Alexander Bezzubov and CORNEAU
> Damien for more than week.
>
> Thanks
>
> [1] http://wso2.com/products/user-engagement-server/
> [2] https://github.com/wso2/jaggery
> --
> Cheers,
> Madhuka Udantha
> http://madhukaudantha.blogspot.com
>



-- 
Cheers,
Madhuka Udantha
http://madhukaudantha.blogspot.com