You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ignite.apache.org by Dmitriy Setrakyan <ds...@apache.org> on 2017/08/21 21:07:29 UTC

[DISCUSS] Ignite Update Checker

Igniters,

There has been lots of talk of proposals about various usage metrics for
Ignite and nothing came of it. I would like to resurrect the topic and
propose something very simple and non-intrusive.

1. Update Checker
The main purpose of the update checker is not to collect metrics, but to
notify users about a new version of Ignite by accessing maven.org and
getting the version out of the metadata file:
http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/maven-metadata.xml

This way we do not send any information anywhere and, at the same time,
urge our users to download and start using the latest version of Ignite.

2. Startup Counter
This piece is optional, but we can also get an insight in how many times a
certain Ignite release gets started. This is just a cool metric for the
community to gauge the project popularity. You can think of it as of a page
visit counter shown on many websites. We can even decide to display this
counter on the Ignite website as well.

To do this, we can simply add a JAR in maven for every release, e.g.
ignite-start-counter.jar, which will contain only 1 byte. Every time an
Ignite node starts, it will download this JAR in the background. Then we
will be able to view the number of the total downloads for this JAR in
Maven Central, which is essentially the number of starts of Ignite nodes.

*Note that neither of the above suggestions require Ignite to send or track
any user information whatsoever.*

Please reply suggesting weather you are OK with this approach.

D.

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@apache.org>.

This will surely works to get the update info to the nodes. And it
sounds like a legit approach.
I cannot judge on the feasibility of the GA though - don't know squat
about it ;)

Thanks!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Fri, Aug 25, 2017 at 1:56 PM, Dmitriy Setrakyan
<ds...@apache.org> wrote:
> Cos, Raul,
>
> Thanks for the feedback. I completely agree about Maven Central being a 3rd
> party repo (did not think about that initially). All your suggestions make
> sense, but I would like to keep it as simple as possible, and so far
> everything suggested required GIT dependencies and extra work.
>
> How about Yakov's suggestion. We simply add a page to the Ignite website
> which will have only the latest version. Every time a node starts, it
> receives the latest version from the page and suggests that users upgrade
> if needed.
>
> Also, since we have GA enabled for the website, we can track how many times
> this page was accessed, which will be equal to the number of starts. This
> way, the counter information is tracked and monitored by the Ignite PMC.
>
> This approach looks pretty innocent to me and everything is kept and
> managed within Apache.
>
> Thoughts?
>
> D.
>
>
> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> I agree with Raul.
>> - providing a ping-back address to a 3rd party might be frown upon by some.
>>   And might have a consequences like stats collection about users'
>>   infrastructure.
>> - checking an ASF git-repo is easy and won't download any binary data:
>>   everything is clear text and could be easily monitored by any of the
>> network
>>   diagnostic tools, shall it be required. But it involves a bit of the
>> release
>>   discipline.
>> - the binary data download in the runtime is my main concern. This is the
>>   vector of MMA. What if someone gains the control over the repository and
>>   replaces the file with some malicious content.
>>
>> As for the particular mechanism: IIRC Ignite used to make a call to an
>> external script to check upon the atest software version available for
>> download. In the past, the endpoint was running on a 3rd party server, I
>> believe the best way would be to put this script on ASF infra and have the
>> "update checker" running in a completely controlled environment. Actually,
>> I
>> recall we had this very discussion during the Incubation; I can probably
>> dig
>> out the corresponding thread.
>>
>> Thoughts?
>>   Cok
>>
>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> > Hey guys
>> >
>> > In my opinion, maven.org is still owned by a third party (Sonatype). We
>> > don't know what kind of data analysis or intelligence extraction they
>> run.
>> >
>> > If Ignite servers all over the world were hitting maven.org periodically
>> > asking for an Ignite Maven artifact, it gives Sonatype a clear indication
>> > of who is running an Ignite server.
>> >
>> > They could reverse lookup the IP address and find out what corporation it
>> > is.
>> >
>> > How about having Ignite check the ASF Git directly?
>> >
>> > We could use the Git ssh API, but that would require a new dependency,
>> > which I advise against.
>> >
>> > Alternatively, we could have it scrape this HTML for new Git tags:
>> > https://git-wip-us.apache.org/repos/asf?p=ignite.git
>> >
>> > Another option is to place a txt file in the root of the master branch
>> (e.g
>> > LATEST), containing a list of the latest GA versions for each major
>> version
>> > line (1.x, 2.x).
>> >
>> > I would advocate this last option, but it requires somebody remembering
>> to
>> > update the file with every release, unless we automate it with a Maven
>> > plugin.
>> >
>> > Hope that helps!
>> >
>> >
>> > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
>> >
>> > I see nothing wrong with this approach.
>> >
>> > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to
>> go?
>> >
>> > —
>> > Denis
>> >
>> > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
>> >
>> > wrote:
>> > >
>> > > Is everyone OK with this approach? Should I file a ticket on it?
>> > >
>> > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org>
>> > > wrote:
>> > >
>> > >> Igniters,
>> > >>
>> > >> There has been lots of talk of proposals about various usage metrics
>> for
>> > >> Ignite and nothing came of it. I would like to resurrect the topic and
>> > >> propose something very simple and non-intrusive.
>> > >>
>> > >> 1. Update Checker
>> > >> The main purpose of the update checker is not to collect metrics, but
>> to
>> > >> notify users about a new version of Ignite by accessing maven.org and
>> > >> getting the version out of the metadata file:
>> > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
>> > >> maven-metadata.xml
>> > >>
>> > >> This way we do not send any information anywhere and, at the same
>> time,
>> > >> urge our users to download and start using the latest version of
>> Ignite.
>> > >>
>> > >> 2. Startup Counter
>> > >> This piece is optional, but we can also get an insight in how many
>> times
>> > a
>> > >> certain Ignite release gets started. This is just a cool metric for
>> the
>> > >> community to gauge the project popularity. You can think of it as of a
>> > page
>> > >> visit counter shown on many websites. We can even decide to display
>> this
>> > >> counter on the Ignite website as well.
>> > >>
>> > >> To do this, we can simply add a JAR in maven for every release, e.g.
>> > >> ignite-start-counter.jar, which will contain only 1 byte. Every time
>> an
>> > >> Ignite node starts, it will download this JAR in the background. Then
>> we
>> > >> will be able to view the number of the total downloads for this JAR in
>> > >> Maven Central, which is essentially the number of starts of Ignite
>> nodes.
>> > >>
>> > >> *Note that neither of the above suggestions require Ignite to send or
>> > >> track any user information whatsoever.*
>> > >>
>> > >> Please reply suggesting weather you are OK with this approach.
>> > >>
>> > >> D.
>> > >>
>>
>>

Re: [DISCUSS] Ignite Update Checker

Posted by ds...@apache.org.

Why would we need googleads for youtube? Any chance to disable it?

⁣D.

On Sep 30, 2017, 10:25 AM, at 10:25 AM, Prachi Garg <pg...@gridgain.com> wrote:
>Yes, I see in dev tools that calls to googleads and doubleclick are
>made
>from Youtube.
>
>On Fri, Sep 29, 2017 at 4:33 PM, Denis Magda <dm...@apache.org> wrote:
>
>> I also did a global search on the Ignite website, but didn't find
>anything
>> for googleads or doubleclick.
>>
>>
>> Could you remove and add screencasts block temporary on your local
>> deployment to see if the calls to commercial scripts reported by Cos
>appear
>> in your Chrome dev toolkit?
>>
>> —
>> Denis
>>
>>
>> On Sep 29, 2017, at 3:56 PM, Prachi Garg <pg...@gridgain.com> wrote:
>>
>> We use the following scripts -
>>
>> https://platform.twitter.com/widgets.js - used on homepage to display
>> tweets
>> https://static.addtoany.com/menu/page.js - used on events page for
>social
>> media sharing
>> https://www.google-analytics.com/analytics.js
>>
>> I also did a global search on the Ignite website, but didn't find
>anything
>> for googleads or doubleclick.
>>
>> -Prachi
>>
>>
>> On Fri, Sep 29, 2017 at 11:03 AM, Denis Magda <dm...@gridgain.com>
>wrote:
>>
>>> That’s definitely worthwhile checking. Prachi, as the one who
>embedded
>>> the screencast, would you check the theory?
>>>
>>> —
>>> Denis
>>>
>>> On Sep 28, 2017, at 11:50 PM, Alexey Kuznetsov
><ak...@apache.org>
>>> wrote:
>>>
>>> Cos, Denis.
>>>
>>> I think it is because we have embedded videos (on YouTube).
>>> Mauricio or Denis, please check my idea.
>>>
>>> On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik <co...@apache.org>
>>> wrote:
>>>
>>>> Sorry guys - I neglected the fact that our lists don't permit
>>>> attachments. I have put the screenshot to an external server [1]
>>>>
>>>> [1] https://imgur.com/a/p9FJ9
>>>>
>>>> Thank you!
>>>> --
>>>>   With regards,
>>>> Konstantin (Cos) Boudnik
>>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>>
>>>> Disclaimer: Opinions expressed in this email are those of the
>author,
>>>> and do not necessarily represent the views of any company the
>author
>>>> might be affiliated with at the moment of writing.
>>>>
>>>>
>>>> On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dm...@apache.org>
>wrote:
>>>> > Cos,
>>>> >
>>>> > The screenshot was not attached. Could you share it some other
>way
>>>> (google drive, etc.)? I’ve never seen any commercial on the site.
>>>> >
>>>> > —
>>>> > Denis
>>>> >
>>>> >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <co...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> I don't see an issue with node version either.
>>>> >>
>>>> >> Just one more, and it might be slightly irrelevant, issue
>though... I
>>>> looked at the Ignite's site and found the following ads and
>trackers (that
>>>> are indeed getting disabled by my browser).
>>>> >> Why are googleads or doubleclick are permitted?
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks,
>>>> >>   Cos
>>>> >>
>>>> >>
>>>> >> --
>>>> >>   With regards,
>>>> >> Konstantin (Cos) Boudnik
>>>> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>> >>
>>>> >> Disclaimer: Opinions expressed in this email are those of the
>author,
>>>> and do not necessarily represent the views of any company the
>author might
>>>> be affiliated with at the moment of writing.
>>>> >>
>>>> >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <
>>>> dsetrakyan@apache.org <ma...@apache.org>> wrote:
>>>> >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <
>>>> vozerov@gridgain.com <ma...@gridgain.com>>
>>>> >> wrote:
>>>> >>
>>>> >> > Folks,
>>>> >> >
>>>> >> > Can we add version of current node to web request? This way we
>will
>>>> better
>>>> >> > understand version distribution, what might help us with
>certain API
>>>> >> > update/deprecate decisions
>>>> >> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <
>>>> http://ignite.apache.org/latest.cgi&ver=2.2.0>
>>>> >>
>>>> >>
>>>> >> Vladimir, I personally do not see a problem with that, as
>sending the
>>>> >> current version to the update checker seems very innocent to me.
>At
>>>> the
>>>> >> same time, it will allow us to examine the usage of each release
>and
>>>> make
>>>> >> decisions about dropping backward compatibility or spotting bugs
>for a
>>>> >> certain release.
>>>> >>
>>>> >> Cos, Raul, any thoughts?
>>>> >>
>>>> >>
>>>> >> >
>>>> >> >
>>>> >> > Vladimir.
>>>> >> >
>>>> >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <
>>>> dsetrakyan@apache.org <ma...@apache.org>>
>>>> >> > wrote:
>>>> >> >
>>>> >> > > I think it is safe to assume at this point that everyone is
>in
>>>> general
>>>> >> > > agreement, since there are no active objections.
>>>> >> > >
>>>> >> > > I have filed a ticket for the 2.3 release. Let's try to make
>it
>>>> happen:
>>>> >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <
>>>> https://issues.apache.org/jira/browse/IGNITE-6305>
>>>> >> > >
>>>> >> > > D.
>>>> >> > >
>>>> >> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>> >> > dsetrakyan@apache.org <ma...@apache.org>>
>>>> >> > > wrote:
>>>> >> > >
>>>> >> > > >
>>>> >> > > >
>>>> >> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>>> raul.asf@evosent.com <ma...@evosent.com>>
>>>> >> > > > wrote:
>>>> >> > > >
>>>> >> > > >> Yeah, I guess that's doable as well and requires less
>>>> management
>>>> >> > effort
>>>> >> > > >> than my suggestion. We could use events [1] to store
>payload
>>>> data
>>>> >> > (e.g.
>>>> >> > > >> IP,
>>>> >> > > >> version, etc.)
>>>> >> > > >
>>>> >> > > >
>>>> >> > > > Yes, we could use events or some other similar API
>provided by
>>>> GA.
>>>> >> > > >
>>>> >> > > >
>>>> >> > > >> What the download page CGI developed in? PHP?
>>>> >> > > >>
>>>> >> > > >
>>>> >> > > > To be honest, no clue. I guess someone in the community
>can
>>>> figure it
>>>> >> > > out:
>>>> >> > > >
>https://svn.apache.org/repos/asf/ignite/site/trunk/download.
>>>> html
><https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>
>>>> >> > > >
>>>> >> > > >
>>>> >> > > >> However, I'm not sure whether storing this data in a 3rd
>party
>>>> >> > (Google)
>>>> >> > > is
>>>> >> > > >> compliant with the ASF policy. I guess it's no biggie,
>but if
>>>> there's
>>>> >> > > >> doubt
>>>> >> > > >> in the PMC, it's better to ask legal@.
>>>> >> > > >
>>>> >> > > >
>>>> >> > > > I am not sure there is anything to ask about. The whole
>Ignite
>>>> website
>>>> >> > is
>>>> >> > > > GA enabled, and all we are doing is accessing a standard
>web
>>>> page from
>>>> >> > > the
>>>> >> > > > Ignite web site. The information gathered from GA is
>available
>>>> only to
>>>> >> > > the
>>>> >> > > > Ignite PMC. Frankly, I think legal@ will be very confused
>by
>>>> this
>>>> >> > > > question.
>>>> >> > > >
>>>> >> > > > Even ASF website itself uses GA: https://www.apache.org/ <
>>>> https://www.apache.org/>
>>>> >> > > > foundation/policies/privacy.html
>>>> >> > > >
>>>> >> > > >
>>>> >> > > >> I think Cos said it's OK; maybe Roman can pitch in.
>>>> >> > > >>
>>>> >> > > >
>>>> >> > > >  Sure, would be nice to hear from Roman as well.
>>>> >> > > >
>>>> >> > > >
>>>> >> > > >> Cheers.
>>>> >> > > >>
>>>> >> > > >> [1]
>>>> >> > > >>
>https://developers.google.com/analytics/devguides/collection <
>>>> https://developers.google.com/analytics/devguides/collection>
>>>> >> > > >> /analyticsjs/events
>>>> >> > > >>
>>>> >> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>> >> > > >> dsetrakyan@apache.org <ma...@apache.org>>
>>>> >> > > >> wrote:
>>>> >> > > >>
>>>> >> > > >> > Raul,
>>>> >> > > >> >
>>>> >> > > >> > Could point about Javascript, it will not work because
>it
>>>> executes
>>>> >> > in
>>>> >> > > >> the
>>>> >> > > >> > browser. This means we need a server-side script, like
>CGI
>>>> we are
>>>> >> > > using
>>>> >> > > >> on
>>>> >> > > >> > our download page.
>>>> >> > > >> >
>>>> >> > > >> > How about this approach. We create something like
>>>> ignite-version.cgi
>>>> >> > > >> script
>>>> >> > > >> > which will invoke a call to GA and then return the
>latest
>>>> version.
>>>> >> > > >> >
>>>> >> > > >> > This should work, right?
>>>> >> > > >> >
>>>> >> > > >> > D.
>>>> >> > > >> >
>>>> >> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>> >> > raul.asf@evosent.com <ma...@evosent.com>
>>>> >> > > >
>>>> >> > > >> > wrote:
>>>> >> > > >> >
>>>> >> > > >> > > Hey Dmitriy and all
>>>> >> > > >> > >
>>>> >> > > >> > > Also, since we have GA enabled for the website, we
>can
>>>> track how
>>>> >> > > many
>>>> >> > > >> > times
>>>> >> > > >> > > > this page was accessed, which will be equal to the
>>>> number of
>>>> >> > > starts.
>>>> >> > > >> > This
>>>> >> > > >> > > > way, the counter information is tracked and
>monitored by
>>>> the
>>>> >> > > Ignite
>>>> >> > > >> > PMC.
>>>> >> > > >> > >
>>>> >> > > >> > >
>>>> >> > > >> > > Unfortunately this won't work because GA is loaded
>via
>>>> Javascript
>>>> >> > on
>>>> >> > > >> the
>>>> >> > > >> > > browser, so Google will never receive the page hit.
>>>> >> > > >> > >
>>>> >> > > >> > > Given the constraints, the most viable solution is an
>HTTPS
>>>> >> > endpoint
>>>> >> > > >> > > running on ASF infra that Ignite invokes via a GET or
>POST
>>>> >> > request.
>>>> >> > > >> The
>>>> >> > > >> > > simplest thing is to write each request in a log
>file,
>>>> along with
>>>> >> > > the
>>>> >> > > >> > > timestamp, the version reported by the client, maybe
>the
>>>> IP (not
>>>> >> > > sure
>>>> >> > > >> > about
>>>> >> > > >> > > the ASF rules about this concerning privacy, I guess
>it's
>>>> OK if
>>>> >> > you
>>>> >> > > >> make
>>>> >> > > >> > it
>>>> >> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID
>the
>>>> node
>>>> >> > > creates
>>>> >> > > >> on
>>>> >> > > >> > > first startup or something.
>>>> >> > > >> > >
>>>> >> > > >> > > This endpoint would need some basic DDoS protection
>and
>>>> >> > blacklisting
>>>> >> > > >> to
>>>> >> > > >> > > prevent data spoofing.
>>>> >> > > >> > >
>>>> >> > > >> > > If we'll be implementing this endpoint anyway, then
>>>> there's no
>>>> >> > point
>>>> >> > > >> > > placing another file on Git or elsewhere for
>reporting the
>>>> latest
>>>> >> > > >> > versions:
>>>> >> > > >> > > the endpoint itself can return them.
>>>> >> > > >> > >
>>>> >> > > >> > > WDYT?
>>>> >> > > >> > >
>>>> >> > > >> > > Cheers.
>>>> >> > > >> > >
>>>> >> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>> >> > > >> > dsetrakyan@apache.org <ma...@apache.org>>
>>>> >> > > >> > > wrote:
>>>> >> > > >> > >
>>>> >> > > >> > > > Cos, Raul,
>>>> >> > > >> > > >
>>>> >> > > >> > > > Thanks for the feedback. I completely agree about
>Maven
>>>> Central
>>>> >> > > >> being a
>>>> >> > > >> > > 3rd
>>>> >> > > >> > > > party repo (did not think about that initially).
>All your
>>>> >> > > >> suggestions
>>>> >> > > >> > > make
>>>> >> > > >> > > > sense, but I would like to keep it as simple as
>>>> possible, and so
>>>> >> > > far
>>>> >> > > >> > > > everything suggested required GIT dependencies and
>extra
>>>> work.
>>>> >> > > >> > > >
>>>> >> > > >> > > > How about Yakov's suggestion. We simply add a page
>to
>>>> the Ignite
>>>> >> > > >> > website
>>>> >> > > >> > > > which will have only the latest version. Every time
>a
>>>> node
>>>> >> > starts,
>>>> >> > > >> it
>>>> >> > > >> > > > receives the latest version from the page and
>suggests
>>>> that
>>>> >> > users
>>>> >> > > >> > upgrade
>>>> >> > > >> > > > if needed.
>>>> >> > > >> > > >
>>>> >> > > >> > > > Also, since we have GA enabled for the website, we
>can
>>>> track how
>>>> >> > > >> many
>>>> >> > > >> > > times
>>>> >> > > >> > > > this page was accessed, which will be equal to the
>>>> number of
>>>> >> > > starts.
>>>> >> > > >> > This
>>>> >> > > >> > > > way, the counter information is tracked and
>monitored by
>>>> the
>>>> >> > > Ignite
>>>> >> > > >> > PMC.
>>>> >> > > >> > > >
>>>> >> > > >> > > > This approach looks pretty innocent to me and
>everything
>>>> is kept
>>>> >> > > and
>>>> >> > > >> > > > managed within Apache.
>>>> >> > > >> > > >
>>>> >> > > >> > > > Thoughts?
>>>> >> > > >> > > >
>>>> >> > > >> > > > D.
>>>> >> > > >> > > >
>>>> >> > > >> > > >
>>>> >> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin
>Boudnik <
>>>> >> > > >> cos@apache.org <ma...@apache.org>>
>>>> >> > > >> > > > wrote:
>>>> >> > > >> > > >
>>>> >> > > >> > > > > I agree with Raul.
>>>> >> > > >> > > > > - providing a ping-back address to a 3rd party
>might
>>>> be frown
>>>> >> > > >> upon by
>>>> >> > > >> > > > some.
>>>> >> > > >> > > > >   And might have a consequences like stats
>collection
>>>> about
>>>> >> > > users'
>>>> >> > > >> > > > >   infrastructure.
>>>> >> > > >> > > > > - checking an ASF git-repo is easy and won't
>download
>>>> any
>>>> >> > binary
>>>> >> > > >> > data:
>>>> >> > > >> > > > >   everything is clear text and could be easily
>>>> monitored by
>>>> >> > any
>>>> >> > > of
>>>> >> > > >> > the
>>>> >> > > >> > > > > network
>>>> >> > > >> > > > >   diagnostic tools, shall it be required. But it
>>>> involves a
>>>> >> > bit
>>>> >> > > of
>>>> >> > > >> > the
>>>> >> > > >> > > > > release
>>>> >> > > >> > > > >   discipline.
>>>> >> > > >> > > > > - the binary data download in the runtime is my
>main
>>>> concern.
>>>> >> > > >> This is
>>>> >> > > >> > > the
>>>> >> > > >> > > > >   vector of MMA. What if someone gains the
>control
>>>> over the
>>>> >> > > >> > repository
>>>> >> > > >> > > > and
>>>> >> > > >> > > > >   replaces the file with some malicious content.
>>>> >> > > >> > > > >
>>>> >> > > >> > > > > As for the particular mechanism: IIRC Ignite used
>to
>>>> make a
>>>> >> > call
>>>> >> > > >> to
>>>> >> > > >> > an
>>>> >> > > >> > > > > external script to check upon the atest software
>>>> version
>>>> >> > > available
>>>> >> > > >> > for
>>>> >> > > >> > > > > download. In the past, the endpoint was running
>on a
>>>> 3rd party
>>>> >> > > >> > server,
>>>> >> > > >> > > I
>>>> >> > > >> > > > > believe the best way would be to put this script
>on
>>>> ASF infra
>>>> >> > > and
>>>> >> > > >> > have
>>>> >> > > >> > > > the
>>>> >> > > >> > > > > "update checker" running in a completely
>controlled
>>>> >> > environment.
>>>> >> > > >> > > > Actually,
>>>> >> > > >> > > > > I
>>>> >> > > >> > > > > recall we had this very discussion during the
>>>> Incubation; I
>>>> >> > can
>>>> >> > > >> > > probably
>>>> >> > > >> > > > > dig
>>>> >> > > >> > > > > out the corresponding thread.
>>>> >> > > >> > > > >
>>>> >> > > >> > > > > Thoughts?
>>>> >> > > >> > > > >   Cok
>>>> >> > > >> > > > >
>>>> >> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani
>wrote:
>>>> >> > > >> > > > > > Hey guys
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > In my opinion, maven.org <http://maven.org/> is
>>>> still owned by a third party
>>>> >> > > >> > (Sonatype).
>>>> >> > > >> > > > We
>>>> >> > > >> > > > > > don't know what kind of data analysis or
>intelligence
>>>> >> > > extraction
>>>> >> > > >> > they
>>>> >> > > >> > > > > run.
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > If Ignite servers all over the world were
>hitting
>>>> maven.org <http://maven.org/>
>>>> >> > > >> > > > periodically
>>>> >> > > >> > > > > > asking for an Ignite Maven artifact, it gives
>>>> Sonatype a
>>>> >> > clear
>>>> >> > > >> > > > indication
>>>> >> > > >> > > > > > of who is running an Ignite server.
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > They could reverse lookup the IP address and
>find
>>>> out what
>>>> >> > > >> > > corporation
>>>> >> > > >> > > > it
>>>> >> > > >> > > > > > is.
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > How about having Ignite check the ASF Git
>directly?
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > We could use the Git ssh API, but that would
>require
>>>> a new
>>>> >> > > >> > > dependency,
>>>> >> > > >> > > > > > which I advise against.
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > Alternatively, we could have it scrape this
>HTML for
>>>> new Git
>>>> >> > > >> tags:
>>>> >> > > >> > > > > >
>https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>> <https://git-wip-us.apache.org/repos/asf?p=ignite.git>
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > Another option is to place a txt file in the
>root of
>>>> the
>>>> >> > > master
>>>> >> > > >> > > branch
>>>> >> > > >> > > > > (e.g
>>>> >> > > >> > > > > > LATEST), containing a list of the latest GA
>versions
>>>> for
>>>> >> > each
>>>> >> > > >> major
>>>> >> > > >> > > > > version
>>>> >> > > >> > > > > > line (1.x, 2.x).
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > I would advocate this last option, but it
>requires
>>>> somebody
>>>> >> > > >> > > remembering
>>>> >> > > >> > > > > to
>>>> >> > > >> > > > > > update the file with every release, unless we
>>>> automate it
>>>> >> > > with a
>>>> >> > > >> > > Maven
>>>> >> > > >> > > > > > plugin.
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > Hope that helps!
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <
>>>> dmagda@apache.org <ma...@apache.org>>
>>>> >> > > wrote:
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > I see nothing wrong with this approach.
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do
>you
>>>> think? Is
>>>> >> > it
>>>> >> > > >> good
>>>> >> > > >> > > to
>>>> >> > > >> > > > > go?
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > —
>>>> >> > > >> > > > > > Denis
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy
>Setrakyan <
>>>> >> > > >> > > > dsetrakyan@apache.org
><ma...@apache.org>
>>>> >> > > >> > > > > >
>>>> >> > > >> > > > > > wrote:
>>>> >> > > >> > > > > > >
>>>> >> > > >> > > > > > > Is everyone OK with this approach? Should I
>file a
>>>> ticket
>>>> >> > on
>>>> >> > > >> it?
>>>> >> > > >> > > > > > >
>>>> >> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy
>Setrakyan
>>>> <
>>>> >> > > >> > > > > dsetrakyan@apache.org
><ma...@apache.org>>
>>>> >> > > >> > > > > > > wrote:
>>>> >> > > >> > > > > > >
>>>> >> > > >> > > > > > >> Igniters,
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> There has been lots of talk of proposals
>about
>>>> various
>>>> >> > > usage
>>>> >> > > >> > > metrics
>>>> >> > > >> > > > > for
>>>> >> > > >> > > > > > >> Ignite and nothing came of it. I would like
>to
>>>> resurrect
>>>> >> > > the
>>>> >> > > >> > topic
>>>> >> > > >> > > > and
>>>> >> > > >> > > > > > >> propose something very simple and
>non-intrusive.
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> 1. Update Checker
>>>> >> > > >> > > > > > >> The main purpose of the update checker is
>not to
>>>> collect
>>>> >> > > >> > metrics,
>>>> >> > > >> > > > but
>>>> >> > > >> > > > > to
>>>> >> > > >> > > > > > >> notify users about a new version of Ignite
>by
>>>> accessing
>>>> >> > > >> > maven.org <http://maven.org/>
>>>> >> > > >> > > > and
>>>> >> > > >> > > > > > >> getting the version out of the metadata
>file:
>>>> >> > > >> > > > > > >> http://repo2.maven.org/maven2/ <
>>>> http://repo2.maven.org/maven2/>
>>>> >> > > org/apache/ignite/ignite-core/
>>>> >> > > >> > > > > > >> maven-metadata.xml
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> This way we do not send any information
>anywhere
>>>> and, at
>>>> >> > > the
>>>> >> > > >> > same
>>>> >> > > >> > > > > time,
>>>> >> > > >> > > > > > >> urge our users to download and start using
>the
>>>> latest
>>>> >> > > >> version of
>>>> >> > > >> > > > > Ignite.
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> 2. Startup Counter
>>>> >> > > >> > > > > > >> This piece is optional, but we can also get
>an
>>>> insight in
>>>> >> > > how
>>>> >> > > >> > many
>>>> >> > > >> > > > > times
>>>> >> > > >> > > > > > a
>>>> >> > > >> > > > > > >> certain Ignite release gets started. This is
>just
>>>> a cool
>>>> >> > > >> metric
>>>> >> > > >> > > for
>>>> >> > > >> > > > > the
>>>> >> > > >> > > > > > >> community to gauge the project popularity.
>You
>>>> can think
>>>> >> > of
>>>> >> > > >> it
>>>> >> > > >> > as
>>>> >> > > >> > > > of a
>>>> >> > > >> > > > > > page
>>>> >> > > >> > > > > > >> visit counter shown on many websites. We can
>even
>>>> decide
>>>> >> > to
>>>> >> > > >> > > display
>>>> >> > > >> > > > > this
>>>> >> > > >> > > > > > >> counter on the Ignite website as well.
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> To do this, we can simply add a JAR in maven
>for
>>>> every
>>>> >> > > >> release,
>>>> >> > > >> > > e.g.
>>>> >> > > >> > > > > > >> ignite-start-counter.jar, which will contain
>only
>>>> 1 byte.
>>>> >> > > >> Every
>>>> >> > > >> > > time
>>>> >> > > >> > > > > an
>>>> >> > > >> > > > > > >> Ignite node starts, it will download this
>JAR in
>>>> the
>>>> >> > > >> background.
>>>> >> > > >> > > > Then
>>>> >> > > >> > > > > we
>>>> >> > > >> > > > > > >> will be able to view the number of the total
>>>> downloads
>>>> >> > for
>>>> >> > > >> this
>>>> >> > > >> > > JAR
>>>> >> > > >> > > > in
>>>> >> > > >> > > > > > >> Maven Central, which is essentially the
>number of
>>>> starts
>>>> >> > of
>>>> >> > > >> > Ignite
>>>> >> > > >> > > > > nodes.
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> *Note that neither of the above suggestions
>>>> require
>>>> >> > Ignite
>>>> >> > > to
>>>> >> > > >> > send
>>>> >> > > >> > > > or
>>>> >> > > >> > > > > > >> track any user information whatsoever.*
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> Please reply suggesting weather you are OK
>with
>>>> this
>>>> >> > > >> approach.
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > > > >> D.
>>>> >> > > >> > > > > > >>
>>>> >> > > >> > > > >
>>>> >> > > >> > > > >
>>>> >> > > >> > > >
>>>> >> > > >> > >
>>>> >> > > >> >
>>>> >> > > >>
>>>> >> > > >
>>>> >> > > >
>>>> >> > >
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Alexey Kuznetsov
>>>
>>>
>>>
>>
>>

Re: [DISCUSS] Ignite Update Checker

Posted by Prachi Garg <pg...@gridgain.com>.

Yes, I see in dev tools that calls to googleads and doubleclick are made
from Youtube.

On Fri, Sep 29, 2017 at 4:33 PM, Denis Magda <dm...@apache.org> wrote:

> I also did a global search on the Ignite website, but didn't find anything
> for googleads or doubleclick.
>
>
> Could you remove and add screencasts block temporary on your local
> deployment to see if the calls to commercial scripts reported by Cos appear
> in your Chrome dev toolkit?
>
> —
> Denis
>
>
> On Sep 29, 2017, at 3:56 PM, Prachi Garg <pg...@gridgain.com> wrote:
>
> We use the following scripts -
>
> https://platform.twitter.com/widgets.js - used on homepage to display
> tweets
> https://static.addtoany.com/menu/page.js - used on events page for social
> media sharing
> https://www.google-analytics.com/analytics.js
>
> I also did a global search on the Ignite website, but didn't find anything
> for googleads or doubleclick.
>
> -Prachi
>
>
> On Fri, Sep 29, 2017 at 11:03 AM, Denis Magda <dm...@gridgain.com> wrote:
>
>> That’s definitely worthwhile checking. Prachi, as the one who embedded
>> the screencast, would you check the theory?
>>
>> —
>> Denis
>>
>> On Sep 28, 2017, at 11:50 PM, Alexey Kuznetsov <ak...@apache.org>
>> wrote:
>>
>> Cos, Denis.
>>
>> I think it is because we have embedded videos (on YouTube).
>> Mauricio or Denis, please check my idea.
>>
>> On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>
>>> Sorry guys - I neglected the fact that our lists don't permit
>>> attachments. I have put the screenshot to an external server [1]
>>>
>>> [1] https://imgur.com/a/p9FJ9
>>>
>>> Thank you!
>>> --
>>>   With regards,
>>> Konstantin (Cos) Boudnik
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>
>>> Disclaimer: Opinions expressed in this email are those of the author,
>>> and do not necessarily represent the views of any company the author
>>> might be affiliated with at the moment of writing.
>>>
>>>
>>> On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dm...@apache.org> wrote:
>>> > Cos,
>>> >
>>> > The screenshot was not attached. Could you share it some other way
>>> (google drive, etc.)? I’ve never seen any commercial on the site.
>>> >
>>> > —
>>> > Denis
>>> >
>>> >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <co...@apache.org>
>>> wrote:
>>> >>
>>> >> I don't see an issue with node version either.
>>> >>
>>> >> Just one more, and it might be slightly irrelevant, issue though... I
>>> looked at the Ignite's site and found the following ads and trackers (that
>>> are indeed getting disabled by my browser).
>>> >> Why are googleads or doubleclick are permitted?
>>> >>
>>> >>
>>> >>
>>> >> Thanks,
>>> >>   Cos
>>> >>
>>> >>
>>> >> --
>>> >>   With regards,
>>> >> Konstantin (Cos) Boudnik
>>> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>> >>
>>> >> Disclaimer: Opinions expressed in this email are those of the author,
>>> and do not necessarily represent the views of any company the author might
>>> be affiliated with at the moment of writing.
>>> >>
>>> >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org <ma...@apache.org>> wrote:
>>> >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <
>>> vozerov@gridgain.com <ma...@gridgain.com>>
>>> >> wrote:
>>> >>
>>> >> > Folks,
>>> >> >
>>> >> > Can we add version of current node to web request? This way we will
>>> better
>>> >> > understand version distribution, what might help us with certain API
>>> >> > update/deprecate decisions
>>> >> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <
>>> http://ignite.apache.org/latest.cgi&ver=2.2.0>
>>> >>
>>> >>
>>> >> Vladimir, I personally do not see a problem with that, as sending the
>>> >> current version to the update checker seems very innocent to me. At
>>> the
>>> >> same time, it will allow us to examine the usage of each release and
>>> make
>>> >> decisions about dropping backward compatibility or spotting bugs for a
>>> >> certain release.
>>> >>
>>> >> Cos, Raul, any thoughts?
>>> >>
>>> >>
>>> >> >
>>> >> >
>>> >> > Vladimir.
>>> >> >
>>> >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org <ma...@apache.org>>
>>> >> > wrote:
>>> >> >
>>> >> > > I think it is safe to assume at this point that everyone is in
>>> general
>>> >> > > agreement, since there are no active objections.
>>> >> > >
>>> >> > > I have filed a ticket for the 2.3 release. Let's try to make it
>>> happen:
>>> >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <
>>> https://issues.apache.org/jira/browse/IGNITE-6305>
>>> >> > >
>>> >> > > D.
>>> >> > >
>>> >> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>> >> > dsetrakyan@apache.org <ma...@apache.org>>
>>> >> > > wrote:
>>> >> > >
>>> >> > > >
>>> >> > > >
>>> >> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>> raul.asf@evosent.com <ma...@evosent.com>>
>>> >> > > > wrote:
>>> >> > > >
>>> >> > > >> Yeah, I guess that's doable as well and requires less
>>> management
>>> >> > effort
>>> >> > > >> than my suggestion. We could use events [1] to store payload
>>> data
>>> >> > (e.g.
>>> >> > > >> IP,
>>> >> > > >> version, etc.)
>>> >> > > >
>>> >> > > >
>>> >> > > > Yes, we could use events or some other similar API provided by
>>> GA.
>>> >> > > >
>>> >> > > >
>>> >> > > >> What the download page CGI developed in? PHP?
>>> >> > > >>
>>> >> > > >
>>> >> > > > To be honest, no clue. I guess someone in the community can
>>> figure it
>>> >> > > out:
>>> >> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.
>>> html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>
>>> >> > > >
>>> >> > > >
>>> >> > > >> However, I'm not sure whether storing this data in a 3rd party
>>> >> > (Google)
>>> >> > > is
>>> >> > > >> compliant with the ASF policy. I guess it's no biggie, but if
>>> there's
>>> >> > > >> doubt
>>> >> > > >> in the PMC, it's better to ask legal@.
>>> >> > > >
>>> >> > > >
>>> >> > > > I am not sure there is anything to ask about. The whole Ignite
>>> website
>>> >> > is
>>> >> > > > GA enabled, and all we are doing is accessing a standard web
>>> page from
>>> >> > > the
>>> >> > > > Ignite web site. The information gathered from GA is available
>>> only to
>>> >> > > the
>>> >> > > > Ignite PMC. Frankly, I think legal@ will be very confused by
>>> this
>>> >> > > > question.
>>> >> > > >
>>> >> > > > Even ASF website itself uses GA: https://www.apache.org/ <
>>> https://www.apache.org/>
>>> >> > > > foundation/policies/privacy.html
>>> >> > > >
>>> >> > > >
>>> >> > > >> I think Cos said it's OK; maybe Roman can pitch in.
>>> >> > > >>
>>> >> > > >
>>> >> > > >  Sure, would be nice to hear from Roman as well.
>>> >> > > >
>>> >> > > >
>>> >> > > >> Cheers.
>>> >> > > >>
>>> >> > > >> [1]
>>> >> > > >> https://developers.google.com/analytics/devguides/collection <
>>> https://developers.google.com/analytics/devguides/collection>
>>> >> > > >> /analyticsjs/events
>>> >> > > >>
>>> >> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>> >> > > >> dsetrakyan@apache.org <ma...@apache.org>>
>>> >> > > >> wrote:
>>> >> > > >>
>>> >> > > >> > Raul,
>>> >> > > >> >
>>> >> > > >> > Could point about Javascript, it will not work because it
>>> executes
>>> >> > in
>>> >> > > >> the
>>> >> > > >> > browser. This means we need a server-side script, like CGI
>>> we are
>>> >> > > using
>>> >> > > >> on
>>> >> > > >> > our download page.
>>> >> > > >> >
>>> >> > > >> > How about this approach. We create something like
>>> ignite-version.cgi
>>> >> > > >> script
>>> >> > > >> > which will invoke a call to GA and then return the latest
>>> version.
>>> >> > > >> >
>>> >> > > >> > This should work, right?
>>> >> > > >> >
>>> >> > > >> > D.
>>> >> > > >> >
>>> >> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>> >> > raul.asf@evosent.com <ma...@evosent.com>
>>> >> > > >
>>> >> > > >> > wrote:
>>> >> > > >> >
>>> >> > > >> > > Hey Dmitriy and all
>>> >> > > >> > >
>>> >> > > >> > > Also, since we have GA enabled for the website, we can
>>> track how
>>> >> > > many
>>> >> > > >> > times
>>> >> > > >> > > > this page was accessed, which will be equal to the
>>> number of
>>> >> > > starts.
>>> >> > > >> > This
>>> >> > > >> > > > way, the counter information is tracked and monitored by
>>> the
>>> >> > > Ignite
>>> >> > > >> > PMC.
>>> >> > > >> > >
>>> >> > > >> > >
>>> >> > > >> > > Unfortunately this won't work because GA is loaded via
>>> Javascript
>>> >> > on
>>> >> > > >> the
>>> >> > > >> > > browser, so Google will never receive the page hit.
>>> >> > > >> > >
>>> >> > > >> > > Given the constraints, the most viable solution is an HTTPS
>>> >> > endpoint
>>> >> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
>>> >> > request.
>>> >> > > >> The
>>> >> > > >> > > simplest thing is to write each request in a log file,
>>> along with
>>> >> > > the
>>> >> > > >> > > timestamp, the version reported by the client, maybe the
>>> IP (not
>>> >> > > sure
>>> >> > > >> > about
>>> >> > > >> > > the ASF rules about this concerning privacy, I guess it's
>>> OK if
>>> >> > you
>>> >> > > >> make
>>> >> > > >> > it
>>> >> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the
>>> node
>>> >> > > creates
>>> >> > > >> on
>>> >> > > >> > > first startup or something.
>>> >> > > >> > >
>>> >> > > >> > > This endpoint would need some basic DDoS protection and
>>> >> > blacklisting
>>> >> > > >> to
>>> >> > > >> > > prevent data spoofing.
>>> >> > > >> > >
>>> >> > > >> > > If we'll be implementing this endpoint anyway, then
>>> there's no
>>> >> > point
>>> >> > > >> > > placing another file on Git or elsewhere for reporting the
>>> latest
>>> >> > > >> > versions:
>>> >> > > >> > > the endpoint itself can return them.
>>> >> > > >> > >
>>> >> > > >> > > WDYT?
>>> >> > > >> > >
>>> >> > > >> > > Cheers.
>>> >> > > >> > >
>>> >> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>> >> > > >> > dsetrakyan@apache.org <ma...@apache.org>>
>>> >> > > >> > > wrote:
>>> >> > > >> > >
>>> >> > > >> > > > Cos, Raul,
>>> >> > > >> > > >
>>> >> > > >> > > > Thanks for the feedback. I completely agree about Maven
>>> Central
>>> >> > > >> being a
>>> >> > > >> > > 3rd
>>> >> > > >> > > > party repo (did not think about that initially). All your
>>> >> > > >> suggestions
>>> >> > > >> > > make
>>> >> > > >> > > > sense, but I would like to keep it as simple as
>>> possible, and so
>>> >> > > far
>>> >> > > >> > > > everything suggested required GIT dependencies and extra
>>> work.
>>> >> > > >> > > >
>>> >> > > >> > > > How about Yakov's suggestion. We simply add a page to
>>> the Ignite
>>> >> > > >> > website
>>> >> > > >> > > > which will have only the latest version. Every time a
>>> node
>>> >> > starts,
>>> >> > > >> it
>>> >> > > >> > > > receives the latest version from the page and suggests
>>> that
>>> >> > users
>>> >> > > >> > upgrade
>>> >> > > >> > > > if needed.
>>> >> > > >> > > >
>>> >> > > >> > > > Also, since we have GA enabled for the website, we can
>>> track how
>>> >> > > >> many
>>> >> > > >> > > times
>>> >> > > >> > > > this page was accessed, which will be equal to the
>>> number of
>>> >> > > starts.
>>> >> > > >> > This
>>> >> > > >> > > > way, the counter information is tracked and monitored by
>>> the
>>> >> > > Ignite
>>> >> > > >> > PMC.
>>> >> > > >> > > >
>>> >> > > >> > > > This approach looks pretty innocent to me and everything
>>> is kept
>>> >> > > and
>>> >> > > >> > > > managed within Apache.
>>> >> > > >> > > >
>>> >> > > >> > > > Thoughts?
>>> >> > > >> > > >
>>> >> > > >> > > > D.
>>> >> > > >> > > >
>>> >> > > >> > > >
>>> >> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>> >> > > >> cos@apache.org <ma...@apache.org>>
>>> >> > > >> > > > wrote:
>>> >> > > >> > > >
>>> >> > > >> > > > > I agree with Raul.
>>> >> > > >> > > > > - providing a ping-back address to a 3rd party might
>>> be frown
>>> >> > > >> upon by
>>> >> > > >> > > > some.
>>> >> > > >> > > > >   And might have a consequences like stats collection
>>> about
>>> >> > > users'
>>> >> > > >> > > > >   infrastructure.
>>> >> > > >> > > > > - checking an ASF git-repo is easy and won't download
>>> any
>>> >> > binary
>>> >> > > >> > data:
>>> >> > > >> > > > >   everything is clear text and could be easily
>>> monitored by
>>> >> > any
>>> >> > > of
>>> >> > > >> > the
>>> >> > > >> > > > > network
>>> >> > > >> > > > >   diagnostic tools, shall it be required. But it
>>> involves a
>>> >> > bit
>>> >> > > of
>>> >> > > >> > the
>>> >> > > >> > > > > release
>>> >> > > >> > > > >   discipline.
>>> >> > > >> > > > > - the binary data download in the runtime is my main
>>> concern.
>>> >> > > >> This is
>>> >> > > >> > > the
>>> >> > > >> > > > >   vector of MMA. What if someone gains the control
>>> over the
>>> >> > > >> > repository
>>> >> > > >> > > > and
>>> >> > > >> > > > >   replaces the file with some malicious content.
>>> >> > > >> > > > >
>>> >> > > >> > > > > As for the particular mechanism: IIRC Ignite used to
>>> make a
>>> >> > call
>>> >> > > >> to
>>> >> > > >> > an
>>> >> > > >> > > > > external script to check upon the atest software
>>> version
>>> >> > > available
>>> >> > > >> > for
>>> >> > > >> > > > > download. In the past, the endpoint was running on a
>>> 3rd party
>>> >> > > >> > server,
>>> >> > > >> > > I
>>> >> > > >> > > > > believe the best way would be to put this script on
>>> ASF infra
>>> >> > > and
>>> >> > > >> > have
>>> >> > > >> > > > the
>>> >> > > >> > > > > "update checker" running in a completely controlled
>>> >> > environment.
>>> >> > > >> > > > Actually,
>>> >> > > >> > > > > I
>>> >> > > >> > > > > recall we had this very discussion during the
>>> Incubation; I
>>> >> > can
>>> >> > > >> > > probably
>>> >> > > >> > > > > dig
>>> >> > > >> > > > > out the corresponding thread.
>>> >> > > >> > > > >
>>> >> > > >> > > > > Thoughts?
>>> >> > > >> > > > >   Cok
>>> >> > > >> > > > >
>>> >> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>> >> > > >> > > > > > Hey guys
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > In my opinion, maven.org <http://maven.org/> is
>>> still owned by a third party
>>> >> > > >> > (Sonatype).
>>> >> > > >> > > > We
>>> >> > > >> > > > > > don't know what kind of data analysis or intelligence
>>> >> > > extraction
>>> >> > > >> > they
>>> >> > > >> > > > > run.
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > If Ignite servers all over the world were hitting
>>> maven.org <http://maven.org/>
>>> >> > > >> > > > periodically
>>> >> > > >> > > > > > asking for an Ignite Maven artifact, it gives
>>> Sonatype a
>>> >> > clear
>>> >> > > >> > > > indication
>>> >> > > >> > > > > > of who is running an Ignite server.
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > They could reverse lookup the IP address and find
>>> out what
>>> >> > > >> > > corporation
>>> >> > > >> > > > it
>>> >> > > >> > > > > > is.
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > How about having Ignite check the ASF Git directly?
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > We could use the Git ssh API, but that would require
>>> a new
>>> >> > > >> > > dependency,
>>> >> > > >> > > > > > which I advise against.
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > Alternatively, we could have it scrape this HTML for
>>> new Git
>>> >> > > >> tags:
>>> >> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>> <https://git-wip-us.apache.org/repos/asf?p=ignite.git>
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > Another option is to place a txt file in the root of
>>> the
>>> >> > > master
>>> >> > > >> > > branch
>>> >> > > >> > > > > (e.g
>>> >> > > >> > > > > > LATEST), containing a list of the latest GA versions
>>> for
>>> >> > each
>>> >> > > >> major
>>> >> > > >> > > > > version
>>> >> > > >> > > > > > line (1.x, 2.x).
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > I would advocate this last option, but it requires
>>> somebody
>>> >> > > >> > > remembering
>>> >> > > >> > > > > to
>>> >> > > >> > > > > > update the file with every release, unless we
>>> automate it
>>> >> > > with a
>>> >> > > >> > > Maven
>>> >> > > >> > > > > > plugin.
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > Hope that helps!
>>> >> > > >> > > > > >
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <
>>> dmagda@apache.org <ma...@apache.org>>
>>> >> > > wrote:
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > I see nothing wrong with this approach.
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you
>>> think? Is
>>> >> > it
>>> >> > > >> good
>>> >> > > >> > > to
>>> >> > > >> > > > > go?
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > —
>>> >> > > >> > > > > > Denis
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>> >> > > >> > > > dsetrakyan@apache.org <ma...@apache.org>
>>> >> > > >> > > > > >
>>> >> > > >> > > > > > wrote:
>>> >> > > >> > > > > > >
>>> >> > > >> > > > > > > Is everyone OK with this approach? Should I file a
>>> ticket
>>> >> > on
>>> >> > > >> it?
>>> >> > > >> > > > > > >
>>> >> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan
>>> <
>>> >> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org>>
>>> >> > > >> > > > > > > wrote:
>>> >> > > >> > > > > > >
>>> >> > > >> > > > > > >> Igniters,
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> There has been lots of talk of proposals about
>>> various
>>> >> > > usage
>>> >> > > >> > > metrics
>>> >> > > >> > > > > for
>>> >> > > >> > > > > > >> Ignite and nothing came of it. I would like to
>>> resurrect
>>> >> > > the
>>> >> > > >> > topic
>>> >> > > >> > > > and
>>> >> > > >> > > > > > >> propose something very simple and non-intrusive.
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> 1. Update Checker
>>> >> > > >> > > > > > >> The main purpose of the update checker is not to
>>> collect
>>> >> > > >> > metrics,
>>> >> > > >> > > > but
>>> >> > > >> > > > > to
>>> >> > > >> > > > > > >> notify users about a new version of Ignite by
>>> accessing
>>> >> > > >> > maven.org <http://maven.org/>
>>> >> > > >> > > > and
>>> >> > > >> > > > > > >> getting the version out of the metadata file:
>>> >> > > >> > > > > > >> http://repo2.maven.org/maven2/ <
>>> http://repo2.maven.org/maven2/>
>>> >> > > org/apache/ignite/ignite-core/
>>> >> > > >> > > > > > >> maven-metadata.xml
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> This way we do not send any information anywhere
>>> and, at
>>> >> > > the
>>> >> > > >> > same
>>> >> > > >> > > > > time,
>>> >> > > >> > > > > > >> urge our users to download and start using the
>>> latest
>>> >> > > >> version of
>>> >> > > >> > > > > Ignite.
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> 2. Startup Counter
>>> >> > > >> > > > > > >> This piece is optional, but we can also get an
>>> insight in
>>> >> > > how
>>> >> > > >> > many
>>> >> > > >> > > > > times
>>> >> > > >> > > > > > a
>>> >> > > >> > > > > > >> certain Ignite release gets started. This is just
>>> a cool
>>> >> > > >> metric
>>> >> > > >> > > for
>>> >> > > >> > > > > the
>>> >> > > >> > > > > > >> community to gauge the project popularity. You
>>> can think
>>> >> > of
>>> >> > > >> it
>>> >> > > >> > as
>>> >> > > >> > > > of a
>>> >> > > >> > > > > > page
>>> >> > > >> > > > > > >> visit counter shown on many websites. We can even
>>> decide
>>> >> > to
>>> >> > > >> > > display
>>> >> > > >> > > > > this
>>> >> > > >> > > > > > >> counter on the Ignite website as well.
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> To do this, we can simply add a JAR in maven for
>>> every
>>> >> > > >> release,
>>> >> > > >> > > e.g.
>>> >> > > >> > > > > > >> ignite-start-counter.jar, which will contain only
>>> 1 byte.
>>> >> > > >> Every
>>> >> > > >> > > time
>>> >> > > >> > > > > an
>>> >> > > >> > > > > > >> Ignite node starts, it will download this JAR in
>>> the
>>> >> > > >> background.
>>> >> > > >> > > > Then
>>> >> > > >> > > > > we
>>> >> > > >> > > > > > >> will be able to view the number of the total
>>> downloads
>>> >> > for
>>> >> > > >> this
>>> >> > > >> > > JAR
>>> >> > > >> > > > in
>>> >> > > >> > > > > > >> Maven Central, which is essentially the number of
>>> starts
>>> >> > of
>>> >> > > >> > Ignite
>>> >> > > >> > > > > nodes.
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> *Note that neither of the above suggestions
>>> require
>>> >> > Ignite
>>> >> > > to
>>> >> > > >> > send
>>> >> > > >> > > > or
>>> >> > > >> > > > > > >> track any user information whatsoever.*
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> Please reply suggesting weather you are OK with
>>> this
>>> >> > > >> approach.
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > > > >> D.
>>> >> > > >> > > > > > >>
>>> >> > > >> > > > >
>>> >> > > >> > > > >
>>> >> > > >> > > >
>>> >> > > >> > >
>>> >> > > >> >
>>> >> > > >>
>>> >> > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> Alexey Kuznetsov
>>
>>
>>
>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@apache.org>.

> I also did a global search on the Ignite website, but didn't find anything for googleads or doubleclick.  

Could you remove and add screencasts block temporary on your local deployment to see if the calls to commercial scripts reported by Cos appear in your Chrome dev toolkit?

—
Denis

> On Sep 29, 2017, at 3:56 PM, Prachi Garg <pg...@gridgain.com> wrote:
> 
> We use the following scripts - 
> 
> https://platform.twitter.com/widgets.js <https://platform.twitter.com/widgets.js> - used on homepage to display tweets
> https://static.addtoany.com/menu/page.js <https://static.addtoany.com/menu/page.js> - used on events page for social media sharing
> https://www.google-analytics.com/analytics.js <https://www.google-analytics.com/analytics.js>
> 
> I also did a global search on the Ignite website, but didn't find anything for googleads or doubleclick.  
> 
> -Prachi
> 
> 
> On Fri, Sep 29, 2017 at 11:03 AM, Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>> wrote:
> That’s definitely worthwhile checking. Prachi, as the one who embedded the screencast, would you check the theory?
> 
> —
> Denis
>> On Sep 28, 2017, at 11:50 PM, Alexey Kuznetsov <akuznetsov@apache.org <ma...@apache.org>> wrote:
>> 
>> Cos, Denis.
>> 
>> I think it is because we have embedded videos (on YouTube).
>> Mauricio or Denis, please check my idea.
>> 
>> On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik <cos@apache.org <ma...@apache.org>> wrote:
>> Sorry guys - I neglected the fact that our lists don't permit
>> attachments. I have put the screenshot to an external server [1]
>> 
>> [1] https://imgur.com/a/p9FJ9 <https://imgur.com/a/p9FJ9>
>> 
>> Thank you!
>> --
>>   With regards,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> 
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>> 
>> 
>> On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dmagda@apache.org <ma...@apache.org>> wrote:
>> > Cos,
>> >
>> > The screenshot was not attached. Could you share it some other way (google drive, etc.)? I’ve never seen any commercial on the site.
>> >
>> > —
>> > Denis
>> >
>> >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <cos@apache.org <ma...@apache.org>> wrote:
>> >>
>> >> I don't see an issue with node version either.
>> >>
>> >> Just one more, and it might be slightly irrelevant, issue though... I looked at the Ignite's site and found the following ads and trackers (that are indeed getting disabled by my browser).
>> >> Why are googleads or doubleclick are permitted?
>> >>
>> >>
>> >>
>> >> Thanks,
>> >>   Cos
>> >>
>> >>
>> >> --
>> >>   With regards,
>> >> Konstantin (Cos) Boudnik
>> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> >>
>> >> Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing.
>> >>
>> >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>> wrote:
>> >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vozerov@gridgain.com <ma...@gridgain.com> <mailto:vozerov@gridgain.com <ma...@gridgain.com>>>
>> >> wrote:
>> >>
>> >> > Folks,
>> >> >
>> >> > Can we add version of current node to web request? This way we will better
>> >> > understand version distribution, what might help us with certain API
>> >> > update/deprecate decisions
>> >> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <http://ignite.apache.org/latest.cgi&ver=2.2.0> <http://ignite.apache.org/latest.cgi&ver=2.2.0 <http://ignite.apache.org/latest.cgi&ver=2.2.0>>
>> >>
>> >>
>> >> Vladimir, I personally do not see a problem with that, as sending the
>> >> current version to the update checker seems very innocent to me. At the
>> >> same time, it will allow us to examine the usage of each release and make
>> >> decisions about dropping backward compatibility or spotting bugs for a
>> >> certain release.
>> >>
>> >> Cos, Raul, any thoughts?
>> >>
>> >>
>> >> >
>> >> >
>> >> > Vladimir.
>> >> >
>> >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
>> >> > wrote:
>> >> >
>> >> > > I think it is safe to assume at this point that everyone is in general
>> >> > > agreement, since there are no active objections.
>> >> > >
>> >> > > I have filed a ticket for the 2.3 release. Let's try to make it happen:
>> >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <https://issues.apache.org/jira/browse/IGNITE-6305> <https://issues.apache.org/jira/browse/IGNITE-6305 <https://issues.apache.org/jira/browse/IGNITE-6305>>
>> >> > >
>> >> > > D.
>> >> > >
>> >> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>> >> > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
>> >> > > wrote:
>> >> > >
>> >> > > >
>> >> > > >
>> >> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com <ma...@evosent.com> <mailto:raul.asf@evosent.com <ma...@evosent.com>>>
>> >> > > > wrote:
>> >> > > >
>> >> > > >> Yeah, I guess that's doable as well and requires less management
>> >> > effort
>> >> > > >> than my suggestion. We could use events [1] to store payload data
>> >> > (e.g.
>> >> > > >> IP,
>> >> > > >> version, etc.)
>> >> > > >
>> >> > > >
>> >> > > > Yes, we could use events or some other similar API provided by GA.
>> >> > > >
>> >> > > >
>> >> > > >> What the download page CGI developed in? PHP?
>> >> > > >>
>> >> > > >
>> >> > > > To be honest, no clue. I guess someone in the community can figure it
>> >> > > out:
>> >> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html> <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>>
>> >> > > >
>> >> > > >
>> >> > > >> However, I'm not sure whether storing this data in a 3rd party
>> >> > (Google)
>> >> > > is
>> >> > > >> compliant with the ASF policy. I guess it's no biggie, but if there's
>> >> > > >> doubt
>> >> > > >> in the PMC, it's better to ask legal@.
>> >> > > >
>> >> > > >
>> >> > > > I am not sure there is anything to ask about. The whole Ignite website
>> >> > is
>> >> > > > GA enabled, and all we are doing is accessing a standard web page from
>> >> > > the
>> >> > > > Ignite web site. The information gathered from GA is available only to
>> >> > > the
>> >> > > > Ignite PMC. Frankly, I think legal@ will be very confused by this
>> >> > > > question.
>> >> > > >
>> >> > > > Even ASF website itself uses GA: https://www.apache.org/ <https://www.apache.org/> <https://www.apache.org/ <https://www.apache.org/>>
>> >> > > > foundation/policies/privacy.ht <http://privacy.ht/>ml
>> >> > > >
>> >> > > >
>> >> > > >> I think Cos said it's OK; maybe Roman can pitch in.
>> >> > > >>
>> >> > > >
>> >> > > >  Sure, would be nice to hear from Roman as well.
>> >> > > >
>> >> > > >
>> >> > > >> Cheers.
>> >> > > >>
>> >> > > >> [1]
>> >> > > >> https://developers.google.com/analytics/devguides/collection <https://developers.google.com/analytics/devguides/collection> <https://developers.google.com/analytics/devguides/collection <https://developers.google.com/analytics/devguides/collection>>
>> >> > > >> /analyticsjs/events
>> >> > > >>
>> >> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>> >> > > >> dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
>> >> > > >> wrote:
>> >> > > >>
>> >> > > >> > Raul,
>> >> > > >> >
>> >> > > >> > Could point about Javascript, it will not work because it executes
>> >> > in
>> >> > > >> the
>> >> > > >> > browser. This means we need a server-side script, like CGI we are
>> >> > > using
>> >> > > >> on
>> >> > > >> > our download page.
>> >> > > >> >
>> >> > > >> > How about this approach. We create something like ignite-version.cgi
>> >> > > >> script
>> >> > > >> > which will invoke a call to GA and then return the latest version.
>> >> > > >> >
>> >> > > >> > This should work, right?
>> >> > > >> >
>> >> > > >> > D.
>> >> > > >> >
>> >> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>> >> > raul.asf@evosent.com <ma...@evosent.com> <mailto:raul.asf@evosent.com <ma...@evosent.com>>
>> >> > > >
>> >> > > >> > wrote:
>> >> > > >> >
>> >> > > >> > > Hey Dmitriy and all
>> >> > > >> > >
>> >> > > >> > > Also, since we have GA enabled for the website, we can track how
>> >> > > many
>> >> > > >> > times
>> >> > > >> > > > this page was accessed, which will be equal to the number of
>> >> > > starts.
>> >> > > >> > This
>> >> > > >> > > > way, the counter information is tracked and monitored by the
>> >> > > Ignite
>> >> > > >> > PMC.
>> >> > > >> > >
>> >> > > >> > >
>> >> > > >> > > Unfortunately this won't work because GA is loaded via Javascript
>> >> > on
>> >> > > >> the
>> >> > > >> > > browser, so Google will never receive the page hit.
>> >> > > >> > >
>> >> > > >> > > Given the constraints, the most viable solution is an HTTPS
>> >> > endpoint
>> >> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
>> >> > request.
>> >> > > >> The
>> >> > > >> > > simplest thing is to write each request in a log file, along with
>> >> > > the
>> >> > > >> > > timestamp, the version reported by the client, maybe the IP (not
>> >> > > sure
>> >> > > >> > about
>> >> > > >> > > the ASF rules about this concerning privacy, I guess it's OK if
>> >> > you
>> >> > > >> make
>> >> > > >> > it
>> >> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
>> >> > > creates
>> >> > > >> on
>> >> > > >> > > first startup or something.
>> >> > > >> > >
>> >> > > >> > > This endpoint would need some basic DDoS protection and
>> >> > blacklisting
>> >> > > >> to
>> >> > > >> > > prevent data spoofing.
>> >> > > >> > >
>> >> > > >> > > If we'll be implementing this endpoint anyway, then there's no
>> >> > point
>> >> > > >> > > placing another file on Git or elsewhere for reporting the latest
>> >> > > >> > versions:
>> >> > > >> > > the endpoint itself can return them.
>> >> > > >> > >
>> >> > > >> > > WDYT?
>> >> > > >> > >
>> >> > > >> > > Cheers.
>> >> > > >> > >
>> >> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>> >> > > >> > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
>> >> > > >> > > wrote:
>> >> > > >> > >
>> >> > > >> > > > Cos, Raul,
>> >> > > >> > > >
>> >> > > >> > > > Thanks for the feedback. I completely agree about Maven Central
>> >> > > >> being a
>> >> > > >> > > 3rd
>> >> > > >> > > > party repo (did not think about that initially). All your
>> >> > > >> suggestions
>> >> > > >> > > make
>> >> > > >> > > > sense, but I would like to keep it as simple as possible, and so
>> >> > > far
>> >> > > >> > > > everything suggested required GIT dependencies and extra work.
>> >> > > >> > > >
>> >> > > >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
>> >> > > >> > website
>> >> > > >> > > > which will have only the latest version. Every time a node
>> >> > starts,
>> >> > > >> it
>> >> > > >> > > > receives the latest version from the page and suggests that
>> >> > users
>> >> > > >> > upgrade
>> >> > > >> > > > if needed.
>> >> > > >> > > >
>> >> > > >> > > > Also, since we have GA enabled for the website, we can track how
>> >> > > >> many
>> >> > > >> > > times
>> >> > > >> > > > this page was accessed, which will be equal to the number of
>> >> > > starts.
>> >> > > >> > This
>> >> > > >> > > > way, the counter information is tracked and monitored by the
>> >> > > Ignite
>> >> > > >> > PMC.
>> >> > > >> > > >
>> >> > > >> > > > This approach looks pretty innocent to me and everything is kept
>> >> > > and
>> >> > > >> > > > managed within Apache.
>> >> > > >> > > >
>> >> > > >> > > > Thoughts?
>> >> > > >> > > >
>> >> > > >> > > > D.
>> >> > > >> > > >
>> >> > > >> > > >
>> >> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>> >> > > >> cos@apache.org <ma...@apache.org> <mailto:cos@apache.org <ma...@apache.org>>>
>> >> > > >> > > > wrote:
>> >> > > >> > > >
>> >> > > >> > > > > I agree with Raul.
>> >> > > >> > > > > - providing a ping-back address to a 3rd party might be frown
>> >> > > >> upon by
>> >> > > >> > > > some.
>> >> > > >> > > > >   And might have a consequences like stats collection about
>> >> > > users'
>> >> > > >> > > > >   infrastructure.
>> >> > > >> > > > > - checking an ASF git-repo is easy and won't download any
>> >> > binary
>> >> > > >> > data:
>> >> > > >> > > > >   everything is clear text and could be easily monitored by
>> >> > any
>> >> > > of
>> >> > > >> > the
>> >> > > >> > > > > network
>> >> > > >> > > > >   diagnostic tools, shall it be required. But it involves a
>> >> > bit
>> >> > > of
>> >> > > >> > the
>> >> > > >> > > > > release
>> >> > > >> > > > >   discipline.
>> >> > > >> > > > > - the binary data download in the runtime is my main concern.
>> >> > > >> This is
>> >> > > >> > > the
>> >> > > >> > > > >   vector of MMA. What if someone gains the control over the
>> >> > > >> > repository
>> >> > > >> > > > and
>> >> > > >> > > > >   replaces the file with some malicious content.
>> >> > > >> > > > >
>> >> > > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
>> >> > call
>> >> > > >> to
>> >> > > >> > an
>> >> > > >> > > > > external script to check upon the atest software version
>> >> > > available
>> >> > > >> > for
>> >> > > >> > > > > download. In the past, the endpoint was running on a 3rd party
>> >> > > >> > server,
>> >> > > >> > > I
>> >> > > >> > > > > believe the best way would be to put this script on ASF infra
>> >> > > and
>> >> > > >> > have
>> >> > > >> > > > the
>> >> > > >> > > > > "update checker" running in a completely controlled
>> >> > environment.
>> >> > > >> > > > Actually,
>> >> > > >> > > > > I
>> >> > > >> > > > > recall we had this very discussion during the Incubation; I
>> >> > can
>> >> > > >> > > probably
>> >> > > >> > > > > dig
>> >> > > >> > > > > out the corresponding thread.
>> >> > > >> > > > >
>> >> > > >> > > > > Thoughts?
>> >> > > >> > > > >   Cok
>> >> > > >> > > > >
>> >> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> >> > > >> > > > > > Hey guys
>> >> > > >> > > > > >
>> >> > > >> > > > > > In my opinion, maven.org <http://maven.org/> <http://maven.org/ <http://maven.org/>> is still owned by a third party
>> >> > > >> > (Sonatype).
>> >> > > >> > > > We
>> >> > > >> > > > > > don't know what kind of data analysis or intelligence
>> >> > > extraction
>> >> > > >> > they
>> >> > > >> > > > > run.
>> >> > > >> > > > > >
>> >> > > >> > > > > > If Ignite servers all over the world were hitting maven.org <http://maven.org/> <http://maven.org/ <http://maven.org/>>
>> >> > > >> > > > periodically
>> >> > > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
>> >> > clear
>> >> > > >> > > > indication
>> >> > > >> > > > > > of who is running an Ignite server.
>> >> > > >> > > > > >
>> >> > > >> > > > > > They could reverse lookup the IP address and find out what
>> >> > > >> > > corporation
>> >> > > >> > > > it
>> >> > > >> > > > > > is.
>> >> > > >> > > > > >
>> >> > > >> > > > > > How about having Ignite check the ASF Git directly?
>> >> > > >> > > > > >
>> >> > > >> > > > > > We could use the Git ssh API, but that would require a new
>> >> > > >> > > dependency,
>> >> > > >> > > > > > which I advise against.
>> >> > > >> > > > > >
>> >> > > >> > > > > > Alternatively, we could have it scrape this HTML for new Git
>> >> > > >> tags:
>> >> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git <https://git-wip-us.apache.org/repos/asf?p=ignite.git> <https://git-wip-us.apache.org/repos/asf?p=ignite.git <https://git-wip-us.apache.org/repos/asf?p=ignite.git>>
>> >> > > >> > > > > >
>> >> > > >> > > > > > Another option is to place a txt file in the root of the
>> >> > > master
>> >> > > >> > > branch
>> >> > > >> > > > > (e.g
>> >> > > >> > > > > > LATEST), containing a list of the latest GA versions for
>> >> > each
>> >> > > >> major
>> >> > > >> > > > > version
>> >> > > >> > > > > > line (1.x, 2.x).
>> >> > > >> > > > > >
>> >> > > >> > > > > > I would advocate this last option, but it requires somebody
>> >> > > >> > > remembering
>> >> > > >> > > > > to
>> >> > > >> > > > > > update the file with every release, unless we automate it
>> >> > > with a
>> >> > > >> > > Maven
>> >> > > >> > > > > > plugin.
>> >> > > >> > > > > >
>> >> > > >> > > > > > Hope that helps!
>> >> > > >> > > > > >
>> >> > > >> > > > > >
>> >> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org <ma...@apache.org> <mailto:dmagda@apache.org <ma...@apache.org>>>
>> >> > > wrote:
>> >> > > >> > > > > >
>> >> > > >> > > > > > I see nothing wrong with this approach.
>> >> > > >> > > > > >
>> >> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is
>> >> > it
>> >> > > >> good
>> >> > > >> > > to
>> >> > > >> > > > > go?
>> >> > > >> > > > > >
>> >> > > >> > > > > > —
>> >> > > >> > > > > > Denis
>> >> > > >> > > > > >
>> >> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>> >> > > >> > > > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>
>> >> > > >> > > > > >
>> >> > > >> > > > > > wrote:
>> >> > > >> > > > > > >
>> >> > > >> > > > > > > Is everyone OK with this approach? Should I file a ticket
>> >> > on
>> >> > > >> it?
>> >> > > >> > > > > > >
>> >> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> >> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
>> >> > > >> > > > > > > wrote:
>> >> > > >> > > > > > >
>> >> > > >> > > > > > >> Igniters,
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> There has been lots of talk of proposals about various
>> >> > > usage
>> >> > > >> > > metrics
>> >> > > >> > > > > for
>> >> > > >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
>> >> > > the
>> >> > > >> > topic
>> >> > > >> > > > and
>> >> > > >> > > > > > >> propose something very simple and non-intrusive.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> 1. Update Checker
>> >> > > >> > > > > > >> The main purpose of the update checker is not to collect
>> >> > > >> > metrics,
>> >> > > >> > > > but
>> >> > > >> > > > > to
>> >> > > >> > > > > > >> notify users about a new version of Ignite by accessing
>> >> > > >> > maven.org <http://maven.org/> <http://maven.org/ <http://maven.org/>>
>> >> > > >> > > > and
>> >> > > >> > > > > > >> getting the version out of the metadata file:
>> >> > > >> > > > > > >> http://repo2.maven.org/maven2/ <http://repo2.maven.org/maven2/> <http://repo2.maven.org/maven2/ <http://repo2.maven.org/maven2/>>
>> >> > > org/apache/ignite/ignite-core/
>> >> > > >> > > > > > >> maven-metadata.xml
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> This way we do not send any information anywhere and, at
>> >> > > the
>> >> > > >> > same
>> >> > > >> > > > > time,
>> >> > > >> > > > > > >> urge our users to download and start using the latest
>> >> > > >> version of
>> >> > > >> > > > > Ignite.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> 2. Startup Counter
>> >> > > >> > > > > > >> This piece is optional, but we can also get an insight in
>> >> > > how
>> >> > > >> > many
>> >> > > >> > > > > times
>> >> > > >> > > > > > a
>> >> > > >> > > > > > >> certain Ignite release gets started. This is just a cool
>> >> > > >> metric
>> >> > > >> > > for
>> >> > > >> > > > > the
>> >> > > >> > > > > > >> community to gauge the project popularity. You can think
>> >> > of
>> >> > > >> it
>> >> > > >> > as
>> >> > > >> > > > of a
>> >> > > >> > > > > > page
>> >> > > >> > > > > > >> visit counter shown on many websites. We can even decide
>> >> > to
>> >> > > >> > > display
>> >> > > >> > > > > this
>> >> > > >> > > > > > >> counter on the Ignite website as well.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> To do this, we can simply add a JAR in maven for every
>> >> > > >> release,
>> >> > > >> > > e.g.
>> >> > > >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
>> >> > > >> Every
>> >> > > >> > > time
>> >> > > >> > > > > an
>> >> > > >> > > > > > >> Ignite node starts, it will download this JAR in the
>> >> > > >> background.
>> >> > > >> > > > Then
>> >> > > >> > > > > we
>> >> > > >> > > > > > >> will be able to view the number of the total downloads
>> >> > for
>> >> > > >> this
>> >> > > >> > > JAR
>> >> > > >> > > > in
>> >> > > >> > > > > > >> Maven Central, which is essentially the number of starts
>> >> > of
>> >> > > >> > Ignite
>> >> > > >> > > > > nodes.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> *Note that neither of the above suggestions require
>> >> > Ignite
>> >> > > to
>> >> > > >> > send
>> >> > > >> > > > or
>> >> > > >> > > > > > >> track any user information whatsoever.*
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> Please reply suggesting weather you are OK with this
>> >> > > >> approach.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> D.
>> >> > > >> > > > > > >>
>> >> > > >> > > > >
>> >> > > >> > > > >
>> >> > > >> > > >
>> >> > > >> > >
>> >> > > >> >
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> 
>> 
>> 
>> -- 
>> Alexey Kuznetsov
> 
>

Re: [DISCUSS] Ignite Update Checker

Posted by Prachi Garg <pg...@gridgain.com>.

We use the following scripts -

https://platform.twitter.com/widgets.js - used on homepage to display tweets
https://static.addtoany.com/menu/page.js - used on events page for social
media sharing
https://www.google-analytics.com/analytics.js

I also did a global search on the Ignite website, but didn't find anything
for googleads or doubleclick.

-Prachi


On Fri, Sep 29, 2017 at 11:03 AM, Denis Magda <dm...@gridgain.com> wrote:

> That’s definitely worthwhile checking. Prachi, as the one who embedded the
> screencast, would you check the theory?
>
> —
> Denis
>
> On Sep 28, 2017, at 11:50 PM, Alexey Kuznetsov <ak...@apache.org>
> wrote:
>
> Cos, Denis.
>
> I think it is because we have embedded videos (on YouTube).
> Mauricio or Denis, please check my idea.
>
> On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
>
>> Sorry guys - I neglected the fact that our lists don't permit
>> attachments. I have put the screenshot to an external server [1]
>>
>> [1] https://imgur.com/a/p9FJ9
>>
>> Thank you!
>> --
>>   With regards,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dm...@apache.org> wrote:
>> > Cos,
>> >
>> > The screenshot was not attached. Could you share it some other way
>> (google drive, etc.)? I’ve never seen any commercial on the site.
>> >
>> > —
>> > Denis
>> >
>> >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> >>
>> >> I don't see an issue with node version either.
>> >>
>> >> Just one more, and it might be slightly irrelevant, issue though... I
>> looked at the Ignite's site and found the following ads and trackers (that
>> are indeed getting disabled by my browser).
>> >> Why are googleads or doubleclick are permitted?
>> >>
>> >>
>> >>
>> >> Thanks,
>> >>   Cos
>> >>
>> >>
>> >> --
>> >>   With regards,
>> >> Konstantin (Cos) Boudnik
>> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> >>
>> >> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author might
>> be affiliated with at the moment of writing.
>> >>
>> >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org <ma...@apache.org>> wrote:
>> >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vozerov@gridgain.com
>> <ma...@gridgain.com>>
>> >> wrote:
>> >>
>> >> > Folks,
>> >> >
>> >> > Can we add version of current node to web request? This way we will
>> better
>> >> > understand version distribution, what might help us with certain API
>> >> > update/deprecate decisions
>> >> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <
>> http://ignite.apache.org/latest.cgi&ver=2.2.0>
>> >>
>> >>
>> >> Vladimir, I personally do not see a problem with that, as sending the
>> >> current version to the update checker seems very innocent to me. At the
>> >> same time, it will allow us to examine the usage of each release and
>> make
>> >> decisions about dropping backward compatibility or spotting bugs for a
>> >> certain release.
>> >>
>> >> Cos, Raul, any thoughts?
>> >>
>> >>
>> >> >
>> >> >
>> >> > Vladimir.
>> >> >
>> >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org <ma...@apache.org>>
>> >> > wrote:
>> >> >
>> >> > > I think it is safe to assume at this point that everyone is in
>> general
>> >> > > agreement, since there are no active objections.
>> >> > >
>> >> > > I have filed a ticket for the 2.3 release. Let's try to make it
>> happen:
>> >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <
>> https://issues.apache.org/jira/browse/IGNITE-6305>
>> >> > >
>> >> > > D.
>> >> > >
>> >> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>> >> > dsetrakyan@apache.org <ma...@apache.org>>
>> >> > > wrote:
>> >> > >
>> >> > > >
>> >> > > >
>> >> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>> raul.asf@evosent.com <ma...@evosent.com>>
>> >> > > > wrote:
>> >> > > >
>> >> > > >> Yeah, I guess that's doable as well and requires less management
>> >> > effort
>> >> > > >> than my suggestion. We could use events [1] to store payload
>> data
>> >> > (e.g.
>> >> > > >> IP,
>> >> > > >> version, etc.)
>> >> > > >
>> >> > > >
>> >> > > > Yes, we could use events or some other similar API provided by
>> GA.
>> >> > > >
>> >> > > >
>> >> > > >> What the download page CGI developed in? PHP?
>> >> > > >>
>> >> > > >
>> >> > > > To be honest, no clue. I guess someone in the community can
>> figure it
>> >> > > out:
>> >> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>> <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>
>> >> > > >
>> >> > > >
>> >> > > >> However, I'm not sure whether storing this data in a 3rd party
>> >> > (Google)
>> >> > > is
>> >> > > >> compliant with the ASF policy. I guess it's no biggie, but if
>> there's
>> >> > > >> doubt
>> >> > > >> in the PMC, it's better to ask legal@.
>> >> > > >
>> >> > > >
>> >> > > > I am not sure there is anything to ask about. The whole Ignite
>> website
>> >> > is
>> >> > > > GA enabled, and all we are doing is accessing a standard web
>> page from
>> >> > > the
>> >> > > > Ignite web site. The information gathered from GA is available
>> only to
>> >> > > the
>> >> > > > Ignite PMC. Frankly, I think legal@ will be very confused by
>> this
>> >> > > > question.
>> >> > > >
>> >> > > > Even ASF website itself uses GA: https://www.apache.org/ <
>> https://www.apache.org/>
>> >> > > > foundation/policies/privacy.html
>> >> > > >
>> >> > > >
>> >> > > >> I think Cos said it's OK; maybe Roman can pitch in.
>> >> > > >>
>> >> > > >
>> >> > > >  Sure, would be nice to hear from Roman as well.
>> >> > > >
>> >> > > >
>> >> > > >> Cheers.
>> >> > > >>
>> >> > > >> [1]
>> >> > > >> https://developers.google.com/analytics/devguides/collection <
>> https://developers.google.com/analytics/devguides/collection>
>> >> > > >> /analyticsjs/events
>> >> > > >>
>> >> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>> >> > > >> dsetrakyan@apache.org <ma...@apache.org>>
>> >> > > >> wrote:
>> >> > > >>
>> >> > > >> > Raul,
>> >> > > >> >
>> >> > > >> > Could point about Javascript, it will not work because it
>> executes
>> >> > in
>> >> > > >> the
>> >> > > >> > browser. This means we need a server-side script, like CGI we
>> are
>> >> > > using
>> >> > > >> on
>> >> > > >> > our download page.
>> >> > > >> >
>> >> > > >> > How about this approach. We create something like
>> ignite-version.cgi
>> >> > > >> script
>> >> > > >> > which will invoke a call to GA and then return the latest
>> version.
>> >> > > >> >
>> >> > > >> > This should work, right?
>> >> > > >> >
>> >> > > >> > D.
>> >> > > >> >
>> >> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>> >> > raul.asf@evosent.com <ma...@evosent.com>
>> >> > > >
>> >> > > >> > wrote:
>> >> > > >> >
>> >> > > >> > > Hey Dmitriy and all
>> >> > > >> > >
>> >> > > >> > > Also, since we have GA enabled for the website, we can
>> track how
>> >> > > many
>> >> > > >> > times
>> >> > > >> > > > this page was accessed, which will be equal to the number
>> of
>> >> > > starts.
>> >> > > >> > This
>> >> > > >> > > > way, the counter information is tracked and monitored by
>> the
>> >> > > Ignite
>> >> > > >> > PMC.
>> >> > > >> > >
>> >> > > >> > >
>> >> > > >> > > Unfortunately this won't work because GA is loaded via
>> Javascript
>> >> > on
>> >> > > >> the
>> >> > > >> > > browser, so Google will never receive the page hit.
>> >> > > >> > >
>> >> > > >> > > Given the constraints, the most viable solution is an HTTPS
>> >> > endpoint
>> >> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
>> >> > request.
>> >> > > >> The
>> >> > > >> > > simplest thing is to write each request in a log file,
>> along with
>> >> > > the
>> >> > > >> > > timestamp, the version reported by the client, maybe the IP
>> (not
>> >> > > sure
>> >> > > >> > about
>> >> > > >> > > the ASF rules about this concerning privacy, I guess it's
>> OK if
>> >> > you
>> >> > > >> make
>> >> > > >> > it
>> >> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the
>> node
>> >> > > creates
>> >> > > >> on
>> >> > > >> > > first startup or something.
>> >> > > >> > >
>> >> > > >> > > This endpoint would need some basic DDoS protection and
>> >> > blacklisting
>> >> > > >> to
>> >> > > >> > > prevent data spoofing.
>> >> > > >> > >
>> >> > > >> > > If we'll be implementing this endpoint anyway, then there's
>> no
>> >> > point
>> >> > > >> > > placing another file on Git or elsewhere for reporting the
>> latest
>> >> > > >> > versions:
>> >> > > >> > > the endpoint itself can return them.
>> >> > > >> > >
>> >> > > >> > > WDYT?
>> >> > > >> > >
>> >> > > >> > > Cheers.
>> >> > > >> > >
>> >> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>> >> > > >> > dsetrakyan@apache.org <ma...@apache.org>>
>> >> > > >> > > wrote:
>> >> > > >> > >
>> >> > > >> > > > Cos, Raul,
>> >> > > >> > > >
>> >> > > >> > > > Thanks for the feedback. I completely agree about Maven
>> Central
>> >> > > >> being a
>> >> > > >> > > 3rd
>> >> > > >> > > > party repo (did not think about that initially). All your
>> >> > > >> suggestions
>> >> > > >> > > make
>> >> > > >> > > > sense, but I would like to keep it as simple as possible,
>> and so
>> >> > > far
>> >> > > >> > > > everything suggested required GIT dependencies and extra
>> work.
>> >> > > >> > > >
>> >> > > >> > > > How about Yakov's suggestion. We simply add a page to the
>> Ignite
>> >> > > >> > website
>> >> > > >> > > > which will have only the latest version. Every time a node
>> >> > starts,
>> >> > > >> it
>> >> > > >> > > > receives the latest version from the page and suggests
>> that
>> >> > users
>> >> > > >> > upgrade
>> >> > > >> > > > if needed.
>> >> > > >> > > >
>> >> > > >> > > > Also, since we have GA enabled for the website, we can
>> track how
>> >> > > >> many
>> >> > > >> > > times
>> >> > > >> > > > this page was accessed, which will be equal to the number
>> of
>> >> > > starts.
>> >> > > >> > This
>> >> > > >> > > > way, the counter information is tracked and monitored by
>> the
>> >> > > Ignite
>> >> > > >> > PMC.
>> >> > > >> > > >
>> >> > > >> > > > This approach looks pretty innocent to me and everything
>> is kept
>> >> > > and
>> >> > > >> > > > managed within Apache.
>> >> > > >> > > >
>> >> > > >> > > > Thoughts?
>> >> > > >> > > >
>> >> > > >> > > > D.
>> >> > > >> > > >
>> >> > > >> > > >
>> >> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>> >> > > >> cos@apache.org <ma...@apache.org>>
>> >> > > >> > > > wrote:
>> >> > > >> > > >
>> >> > > >> > > > > I agree with Raul.
>> >> > > >> > > > > - providing a ping-back address to a 3rd party might be
>> frown
>> >> > > >> upon by
>> >> > > >> > > > some.
>> >> > > >> > > > >   And might have a consequences like stats collection
>> about
>> >> > > users'
>> >> > > >> > > > >   infrastructure.
>> >> > > >> > > > > - checking an ASF git-repo is easy and won't download
>> any
>> >> > binary
>> >> > > >> > data:
>> >> > > >> > > > >   everything is clear text and could be easily
>> monitored by
>> >> > any
>> >> > > of
>> >> > > >> > the
>> >> > > >> > > > > network
>> >> > > >> > > > >   diagnostic tools, shall it be required. But it
>> involves a
>> >> > bit
>> >> > > of
>> >> > > >> > the
>> >> > > >> > > > > release
>> >> > > >> > > > >   discipline.
>> >> > > >> > > > > - the binary data download in the runtime is my main
>> concern.
>> >> > > >> This is
>> >> > > >> > > the
>> >> > > >> > > > >   vector of MMA. What if someone gains the control over
>> the
>> >> > > >> > repository
>> >> > > >> > > > and
>> >> > > >> > > > >   replaces the file with some malicious content.
>> >> > > >> > > > >
>> >> > > >> > > > > As for the particular mechanism: IIRC Ignite used to
>> make a
>> >> > call
>> >> > > >> to
>> >> > > >> > an
>> >> > > >> > > > > external script to check upon the atest software version
>> >> > > available
>> >> > > >> > for
>> >> > > >> > > > > download. In the past, the endpoint was running on a
>> 3rd party
>> >> > > >> > server,
>> >> > > >> > > I
>> >> > > >> > > > > believe the best way would be to put this script on ASF
>> infra
>> >> > > and
>> >> > > >> > have
>> >> > > >> > > > the
>> >> > > >> > > > > "update checker" running in a completely controlled
>> >> > environment.
>> >> > > >> > > > Actually,
>> >> > > >> > > > > I
>> >> > > >> > > > > recall we had this very discussion during the
>> Incubation; I
>> >> > can
>> >> > > >> > > probably
>> >> > > >> > > > > dig
>> >> > > >> > > > > out the corresponding thread.
>> >> > > >> > > > >
>> >> > > >> > > > > Thoughts?
>> >> > > >> > > > >   Cok
>> >> > > >> > > > >
>> >> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> >> > > >> > > > > > Hey guys
>> >> > > >> > > > > >
>> >> > > >> > > > > > In my opinion, maven.org <http://maven.org/> is
>> still owned by a third party
>> >> > > >> > (Sonatype).
>> >> > > >> > > > We
>> >> > > >> > > > > > don't know what kind of data analysis or intelligence
>> >> > > extraction
>> >> > > >> > they
>> >> > > >> > > > > run.
>> >> > > >> > > > > >
>> >> > > >> > > > > > If Ignite servers all over the world were hitting
>> maven.org <http://maven.org/>
>> >> > > >> > > > periodically
>> >> > > >> > > > > > asking for an Ignite Maven artifact, it gives
>> Sonatype a
>> >> > clear
>> >> > > >> > > > indication
>> >> > > >> > > > > > of who is running an Ignite server.
>> >> > > >> > > > > >
>> >> > > >> > > > > > They could reverse lookup the IP address and find out
>> what
>> >> > > >> > > corporation
>> >> > > >> > > > it
>> >> > > >> > > > > > is.
>> >> > > >> > > > > >
>> >> > > >> > > > > > How about having Ignite check the ASF Git directly?
>> >> > > >> > > > > >
>> >> > > >> > > > > > We could use the Git ssh API, but that would require
>> a new
>> >> > > >> > > dependency,
>> >> > > >> > > > > > which I advise against.
>> >> > > >> > > > > >
>> >> > > >> > > > > > Alternatively, we could have it scrape this HTML for
>> new Git
>> >> > > >> tags:
>> >> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
>> <https://git-wip-us.apache.org/repos/asf?p=ignite.git>
>> >> > > >> > > > > >
>> >> > > >> > > > > > Another option is to place a txt file in the root of
>> the
>> >> > > master
>> >> > > >> > > branch
>> >> > > >> > > > > (e.g
>> >> > > >> > > > > > LATEST), containing a list of the latest GA versions
>> for
>> >> > each
>> >> > > >> major
>> >> > > >> > > > > version
>> >> > > >> > > > > > line (1.x, 2.x).
>> >> > > >> > > > > >
>> >> > > >> > > > > > I would advocate this last option, but it requires
>> somebody
>> >> > > >> > > remembering
>> >> > > >> > > > > to
>> >> > > >> > > > > > update the file with every release, unless we
>> automate it
>> >> > > with a
>> >> > > >> > > Maven
>> >> > > >> > > > > > plugin.
>> >> > > >> > > > > >
>> >> > > >> > > > > > Hope that helps!
>> >> > > >> > > > > >
>> >> > > >> > > > > >
>> >> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <
>> dmagda@apache.org <ma...@apache.org>>
>> >> > > wrote:
>> >> > > >> > > > > >
>> >> > > >> > > > > > I see nothing wrong with this approach.
>> >> > > >> > > > > >
>> >> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you
>> think? Is
>> >> > it
>> >> > > >> good
>> >> > > >> > > to
>> >> > > >> > > > > go?
>> >> > > >> > > > > >
>> >> > > >> > > > > > —
>> >> > > >> > > > > > Denis
>> >> > > >> > > > > >
>> >> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>> >> > > >> > > > dsetrakyan@apache.org <ma...@apache.org>
>> >> > > >> > > > > >
>> >> > > >> > > > > > wrote:
>> >> > > >> > > > > > >
>> >> > > >> > > > > > > Is everyone OK with this approach? Should I file a
>> ticket
>> >> > on
>> >> > > >> it?
>> >> > > >> > > > > > >
>> >> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> >> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org>>
>> >> > > >> > > > > > > wrote:
>> >> > > >> > > > > > >
>> >> > > >> > > > > > >> Igniters,
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> There has been lots of talk of proposals about
>> various
>> >> > > usage
>> >> > > >> > > metrics
>> >> > > >> > > > > for
>> >> > > >> > > > > > >> Ignite and nothing came of it. I would like to
>> resurrect
>> >> > > the
>> >> > > >> > topic
>> >> > > >> > > > and
>> >> > > >> > > > > > >> propose something very simple and non-intrusive.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> 1. Update Checker
>> >> > > >> > > > > > >> The main purpose of the update checker is not to
>> collect
>> >> > > >> > metrics,
>> >> > > >> > > > but
>> >> > > >> > > > > to
>> >> > > >> > > > > > >> notify users about a new version of Ignite by
>> accessing
>> >> > > >> > maven.org <http://maven.org/>
>> >> > > >> > > > and
>> >> > > >> > > > > > >> getting the version out of the metadata file:
>> >> > > >> > > > > > >> http://repo2.maven.org/maven2/ <
>> http://repo2.maven.org/maven2/>
>> >> > > org/apache/ignite/ignite-core/
>> >> > > >> > > > > > >> maven-metadata.xml
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> This way we do not send any information anywhere
>> and, at
>> >> > > the
>> >> > > >> > same
>> >> > > >> > > > > time,
>> >> > > >> > > > > > >> urge our users to download and start using the
>> latest
>> >> > > >> version of
>> >> > > >> > > > > Ignite.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> 2. Startup Counter
>> >> > > >> > > > > > >> This piece is optional, but we can also get an
>> insight in
>> >> > > how
>> >> > > >> > many
>> >> > > >> > > > > times
>> >> > > >> > > > > > a
>> >> > > >> > > > > > >> certain Ignite release gets started. This is just
>> a cool
>> >> > > >> metric
>> >> > > >> > > for
>> >> > > >> > > > > the
>> >> > > >> > > > > > >> community to gauge the project popularity. You can
>> think
>> >> > of
>> >> > > >> it
>> >> > > >> > as
>> >> > > >> > > > of a
>> >> > > >> > > > > > page
>> >> > > >> > > > > > >> visit counter shown on many websites. We can even
>> decide
>> >> > to
>> >> > > >> > > display
>> >> > > >> > > > > this
>> >> > > >> > > > > > >> counter on the Ignite website as well.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> To do this, we can simply add a JAR in maven for
>> every
>> >> > > >> release,
>> >> > > >> > > e.g.
>> >> > > >> > > > > > >> ignite-start-counter.jar, which will contain only
>> 1 byte.
>> >> > > >> Every
>> >> > > >> > > time
>> >> > > >> > > > > an
>> >> > > >> > > > > > >> Ignite node starts, it will download this JAR in
>> the
>> >> > > >> background.
>> >> > > >> > > > Then
>> >> > > >> > > > > we
>> >> > > >> > > > > > >> will be able to view the number of the total
>> downloads
>> >> > for
>> >> > > >> this
>> >> > > >> > > JAR
>> >> > > >> > > > in
>> >> > > >> > > > > > >> Maven Central, which is essentially the number of
>> starts
>> >> > of
>> >> > > >> > Ignite
>> >> > > >> > > > > nodes.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> *Note that neither of the above suggestions require
>> >> > Ignite
>> >> > > to
>> >> > > >> > send
>> >> > > >> > > > or
>> >> > > >> > > > > > >> track any user information whatsoever.*
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> Please reply suggesting weather you are OK with
>> this
>> >> > > >> approach.
>> >> > > >> > > > > > >>
>> >> > > >> > > > > > >> D.
>> >> > > >> > > > > > >>
>> >> > > >> > > > >
>> >> > > >> > > > >
>> >> > > >> > > >
>> >> > > >> > >
>> >> > > >> >
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
>
>
> --
> Alexey Kuznetsov
>
>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@gridgain.com>.

That’s definitely worthwhile checking. Prachi, as the one who embedded the screencast, would you check the theory?

—
Denis
> On Sep 28, 2017, at 11:50 PM, Alexey Kuznetsov <ak...@apache.org> wrote:
> 
> Cos, Denis.
> 
> I think it is because we have embedded videos (on YouTube).
> Mauricio or Denis, please check my idea.
> 
> On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik <cos@apache.org <ma...@apache.org>> wrote:
> Sorry guys - I neglected the fact that our lists don't permit
> attachments. I have put the screenshot to an external server [1]
> 
> [1] https://imgur.com/a/p9FJ9 <https://imgur.com/a/p9FJ9>
> 
> Thank you!
> --
>   With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> 
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
> 
> 
> On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dmagda@apache.org <ma...@apache.org>> wrote:
> > Cos,
> >
> > The screenshot was not attached. Could you share it some other way (google drive, etc.)? I’ve never seen any commercial on the site.
> >
> > —
> > Denis
> >
> >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <cos@apache.org <ma...@apache.org>> wrote:
> >>
> >> I don't see an issue with node version either.
> >>
> >> Just one more, and it might be slightly irrelevant, issue though... I looked at the Ignite's site and found the following ads and trackers (that are indeed getting disabled by my browser).
> >> Why are googleads or doubleclick are permitted?
> >>
> >>
> >>
> >> Thanks,
> >>   Cos
> >>
> >>
> >> --
> >>   With regards,
> >> Konstantin (Cos) Boudnik
> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>
> >> Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing.
> >>
> >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>> wrote:
> >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vozerov@gridgain.com <ma...@gridgain.com> <mailto:vozerov@gridgain.com <ma...@gridgain.com>>>
> >> wrote:
> >>
> >> > Folks,
> >> >
> >> > Can we add version of current node to web request? This way we will better
> >> > understand version distribution, what might help us with certain API
> >> > update/deprecate decisions
> >> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <http://ignite.apache.org/latest.cgi&ver=2.2.0> <http://ignite.apache.org/latest.cgi&ver=2.2.0 <http://ignite.apache.org/latest.cgi&ver=2.2.0>>
> >>
> >>
> >> Vladimir, I personally do not see a problem with that, as sending the
> >> current version to the update checker seems very innocent to me. At the
> >> same time, it will allow us to examine the usage of each release and make
> >> decisions about dropping backward compatibility or spotting bugs for a
> >> certain release.
> >>
> >> Cos, Raul, any thoughts?
> >>
> >>
> >> >
> >> >
> >> > Vladimir.
> >> >
> >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
> >> > wrote:
> >> >
> >> > > I think it is safe to assume at this point that everyone is in general
> >> > > agreement, since there are no active objections.
> >> > >
> >> > > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <https://issues.apache.org/jira/browse/IGNITE-6305> <https://issues.apache.org/jira/browse/IGNITE-6305 <https://issues.apache.org/jira/browse/IGNITE-6305>>
> >> > >
> >> > > D.
> >> > >
> >> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >> > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
> >> > > wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com <ma...@evosent.com> <mailto:raul.asf@evosent.com <ma...@evosent.com>>>
> >> > > > wrote:
> >> > > >
> >> > > >> Yeah, I guess that's doable as well and requires less management
> >> > effort
> >> > > >> than my suggestion. We could use events [1] to store payload data
> >> > (e.g.
> >> > > >> IP,
> >> > > >> version, etc.)
> >> > > >
> >> > > >
> >> > > > Yes, we could use events or some other similar API provided by GA.
> >> > > >
> >> > > >
> >> > > >> What the download page CGI developed in? PHP?
> >> > > >>
> >> > > >
> >> > > > To be honest, no clue. I guess someone in the community can figure it
> >> > > out:
> >> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html> <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>>
> >> > > >
> >> > > >
> >> > > >> However, I'm not sure whether storing this data in a 3rd party
> >> > (Google)
> >> > > is
> >> > > >> compliant with the ASF policy. I guess it's no biggie, but if there's
> >> > > >> doubt
> >> > > >> in the PMC, it's better to ask legal@.
> >> > > >
> >> > > >
> >> > > > I am not sure there is anything to ask about. The whole Ignite website
> >> > is
> >> > > > GA enabled, and all we are doing is accessing a standard web page from
> >> > > the
> >> > > > Ignite web site. The information gathered from GA is available only to
> >> > > the
> >> > > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> >> > > > question.
> >> > > >
> >> > > > Even ASF website itself uses GA: https://www.apache.org/ <https://www.apache.org/> <https://www.apache.org/ <https://www.apache.org/>>
> >> > > > foundation/policies/privacy.html
> >> > > >
> >> > > >
> >> > > >> I think Cos said it's OK; maybe Roman can pitch in.
> >> > > >>
> >> > > >
> >> > > >  Sure, would be nice to hear from Roman as well.
> >> > > >
> >> > > >
> >> > > >> Cheers.
> >> > > >>
> >> > > >> [1]
> >> > > >> https://developers.google.com/analytics/devguides/collection <https://developers.google.com/analytics/devguides/collection> <https://developers.google.com/analytics/devguides/collection <https://developers.google.com/analytics/devguides/collection>>
> >> > > >> /analyticsjs/events
> >> > > >>
> >> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >> > > >> dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Raul,
> >> > > >> >
> >> > > >> > Could point about Javascript, it will not work because it executes
> >> > in
> >> > > >> the
> >> > > >> > browser. This means we need a server-side script, like CGI we are
> >> > > using
> >> > > >> on
> >> > > >> > our download page.
> >> > > >> >
> >> > > >> > How about this approach. We create something like ignite-version.cgi
> >> > > >> script
> >> > > >> > which will invoke a call to GA and then return the latest version.
> >> > > >> >
> >> > > >> > This should work, right?
> >> > > >> >
> >> > > >> > D.
> >> > > >> >
> >> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> >> > raul.asf@evosent.com <ma...@evosent.com> <mailto:raul.asf@evosent.com <ma...@evosent.com>>
> >> > > >
> >> > > >> > wrote:
> >> > > >> >
> >> > > >> > > Hey Dmitriy and all
> >> > > >> > >
> >> > > >> > > Also, since we have GA enabled for the website, we can track how
> >> > > many
> >> > > >> > times
> >> > > >> > > > this page was accessed, which will be equal to the number of
> >> > > starts.
> >> > > >> > This
> >> > > >> > > > way, the counter information is tracked and monitored by the
> >> > > Ignite
> >> > > >> > PMC.
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > Unfortunately this won't work because GA is loaded via Javascript
> >> > on
> >> > > >> the
> >> > > >> > > browser, so Google will never receive the page hit.
> >> > > >> > >
> >> > > >> > > Given the constraints, the most viable solution is an HTTPS
> >> > endpoint
> >> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
> >> > request.
> >> > > >> The
> >> > > >> > > simplest thing is to write each request in a log file, along with
> >> > > the
> >> > > >> > > timestamp, the version reported by the client, maybe the IP (not
> >> > > sure
> >> > > >> > about
> >> > > >> > > the ASF rules about this concerning privacy, I guess it's OK if
> >> > you
> >> > > >> make
> >> > > >> > it
> >> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> >> > > creates
> >> > > >> on
> >> > > >> > > first startup or something.
> >> > > >> > >
> >> > > >> > > This endpoint would need some basic DDoS protection and
> >> > blacklisting
> >> > > >> to
> >> > > >> > > prevent data spoofing.
> >> > > >> > >
> >> > > >> > > If we'll be implementing this endpoint anyway, then there's no
> >> > point
> >> > > >> > > placing another file on Git or elsewhere for reporting the latest
> >> > > >> > versions:
> >> > > >> > > the endpoint itself can return them.
> >> > > >> > >
> >> > > >> > > WDYT?
> >> > > >> > >
> >> > > >> > > Cheers.
> >> > > >> > >
> >> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >> > > >> > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
> >> > > >> > > wrote:
> >> > > >> > >
> >> > > >> > > > Cos, Raul,
> >> > > >> > > >
> >> > > >> > > > Thanks for the feedback. I completely agree about Maven Central
> >> > > >> being a
> >> > > >> > > 3rd
> >> > > >> > > > party repo (did not think about that initially). All your
> >> > > >> suggestions
> >> > > >> > > make
> >> > > >> > > > sense, but I would like to keep it as simple as possible, and so
> >> > > far
> >> > > >> > > > everything suggested required GIT dependencies and extra work.
> >> > > >> > > >
> >> > > >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> >> > > >> > website
> >> > > >> > > > which will have only the latest version. Every time a node
> >> > starts,
> >> > > >> it
> >> > > >> > > > receives the latest version from the page and suggests that
> >> > users
> >> > > >> > upgrade
> >> > > >> > > > if needed.
> >> > > >> > > >
> >> > > >> > > > Also, since we have GA enabled for the website, we can track how
> >> > > >> many
> >> > > >> > > times
> >> > > >> > > > this page was accessed, which will be equal to the number of
> >> > > starts.
> >> > > >> > This
> >> > > >> > > > way, the counter information is tracked and monitored by the
> >> > > Ignite
> >> > > >> > PMC.
> >> > > >> > > >
> >> > > >> > > > This approach looks pretty innocent to me and everything is kept
> >> > > and
> >> > > >> > > > managed within Apache.
> >> > > >> > > >
> >> > > >> > > > Thoughts?
> >> > > >> > > >
> >> > > >> > > > D.
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >> > > >> cos@apache.org <ma...@apache.org> <mailto:cos@apache.org <ma...@apache.org>>>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > > I agree with Raul.
> >> > > >> > > > > - providing a ping-back address to a 3rd party might be frown
> >> > > >> upon by
> >> > > >> > > > some.
> >> > > >> > > > >   And might have a consequences like stats collection about
> >> > > users'
> >> > > >> > > > >   infrastructure.
> >> > > >> > > > > - checking an ASF git-repo is easy and won't download any
> >> > binary
> >> > > >> > data:
> >> > > >> > > > >   everything is clear text and could be easily monitored by
> >> > any
> >> > > of
> >> > > >> > the
> >> > > >> > > > > network
> >> > > >> > > > >   diagnostic tools, shall it be required. But it involves a
> >> > bit
> >> > > of
> >> > > >> > the
> >> > > >> > > > > release
> >> > > >> > > > >   discipline.
> >> > > >> > > > > - the binary data download in the runtime is my main concern.
> >> > > >> This is
> >> > > >> > > the
> >> > > >> > > > >   vector of MMA. What if someone gains the control over the
> >> > > >> > repository
> >> > > >> > > > and
> >> > > >> > > > >   replaces the file with some malicious content.
> >> > > >> > > > >
> >> > > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
> >> > call
> >> > > >> to
> >> > > >> > an
> >> > > >> > > > > external script to check upon the atest software version
> >> > > available
> >> > > >> > for
> >> > > >> > > > > download. In the past, the endpoint was running on a 3rd party
> >> > > >> > server,
> >> > > >> > > I
> >> > > >> > > > > believe the best way would be to put this script on ASF infra
> >> > > and
> >> > > >> > have
> >> > > >> > > > the
> >> > > >> > > > > "update checker" running in a completely controlled
> >> > environment.
> >> > > >> > > > Actually,
> >> > > >> > > > > I
> >> > > >> > > > > recall we had this very discussion during the Incubation; I
> >> > can
> >> > > >> > > probably
> >> > > >> > > > > dig
> >> > > >> > > > > out the corresponding thread.
> >> > > >> > > > >
> >> > > >> > > > > Thoughts?
> >> > > >> > > > >   Cok
> >> > > >> > > > >
> >> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >> > > >> > > > > > Hey guys
> >> > > >> > > > > >
> >> > > >> > > > > > In my opinion, maven.org <http://maven.org/> <http://maven.org/ <http://maven.org/>> is still owned by a third party
> >> > > >> > (Sonatype).
> >> > > >> > > > We
> >> > > >> > > > > > don't know what kind of data analysis or intelligence
> >> > > extraction
> >> > > >> > they
> >> > > >> > > > > run.
> >> > > >> > > > > >
> >> > > >> > > > > > If Ignite servers all over the world were hitting maven.org <http://maven.org/> <http://maven.org/ <http://maven.org/>>
> >> > > >> > > > periodically
> >> > > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
> >> > clear
> >> > > >> > > > indication
> >> > > >> > > > > > of who is running an Ignite server.
> >> > > >> > > > > >
> >> > > >> > > > > > They could reverse lookup the IP address and find out what
> >> > > >> > > corporation
> >> > > >> > > > it
> >> > > >> > > > > > is.
> >> > > >> > > > > >
> >> > > >> > > > > > How about having Ignite check the ASF Git directly?
> >> > > >> > > > > >
> >> > > >> > > > > > We could use the Git ssh API, but that would require a new
> >> > > >> > > dependency,
> >> > > >> > > > > > which I advise against.
> >> > > >> > > > > >
> >> > > >> > > > > > Alternatively, we could have it scrape this HTML for new Git
> >> > > >> tags:
> >> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git <https://git-wip-us.apache.org/repos/asf?p=ignite.git> <https://git-wip-us.apache.org/repos/asf?p=ignite.git <https://git-wip-us.apache.org/repos/asf?p=ignite.git>>
> >> > > >> > > > > >
> >> > > >> > > > > > Another option is to place a txt file in the root of the
> >> > > master
> >> > > >> > > branch
> >> > > >> > > > > (e.g
> >> > > >> > > > > > LATEST), containing a list of the latest GA versions for
> >> > each
> >> > > >> major
> >> > > >> > > > > version
> >> > > >> > > > > > line (1.x, 2.x).
> >> > > >> > > > > >
> >> > > >> > > > > > I would advocate this last option, but it requires somebody
> >> > > >> > > remembering
> >> > > >> > > > > to
> >> > > >> > > > > > update the file with every release, unless we automate it
> >> > > with a
> >> > > >> > > Maven
> >> > > >> > > > > > plugin.
> >> > > >> > > > > >
> >> > > >> > > > > > Hope that helps!
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org <ma...@apache.org> <mailto:dmagda@apache.org <ma...@apache.org>>>
> >> > > wrote:
> >> > > >> > > > > >
> >> > > >> > > > > > I see nothing wrong with this approach.
> >> > > >> > > > > >
> >> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is
> >> > it
> >> > > >> good
> >> > > >> > > to
> >> > > >> > > > > go?
> >> > > >> > > > > >
> >> > > >> > > > > > —
> >> > > >> > > > > > Denis
> >> > > >> > > > > >
> >> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >> > > >> > > > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>
> >> > > >> > > > > >
> >> > > >> > > > > > wrote:
> >> > > >> > > > > > >
> >> > > >> > > > > > > Is everyone OK with this approach? Should I file a ticket
> >> > on
> >> > > >> it?
> >> > > >> > > > > > >
> >> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org> <mailto:dsetrakyan@apache.org <ma...@apache.org>>>
> >> > > >> > > > > > > wrote:
> >> > > >> > > > > > >
> >> > > >> > > > > > >> Igniters,
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> There has been lots of talk of proposals about various
> >> > > usage
> >> > > >> > > metrics
> >> > > >> > > > > for
> >> > > >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
> >> > > the
> >> > > >> > topic
> >> > > >> > > > and
> >> > > >> > > > > > >> propose something very simple and non-intrusive.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> 1. Update Checker
> >> > > >> > > > > > >> The main purpose of the update checker is not to collect
> >> > > >> > metrics,
> >> > > >> > > > but
> >> > > >> > > > > to
> >> > > >> > > > > > >> notify users about a new version of Ignite by accessing
> >> > > >> > maven.org <http://maven.org/> <http://maven.org/ <http://maven.org/>>
> >> > > >> > > > and
> >> > > >> > > > > > >> getting the version out of the metadata file:
> >> > > >> > > > > > >> http://repo2.maven.org/maven2/ <http://repo2.maven.org/maven2/> <http://repo2.maven.org/maven2/ <http://repo2.maven.org/maven2/>>
> >> > > org/apache/ignite/ignite-core/
> >> > > >> > > > > > >> maven-metadata.xml
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> This way we do not send any information anywhere and, at
> >> > > the
> >> > > >> > same
> >> > > >> > > > > time,
> >> > > >> > > > > > >> urge our users to download and start using the latest
> >> > > >> version of
> >> > > >> > > > > Ignite.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> 2. Startup Counter
> >> > > >> > > > > > >> This piece is optional, but we can also get an insight in
> >> > > how
> >> > > >> > many
> >> > > >> > > > > times
> >> > > >> > > > > > a
> >> > > >> > > > > > >> certain Ignite release gets started. This is just a cool
> >> > > >> metric
> >> > > >> > > for
> >> > > >> > > > > the
> >> > > >> > > > > > >> community to gauge the project popularity. You can think
> >> > of
> >> > > >> it
> >> > > >> > as
> >> > > >> > > > of a
> >> > > >> > > > > > page
> >> > > >> > > > > > >> visit counter shown on many websites. We can even decide
> >> > to
> >> > > >> > > display
> >> > > >> > > > > this
> >> > > >> > > > > > >> counter on the Ignite website as well.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> To do this, we can simply add a JAR in maven for every
> >> > > >> release,
> >> > > >> > > e.g.
> >> > > >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> >> > > >> Every
> >> > > >> > > time
> >> > > >> > > > > an
> >> > > >> > > > > > >> Ignite node starts, it will download this JAR in the
> >> > > >> background.
> >> > > >> > > > Then
> >> > > >> > > > > we
> >> > > >> > > > > > >> will be able to view the number of the total downloads
> >> > for
> >> > > >> this
> >> > > >> > > JAR
> >> > > >> > > > in
> >> > > >> > > > > > >> Maven Central, which is essentially the number of starts
> >> > of
> >> > > >> > Ignite
> >> > > >> > > > > nodes.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> *Note that neither of the above suggestions require
> >> > Ignite
> >> > > to
> >> > > >> > send
> >> > > >> > > > or
> >> > > >> > > > > > >> track any user information whatsoever.*
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> Please reply suggesting weather you are OK with this
> >> > > >> approach.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> D.
> >> > > >> > > > > > >>
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> 
> 
> 
> -- 
> Alexey Kuznetsov

Re: [DISCUSS] Ignite Update Checker

Posted by Alexey Kuznetsov <ak...@apache.org>.

Cos, Denis.

I think it is because we have embedded videos (on YouTube).
Mauricio or Denis, please check my idea.

On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik <co...@apache.org> wrote:

> Sorry guys - I neglected the fact that our lists don't permit
> attachments. I have put the screenshot to an external server [1]
>
> [1] https://imgur.com/a/p9FJ9
>
> Thank you!
> --
>   With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dm...@apache.org> wrote:
> > Cos,
> >
> > The screenshot was not attached. Could you share it some other way
> (google drive, etc.)? I’ve never seen any commercial on the site.
> >
> > —
> > Denis
> >
> >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <co...@apache.org> wrote:
> >>
> >> I don't see an issue with node version either.
> >>
> >> Just one more, and it might be slightly irrelevant, issue though... I
> looked at the Ignite's site and found the following ads and trackers (that
> are indeed getting disabled by my browser).
> >> Why are googleads or doubleclick are permitted?
> >>
> >>
> >>
> >> Thanks,
> >>   Cos
> >>
> >>
> >> --
> >>   With regards,
> >> Konstantin (Cos) Boudnik
> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>
> >> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author might
> be affiliated with at the moment of writing.
> >>
> >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org <ma...@apache.org>> wrote:
> >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vozerov@gridgain.com
> <ma...@gridgain.com>>
> >> wrote:
> >>
> >> > Folks,
> >> >
> >> > Can we add version of current node to web request? This way we will
> better
> >> > understand version distribution, what might help us with certain API
> >> > update/deprecate decisions
> >> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <
> http://ignite.apache.org/latest.cgi&ver=2.2.0>
> >>
> >>
> >> Vladimir, I personally do not see a problem with that, as sending the
> >> current version to the update checker seems very innocent to me. At the
> >> same time, it will allow us to examine the usage of each release and
> make
> >> decisions about dropping backward compatibility or spotting bugs for a
> >> certain release.
> >>
> >> Cos, Raul, any thoughts?
> >>
> >>
> >> >
> >> >
> >> > Vladimir.
> >> >
> >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <
> dsetrakyan@apache.org <ma...@apache.org>>
> >> > wrote:
> >> >
> >> > > I think it is safe to assume at this point that everyone is in
> general
> >> > > agreement, since there are no active objections.
> >> > >
> >> > > I have filed a ticket for the 2.3 release. Let's try to make it
> happen:
> >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <
> https://issues.apache.org/jira/browse/IGNITE-6305>
> >> > >
> >> > > D.
> >> > >
> >> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >> > dsetrakyan@apache.org <ma...@apache.org>>
> >> > > wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> raul.asf@evosent.com <ma...@evosent.com>>
> >> > > > wrote:
> >> > > >
> >> > > >> Yeah, I guess that's doable as well and requires less management
> >> > effort
> >> > > >> than my suggestion. We could use events [1] to store payload data
> >> > (e.g.
> >> > > >> IP,
> >> > > >> version, etc.)
> >> > > >
> >> > > >
> >> > > > Yes, we could use events or some other similar API provided by GA.
> >> > > >
> >> > > >
> >> > > >> What the download page CGI developed in? PHP?
> >> > > >>
> >> > > >
> >> > > > To be honest, no clue. I guess someone in the community can
> figure it
> >> > > out:
> >> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>
> >> > > >
> >> > > >
> >> > > >> However, I'm not sure whether storing this data in a 3rd party
> >> > (Google)
> >> > > is
> >> > > >> compliant with the ASF policy. I guess it's no biggie, but if
> there's
> >> > > >> doubt
> >> > > >> in the PMC, it's better to ask legal@.
> >> > > >
> >> > > >
> >> > > > I am not sure there is anything to ask about. The whole Ignite
> website
> >> > is
> >> > > > GA enabled, and all we are doing is accessing a standard web page
> from
> >> > > the
> >> > > > Ignite web site. The information gathered from GA is available
> only to
> >> > > the
> >> > > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> >> > > > question.
> >> > > >
> >> > > > Even ASF website itself uses GA: https://www.apache.org/ <
> https://www.apache.org/>
> >> > > > foundation/policies/privacy.html
> >> > > >
> >> > > >
> >> > > >> I think Cos said it's OK; maybe Roman can pitch in.
> >> > > >>
> >> > > >
> >> > > >  Sure, would be nice to hear from Roman as well.
> >> > > >
> >> > > >
> >> > > >> Cheers.
> >> > > >>
> >> > > >> [1]
> >> > > >> https://developers.google.com/analytics/devguides/collection <
> https://developers.google.com/analytics/devguides/collection>
> >> > > >> /analyticsjs/events
> >> > > >>
> >> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >> > > >> dsetrakyan@apache.org <ma...@apache.org>>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Raul,
> >> > > >> >
> >> > > >> > Could point about Javascript, it will not work because it
> executes
> >> > in
> >> > > >> the
> >> > > >> > browser. This means we need a server-side script, like CGI we
> are
> >> > > using
> >> > > >> on
> >> > > >> > our download page.
> >> > > >> >
> >> > > >> > How about this approach. We create something like
> ignite-version.cgi
> >> > > >> script
> >> > > >> > which will invoke a call to GA and then return the latest
> version.
> >> > > >> >
> >> > > >> > This should work, right?
> >> > > >> >
> >> > > >> > D.
> >> > > >> >
> >> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> >> > raul.asf@evosent.com <ma...@evosent.com>
> >> > > >
> >> > > >> > wrote:
> >> > > >> >
> >> > > >> > > Hey Dmitriy and all
> >> > > >> > >
> >> > > >> > > Also, since we have GA enabled for the website, we can track
> how
> >> > > many
> >> > > >> > times
> >> > > >> > > > this page was accessed, which will be equal to the number
> of
> >> > > starts.
> >> > > >> > This
> >> > > >> > > > way, the counter information is tracked and monitored by
> the
> >> > > Ignite
> >> > > >> > PMC.
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > Unfortunately this won't work because GA is loaded via
> Javascript
> >> > on
> >> > > >> the
> >> > > >> > > browser, so Google will never receive the page hit.
> >> > > >> > >
> >> > > >> > > Given the constraints, the most viable solution is an HTTPS
> >> > endpoint
> >> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
> >> > request.
> >> > > >> The
> >> > > >> > > simplest thing is to write each request in a log file, along
> with
> >> > > the
> >> > > >> > > timestamp, the version reported by the client, maybe the IP
> (not
> >> > > sure
> >> > > >> > about
> >> > > >> > > the ASF rules about this concerning privacy, I guess it's OK
> if
> >> > you
> >> > > >> make
> >> > > >> > it
> >> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> >> > > creates
> >> > > >> on
> >> > > >> > > first startup or something.
> >> > > >> > >
> >> > > >> > > This endpoint would need some basic DDoS protection and
> >> > blacklisting
> >> > > >> to
> >> > > >> > > prevent data spoofing.
> >> > > >> > >
> >> > > >> > > If we'll be implementing this endpoint anyway, then there's
> no
> >> > point
> >> > > >> > > placing another file on Git or elsewhere for reporting the
> latest
> >> > > >> > versions:
> >> > > >> > > the endpoint itself can return them.
> >> > > >> > >
> >> > > >> > > WDYT?
> >> > > >> > >
> >> > > >> > > Cheers.
> >> > > >> > >
> >> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >> > > >> > dsetrakyan@apache.org <ma...@apache.org>>
> >> > > >> > > wrote:
> >> > > >> > >
> >> > > >> > > > Cos, Raul,
> >> > > >> > > >
> >> > > >> > > > Thanks for the feedback. I completely agree about Maven
> Central
> >> > > >> being a
> >> > > >> > > 3rd
> >> > > >> > > > party repo (did not think about that initially). All your
> >> > > >> suggestions
> >> > > >> > > make
> >> > > >> > > > sense, but I would like to keep it as simple as possible,
> and so
> >> > > far
> >> > > >> > > > everything suggested required GIT dependencies and extra
> work.
> >> > > >> > > >
> >> > > >> > > > How about Yakov's suggestion. We simply add a page to the
> Ignite
> >> > > >> > website
> >> > > >> > > > which will have only the latest version. Every time a node
> >> > starts,
> >> > > >> it
> >> > > >> > > > receives the latest version from the page and suggests that
> >> > users
> >> > > >> > upgrade
> >> > > >> > > > if needed.
> >> > > >> > > >
> >> > > >> > > > Also, since we have GA enabled for the website, we can
> track how
> >> > > >> many
> >> > > >> > > times
> >> > > >> > > > this page was accessed, which will be equal to the number
> of
> >> > > starts.
> >> > > >> > This
> >> > > >> > > > way, the counter information is tracked and monitored by
> the
> >> > > Ignite
> >> > > >> > PMC.
> >> > > >> > > >
> >> > > >> > > > This approach looks pretty innocent to me and everything
> is kept
> >> > > and
> >> > > >> > > > managed within Apache.
> >> > > >> > > >
> >> > > >> > > > Thoughts?
> >> > > >> > > >
> >> > > >> > > > D.
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >> > > >> cos@apache.org <ma...@apache.org>>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > > I agree with Raul.
> >> > > >> > > > > - providing a ping-back address to a 3rd party might be
> frown
> >> > > >> upon by
> >> > > >> > > > some.
> >> > > >> > > > >   And might have a consequences like stats collection
> about
> >> > > users'
> >> > > >> > > > >   infrastructure.
> >> > > >> > > > > - checking an ASF git-repo is easy and won't download any
> >> > binary
> >> > > >> > data:
> >> > > >> > > > >   everything is clear text and could be easily monitored
> by
> >> > any
> >> > > of
> >> > > >> > the
> >> > > >> > > > > network
> >> > > >> > > > >   diagnostic tools, shall it be required. But it
> involves a
> >> > bit
> >> > > of
> >> > > >> > the
> >> > > >> > > > > release
> >> > > >> > > > >   discipline.
> >> > > >> > > > > - the binary data download in the runtime is my main
> concern.
> >> > > >> This is
> >> > > >> > > the
> >> > > >> > > > >   vector of MMA. What if someone gains the control over
> the
> >> > > >> > repository
> >> > > >> > > > and
> >> > > >> > > > >   replaces the file with some malicious content.
> >> > > >> > > > >
> >> > > >> > > > > As for the particular mechanism: IIRC Ignite used to
> make a
> >> > call
> >> > > >> to
> >> > > >> > an
> >> > > >> > > > > external script to check upon the atest software version
> >> > > available
> >> > > >> > for
> >> > > >> > > > > download. In the past, the endpoint was running on a 3rd
> party
> >> > > >> > server,
> >> > > >> > > I
> >> > > >> > > > > believe the best way would be to put this script on ASF
> infra
> >> > > and
> >> > > >> > have
> >> > > >> > > > the
> >> > > >> > > > > "update checker" running in a completely controlled
> >> > environment.
> >> > > >> > > > Actually,
> >> > > >> > > > > I
> >> > > >> > > > > recall we had this very discussion during the
> Incubation; I
> >> > can
> >> > > >> > > probably
> >> > > >> > > > > dig
> >> > > >> > > > > out the corresponding thread.
> >> > > >> > > > >
> >> > > >> > > > > Thoughts?
> >> > > >> > > > >   Cok
> >> > > >> > > > >
> >> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >> > > >> > > > > > Hey guys
> >> > > >> > > > > >
> >> > > >> > > > > > In my opinion, maven.org <http://maven.org/> is still
> owned by a third party
> >> > > >> > (Sonatype).
> >> > > >> > > > We
> >> > > >> > > > > > don't know what kind of data analysis or intelligence
> >> > > extraction
> >> > > >> > they
> >> > > >> > > > > run.
> >> > > >> > > > > >
> >> > > >> > > > > > If Ignite servers all over the world were hitting
> maven.org <http://maven.org/>
> >> > > >> > > > periodically
> >> > > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype
> a
> >> > clear
> >> > > >> > > > indication
> >> > > >> > > > > > of who is running an Ignite server.
> >> > > >> > > > > >
> >> > > >> > > > > > They could reverse lookup the IP address and find out
> what
> >> > > >> > > corporation
> >> > > >> > > > it
> >> > > >> > > > > > is.
> >> > > >> > > > > >
> >> > > >> > > > > > How about having Ignite check the ASF Git directly?
> >> > > >> > > > > >
> >> > > >> > > > > > We could use the Git ssh API, but that would require a
> new
> >> > > >> > > dependency,
> >> > > >> > > > > > which I advise against.
> >> > > >> > > > > >
> >> > > >> > > > > > Alternatively, we could have it scrape this HTML for
> new Git
> >> > > >> tags:
> >> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git <
> https://git-wip-us.apache.org/repos/asf?p=ignite.git>
> >> > > >> > > > > >
> >> > > >> > > > > > Another option is to place a txt file in the root of
> the
> >> > > master
> >> > > >> > > branch
> >> > > >> > > > > (e.g
> >> > > >> > > > > > LATEST), containing a list of the latest GA versions
> for
> >> > each
> >> > > >> major
> >> > > >> > > > > version
> >> > > >> > > > > > line (1.x, 2.x).
> >> > > >> > > > > >
> >> > > >> > > > > > I would advocate this last option, but it requires
> somebody
> >> > > >> > > remembering
> >> > > >> > > > > to
> >> > > >> > > > > > update the file with every release, unless we automate
> it
> >> > > with a
> >> > > >> > > Maven
> >> > > >> > > > > > plugin.
> >> > > >> > > > > >
> >> > > >> > > > > > Hope that helps!
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> <ma...@apache.org>>
> >> > > wrote:
> >> > > >> > > > > >
> >> > > >> > > > > > I see nothing wrong with this approach.
> >> > > >> > > > > >
> >> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you
> think? Is
> >> > it
> >> > > >> good
> >> > > >> > > to
> >> > > >> > > > > go?
> >> > > >> > > > > >
> >> > > >> > > > > > —
> >> > > >> > > > > > Denis
> >> > > >> > > > > >
> >> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >> > > >> > > > dsetrakyan@apache.org <ma...@apache.org>
> >> > > >> > > > > >
> >> > > >> > > > > > wrote:
> >> > > >> > > > > > >
> >> > > >> > > > > > > Is everyone OK with this approach? Should I file a
> ticket
> >> > on
> >> > > >> it?
> >> > > >> > > > > > >
> >> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org>>
> >> > > >> > > > > > > wrote:
> >> > > >> > > > > > >
> >> > > >> > > > > > >> Igniters,
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> There has been lots of talk of proposals about
> various
> >> > > usage
> >> > > >> > > metrics
> >> > > >> > > > > for
> >> > > >> > > > > > >> Ignite and nothing came of it. I would like to
> resurrect
> >> > > the
> >> > > >> > topic
> >> > > >> > > > and
> >> > > >> > > > > > >> propose something very simple and non-intrusive.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> 1. Update Checker
> >> > > >> > > > > > >> The main purpose of the update checker is not to
> collect
> >> > > >> > metrics,
> >> > > >> > > > but
> >> > > >> > > > > to
> >> > > >> > > > > > >> notify users about a new version of Ignite by
> accessing
> >> > > >> > maven.org <http://maven.org/>
> >> > > >> > > > and
> >> > > >> > > > > > >> getting the version out of the metadata file:
> >> > > >> > > > > > >> http://repo2.maven.org/maven2/ <
> http://repo2.maven.org/maven2/>
> >> > > org/apache/ignite/ignite-core/
> >> > > >> > > > > > >> maven-metadata.xml
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> This way we do not send any information anywhere
> and, at
> >> > > the
> >> > > >> > same
> >> > > >> > > > > time,
> >> > > >> > > > > > >> urge our users to download and start using the
> latest
> >> > > >> version of
> >> > > >> > > > > Ignite.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> 2. Startup Counter
> >> > > >> > > > > > >> This piece is optional, but we can also get an
> insight in
> >> > > how
> >> > > >> > many
> >> > > >> > > > > times
> >> > > >> > > > > > a
> >> > > >> > > > > > >> certain Ignite release gets started. This is just a
> cool
> >> > > >> metric
> >> > > >> > > for
> >> > > >> > > > > the
> >> > > >> > > > > > >> community to gauge the project popularity. You can
> think
> >> > of
> >> > > >> it
> >> > > >> > as
> >> > > >> > > > of a
> >> > > >> > > > > > page
> >> > > >> > > > > > >> visit counter shown on many websites. We can even
> decide
> >> > to
> >> > > >> > > display
> >> > > >> > > > > this
> >> > > >> > > > > > >> counter on the Ignite website as well.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> To do this, we can simply add a JAR in maven for
> every
> >> > > >> release,
> >> > > >> > > e.g.
> >> > > >> > > > > > >> ignite-start-counter.jar, which will contain only 1
> byte.
> >> > > >> Every
> >> > > >> > > time
> >> > > >> > > > > an
> >> > > >> > > > > > >> Ignite node starts, it will download this JAR in the
> >> > > >> background.
> >> > > >> > > > Then
> >> > > >> > > > > we
> >> > > >> > > > > > >> will be able to view the number of the total
> downloads
> >> > for
> >> > > >> this
> >> > > >> > > JAR
> >> > > >> > > > in
> >> > > >> > > > > > >> Maven Central, which is essentially the number of
> starts
> >> > of
> >> > > >> > Ignite
> >> > > >> > > > > nodes.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> *Note that neither of the above suggestions require
> >> > Ignite
> >> > > to
> >> > > >> > send
> >> > > >> > > > or
> >> > > >> > > > > > >> track any user information whatsoever.*
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> Please reply suggesting weather you are OK with this
> >> > > >> approach.
> >> > > >> > > > > > >>
> >> > > >> > > > > > >> D.
> >> > > >> > > > > > >>
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>



-- 
Alexey Kuznetsov

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@apache.org>.

Sorry guys - I neglected the fact that our lists don't permit
attachments. I have put the screenshot to an external server [1]

[1] https://imgur.com/a/p9FJ9

Thank you!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda <dm...@apache.org> wrote:
> Cos,
>
> The screenshot was not attached. Could you share it some other way (google drive, etc.)? I’ve never seen any commercial on the site.
>
> —
> Denis
>
>> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>> I don't see an issue with node version either.
>>
>> Just one more, and it might be slightly irrelevant, issue though... I looked at the Ignite's site and found the following ads and trackers (that are indeed getting disabled by my browser).
>> Why are googleads or doubleclick are permitted?
>>
>>
>>
>> Thanks,
>>   Cos
>>
>>
>> --
>>   With regards,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing.
>>
>> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org>> wrote:
>> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vozerov@gridgain.com <ma...@gridgain.com>>
>> wrote:
>>
>> > Folks,
>> >
>> > Can we add version of current node to web request? This way we will better
>> > understand version distribution, what might help us with certain API
>> > update/deprecate decisions
>> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <http://ignite.apache.org/latest.cgi&ver=2.2.0>
>>
>>
>> Vladimir, I personally do not see a problem with that, as sending the
>> current version to the update checker seems very innocent to me. At the
>> same time, it will allow us to examine the usage of each release and make
>> decisions about dropping backward compatibility or spotting bugs for a
>> certain release.
>>
>> Cos, Raul, any thoughts?
>>
>>
>> >
>> >
>> > Vladimir.
>> >
>> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org>>
>> > wrote:
>> >
>> > > I think it is safe to assume at this point that everyone is in general
>> > > agreement, since there are no active objections.
>> > >
>> > > I have filed a ticket for the 2.3 release. Let's try to make it happen:
>> > > https://issues.apache.org/jira/browse/IGNITE-6305 <https://issues.apache.org/jira/browse/IGNITE-6305>
>> > >
>> > > D.
>> > >
>> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>> > dsetrakyan@apache.org <ma...@apache.org>>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com <ma...@evosent.com>>
>> > > > wrote:
>> > > >
>> > > >> Yeah, I guess that's doable as well and requires less management
>> > effort
>> > > >> than my suggestion. We could use events [1] to store payload data
>> > (e.g.
>> > > >> IP,
>> > > >> version, etc.)
>> > > >
>> > > >
>> > > > Yes, we could use events or some other similar API provided by GA.
>> > > >
>> > > >
>> > > >> What the download page CGI developed in? PHP?
>> > > >>
>> > > >
>> > > > To be honest, no clue. I guess someone in the community can figure it
>> > > out:
>> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>
>> > > >
>> > > >
>> > > >> However, I'm not sure whether storing this data in a 3rd party
>> > (Google)
>> > > is
>> > > >> compliant with the ASF policy. I guess it's no biggie, but if there's
>> > > >> doubt
>> > > >> in the PMC, it's better to ask legal@.
>> > > >
>> > > >
>> > > > I am not sure there is anything to ask about. The whole Ignite website
>> > is
>> > > > GA enabled, and all we are doing is accessing a standard web page from
>> > > the
>> > > > Ignite web site. The information gathered from GA is available only to
>> > > the
>> > > > Ignite PMC. Frankly, I think legal@ will be very confused by this
>> > > > question.
>> > > >
>> > > > Even ASF website itself uses GA: https://www.apache.org/ <https://www.apache.org/>
>> > > > foundation/policies/privacy.html
>> > > >
>> > > >
>> > > >> I think Cos said it's OK; maybe Roman can pitch in.
>> > > >>
>> > > >
>> > > >  Sure, would be nice to hear from Roman as well.
>> > > >
>> > > >
>> > > >> Cheers.
>> > > >>
>> > > >> [1]
>> > > >> https://developers.google.com/analytics/devguides/collection <https://developers.google.com/analytics/devguides/collection>
>> > > >> /analyticsjs/events
>> > > >>
>> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>> > > >> dsetrakyan@apache.org <ma...@apache.org>>
>> > > >> wrote:
>> > > >>
>> > > >> > Raul,
>> > > >> >
>> > > >> > Could point about Javascript, it will not work because it executes
>> > in
>> > > >> the
>> > > >> > browser. This means we need a server-side script, like CGI we are
>> > > using
>> > > >> on
>> > > >> > our download page.
>> > > >> >
>> > > >> > How about this approach. We create something like ignite-version.cgi
>> > > >> script
>> > > >> > which will invoke a call to GA and then return the latest version.
>> > > >> >
>> > > >> > This should work, right?
>> > > >> >
>> > > >> > D.
>> > > >> >
>> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>> > raul.asf@evosent.com <ma...@evosent.com>
>> > > >
>> > > >> > wrote:
>> > > >> >
>> > > >> > > Hey Dmitriy and all
>> > > >> > >
>> > > >> > > Also, since we have GA enabled for the website, we can track how
>> > > many
>> > > >> > times
>> > > >> > > > this page was accessed, which will be equal to the number of
>> > > starts.
>> > > >> > This
>> > > >> > > > way, the counter information is tracked and monitored by the
>> > > Ignite
>> > > >> > PMC.
>> > > >> > >
>> > > >> > >
>> > > >> > > Unfortunately this won't work because GA is loaded via Javascript
>> > on
>> > > >> the
>> > > >> > > browser, so Google will never receive the page hit.
>> > > >> > >
>> > > >> > > Given the constraints, the most viable solution is an HTTPS
>> > endpoint
>> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
>> > request.
>> > > >> The
>> > > >> > > simplest thing is to write each request in a log file, along with
>> > > the
>> > > >> > > timestamp, the version reported by the client, maybe the IP (not
>> > > sure
>> > > >> > about
>> > > >> > > the ASF rules about this concerning privacy, I guess it's OK if
>> > you
>> > > >> make
>> > > >> > it
>> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
>> > > creates
>> > > >> on
>> > > >> > > first startup or something.
>> > > >> > >
>> > > >> > > This endpoint would need some basic DDoS protection and
>> > blacklisting
>> > > >> to
>> > > >> > > prevent data spoofing.
>> > > >> > >
>> > > >> > > If we'll be implementing this endpoint anyway, then there's no
>> > point
>> > > >> > > placing another file on Git or elsewhere for reporting the latest
>> > > >> > versions:
>> > > >> > > the endpoint itself can return them.
>> > > >> > >
>> > > >> > > WDYT?
>> > > >> > >
>> > > >> > > Cheers.
>> > > >> > >
>> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>> > > >> > dsetrakyan@apache.org <ma...@apache.org>>
>> > > >> > > wrote:
>> > > >> > >
>> > > >> > > > Cos, Raul,
>> > > >> > > >
>> > > >> > > > Thanks for the feedback. I completely agree about Maven Central
>> > > >> being a
>> > > >> > > 3rd
>> > > >> > > > party repo (did not think about that initially). All your
>> > > >> suggestions
>> > > >> > > make
>> > > >> > > > sense, but I would like to keep it as simple as possible, and so
>> > > far
>> > > >> > > > everything suggested required GIT dependencies and extra work.
>> > > >> > > >
>> > > >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
>> > > >> > website
>> > > >> > > > which will have only the latest version. Every time a node
>> > starts,
>> > > >> it
>> > > >> > > > receives the latest version from the page and suggests that
>> > users
>> > > >> > upgrade
>> > > >> > > > if needed.
>> > > >> > > >
>> > > >> > > > Also, since we have GA enabled for the website, we can track how
>> > > >> many
>> > > >> > > times
>> > > >> > > > this page was accessed, which will be equal to the number of
>> > > starts.
>> > > >> > This
>> > > >> > > > way, the counter information is tracked and monitored by the
>> > > Ignite
>> > > >> > PMC.
>> > > >> > > >
>> > > >> > > > This approach looks pretty innocent to me and everything is kept
>> > > and
>> > > >> > > > managed within Apache.
>> > > >> > > >
>> > > >> > > > Thoughts?
>> > > >> > > >
>> > > >> > > > D.
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>> > > >> cos@apache.org <ma...@apache.org>>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > > I agree with Raul.
>> > > >> > > > > - providing a ping-back address to a 3rd party might be frown
>> > > >> upon by
>> > > >> > > > some.
>> > > >> > > > >   And might have a consequences like stats collection about
>> > > users'
>> > > >> > > > >   infrastructure.
>> > > >> > > > > - checking an ASF git-repo is easy and won't download any
>> > binary
>> > > >> > data:
>> > > >> > > > >   everything is clear text and could be easily monitored by
>> > any
>> > > of
>> > > >> > the
>> > > >> > > > > network
>> > > >> > > > >   diagnostic tools, shall it be required. But it involves a
>> > bit
>> > > of
>> > > >> > the
>> > > >> > > > > release
>> > > >> > > > >   discipline.
>> > > >> > > > > - the binary data download in the runtime is my main concern.
>> > > >> This is
>> > > >> > > the
>> > > >> > > > >   vector of MMA. What if someone gains the control over the
>> > > >> > repository
>> > > >> > > > and
>> > > >> > > > >   replaces the file with some malicious content.
>> > > >> > > > >
>> > > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
>> > call
>> > > >> to
>> > > >> > an
>> > > >> > > > > external script to check upon the atest software version
>> > > available
>> > > >> > for
>> > > >> > > > > download. In the past, the endpoint was running on a 3rd party
>> > > >> > server,
>> > > >> > > I
>> > > >> > > > > believe the best way would be to put this script on ASF infra
>> > > and
>> > > >> > have
>> > > >> > > > the
>> > > >> > > > > "update checker" running in a completely controlled
>> > environment.
>> > > >> > > > Actually,
>> > > >> > > > > I
>> > > >> > > > > recall we had this very discussion during the Incubation; I
>> > can
>> > > >> > > probably
>> > > >> > > > > dig
>> > > >> > > > > out the corresponding thread.
>> > > >> > > > >
>> > > >> > > > > Thoughts?
>> > > >> > > > >   Cok
>> > > >> > > > >
>> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> > > >> > > > > > Hey guys
>> > > >> > > > > >
>> > > >> > > > > > In my opinion, maven.org <http://maven.org/> is still owned by a third party
>> > > >> > (Sonatype).
>> > > >> > > > We
>> > > >> > > > > > don't know what kind of data analysis or intelligence
>> > > extraction
>> > > >> > they
>> > > >> > > > > run.
>> > > >> > > > > >
>> > > >> > > > > > If Ignite servers all over the world were hitting maven.org <http://maven.org/>
>> > > >> > > > periodically
>> > > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
>> > clear
>> > > >> > > > indication
>> > > >> > > > > > of who is running an Ignite server.
>> > > >> > > > > >
>> > > >> > > > > > They could reverse lookup the IP address and find out what
>> > > >> > > corporation
>> > > >> > > > it
>> > > >> > > > > > is.
>> > > >> > > > > >
>> > > >> > > > > > How about having Ignite check the ASF Git directly?
>> > > >> > > > > >
>> > > >> > > > > > We could use the Git ssh API, but that would require a new
>> > > >> > > dependency,
>> > > >> > > > > > which I advise against.
>> > > >> > > > > >
>> > > >> > > > > > Alternatively, we could have it scrape this HTML for new Git
>> > > >> tags:
>> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git <https://git-wip-us.apache.org/repos/asf?p=ignite.git>
>> > > >> > > > > >
>> > > >> > > > > > Another option is to place a txt file in the root of the
>> > > master
>> > > >> > > branch
>> > > >> > > > > (e.g
>> > > >> > > > > > LATEST), containing a list of the latest GA versions for
>> > each
>> > > >> major
>> > > >> > > > > version
>> > > >> > > > > > line (1.x, 2.x).
>> > > >> > > > > >
>> > > >> > > > > > I would advocate this last option, but it requires somebody
>> > > >> > > remembering
>> > > >> > > > > to
>> > > >> > > > > > update the file with every release, unless we automate it
>> > > with a
>> > > >> > > Maven
>> > > >> > > > > > plugin.
>> > > >> > > > > >
>> > > >> > > > > > Hope that helps!
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org <ma...@apache.org>>
>> > > wrote:
>> > > >> > > > > >
>> > > >> > > > > > I see nothing wrong with this approach.
>> > > >> > > > > >
>> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is
>> > it
>> > > >> good
>> > > >> > > to
>> > > >> > > > > go?
>> > > >> > > > > >
>> > > >> > > > > > —
>> > > >> > > > > > Denis
>> > > >> > > > > >
>> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>> > > >> > > > dsetrakyan@apache.org <ma...@apache.org>
>> > > >> > > > > >
>> > > >> > > > > > wrote:
>> > > >> > > > > > >
>> > > >> > > > > > > Is everyone OK with this approach? Should I file a ticket
>> > on
>> > > >> it?
>> > > >> > > > > > >
>> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org>>
>> > > >> > > > > > > wrote:
>> > > >> > > > > > >
>> > > >> > > > > > >> Igniters,
>> > > >> > > > > > >>
>> > > >> > > > > > >> There has been lots of talk of proposals about various
>> > > usage
>> > > >> > > metrics
>> > > >> > > > > for
>> > > >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
>> > > the
>> > > >> > topic
>> > > >> > > > and
>> > > >> > > > > > >> propose something very simple and non-intrusive.
>> > > >> > > > > > >>
>> > > >> > > > > > >> 1. Update Checker
>> > > >> > > > > > >> The main purpose of the update checker is not to collect
>> > > >> > metrics,
>> > > >> > > > but
>> > > >> > > > > to
>> > > >> > > > > > >> notify users about a new version of Ignite by accessing
>> > > >> > maven.org <http://maven.org/>
>> > > >> > > > and
>> > > >> > > > > > >> getting the version out of the metadata file:
>> > > >> > > > > > >> http://repo2.maven.org/maven2/ <http://repo2.maven.org/maven2/>
>> > > org/apache/ignite/ignite-core/
>> > > >> > > > > > >> maven-metadata.xml
>> > > >> > > > > > >>
>> > > >> > > > > > >> This way we do not send any information anywhere and, at
>> > > the
>> > > >> > same
>> > > >> > > > > time,
>> > > >> > > > > > >> urge our users to download and start using the latest
>> > > >> version of
>> > > >> > > > > Ignite.
>> > > >> > > > > > >>
>> > > >> > > > > > >> 2. Startup Counter
>> > > >> > > > > > >> This piece is optional, but we can also get an insight in
>> > > how
>> > > >> > many
>> > > >> > > > > times
>> > > >> > > > > > a
>> > > >> > > > > > >> certain Ignite release gets started. This is just a cool
>> > > >> metric
>> > > >> > > for
>> > > >> > > > > the
>> > > >> > > > > > >> community to gauge the project popularity. You can think
>> > of
>> > > >> it
>> > > >> > as
>> > > >> > > > of a
>> > > >> > > > > > page
>> > > >> > > > > > >> visit counter shown on many websites. We can even decide
>> > to
>> > > >> > > display
>> > > >> > > > > this
>> > > >> > > > > > >> counter on the Ignite website as well.
>> > > >> > > > > > >>
>> > > >> > > > > > >> To do this, we can simply add a JAR in maven for every
>> > > >> release,
>> > > >> > > e.g.
>> > > >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
>> > > >> Every
>> > > >> > > time
>> > > >> > > > > an
>> > > >> > > > > > >> Ignite node starts, it will download this JAR in the
>> > > >> background.
>> > > >> > > > Then
>> > > >> > > > > we
>> > > >> > > > > > >> will be able to view the number of the total downloads
>> > for
>> > > >> this
>> > > >> > > JAR
>> > > >> > > > in
>> > > >> > > > > > >> Maven Central, which is essentially the number of starts
>> > of
>> > > >> > Ignite
>> > > >> > > > > nodes.
>> > > >> > > > > > >>
>> > > >> > > > > > >> *Note that neither of the above suggestions require
>> > Ignite
>> > > to
>> > > >> > send
>> > > >> > > > or
>> > > >> > > > > > >> track any user information whatsoever.*
>> > > >> > > > > > >>
>> > > >> > > > > > >> Please reply suggesting weather you are OK with this
>> > > >> approach.
>> > > >> > > > > > >>
>> > > >> > > > > > >> D.
>> > > >> > > > > > >>
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@apache.org>.

Cos, 

The screenshot was not attached. Could you share it some other way (google drive, etc.)? I’ve never seen any commercial on the site.

—
Denis

> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> I don't see an issue with node version either. 
> 
> Just one more, and it might be slightly irrelevant, issue though... I looked at the Ignite's site and found the following ads and trackers (that are indeed getting disabled by my browser).
> Why are googleads or doubleclick are permitted?
> 
> 
> 
> Thanks,
>   Cos
> 
> 
> --
>   With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> 
> Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any company the author might be affiliated with at the moment of writing.
> 
> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org>> wrote:
> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vozerov@gridgain.com <ma...@gridgain.com>>
> wrote:
> 
> > Folks,
> >
> > Can we add version of current node to web request? This way we will better
> > understand version distribution, what might help us with certain API
> > update/deprecate decisions
> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0 <http://ignite.apache.org/latest.cgi&ver=2.2.0>
> 
> 
> Vladimir, I personally do not see a problem with that, as sending the
> current version to the update checker seems very innocent to me. At the
> same time, it will allow us to examine the usage of each release and make
> decisions about dropping backward compatibility or spotting bugs for a
> certain release.
> 
> Cos, Raul, any thoughts?
> 
> 
> >
> >
> > Vladimir.
> >
> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org>>
> > wrote:
> >
> > > I think it is safe to assume at this point that everyone is in general
> > > agreement, since there are no active objections.
> > >
> > > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> > > https://issues.apache.org/jira/browse/IGNITE-6305 <https://issues.apache.org/jira/browse/IGNITE-6305>
> > >
> > > D.
> > >
> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org <ma...@apache.org>>
> > > wrote:
> > >
> > > >
> > > >
> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com <ma...@evosent.com>>
> > > > wrote:
> > > >
> > > >> Yeah, I guess that's doable as well and requires less management
> > effort
> > > >> than my suggestion. We could use events [1] to store payload data
> > (e.g.
> > > >> IP,
> > > >> version, etc.)
> > > >
> > > >
> > > > Yes, we could use events or some other similar API provided by GA.
> > > >
> > > >
> > > >> What the download page CGI developed in? PHP?
> > > >>
> > > >
> > > > To be honest, no clue. I guess someone in the community can figure it
> > > out:
> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html <https://svn.apache.org/repos/asf/ignite/site/trunk/download.html>
> > > >
> > > >
> > > >> However, I'm not sure whether storing this data in a 3rd party
> > (Google)
> > > is
> > > >> compliant with the ASF policy. I guess it's no biggie, but if there's
> > > >> doubt
> > > >> in the PMC, it's better to ask legal@.
> > > >
> > > >
> > > > I am not sure there is anything to ask about. The whole Ignite website
> > is
> > > > GA enabled, and all we are doing is accessing a standard web page from
> > > the
> > > > Ignite web site. The information gathered from GA is available only to
> > > the
> > > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > > > question.
> > > >
> > > > Even ASF website itself uses GA: https://www.apache.org/ <https://www.apache.org/>
> > > > foundation/policies/privacy.html
> > > >
> > > >
> > > >> I think Cos said it's OK; maybe Roman can pitch in.
> > > >>
> > > >
> > > >  Sure, would be nice to hear from Roman as well.
> > > >
> > > >
> > > >> Cheers.
> > > >>
> > > >> [1]
> > > >> https://developers.google.com/analytics/devguides/collection <https://developers.google.com/analytics/devguides/collection>
> > > >> /analyticsjs/events
> > > >>
> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > > >> dsetrakyan@apache.org <ma...@apache.org>>
> > > >> wrote:
> > > >>
> > > >> > Raul,
> > > >> >
> > > >> > Could point about Javascript, it will not work because it executes
> > in
> > > >> the
> > > >> > browser. This means we need a server-side script, like CGI we are
> > > using
> > > >> on
> > > >> > our download page.
> > > >> >
> > > >> > How about this approach. We create something like ignite-version.cgi
> > > >> script
> > > >> > which will invoke a call to GA and then return the latest version.
> > > >> >
> > > >> > This should work, right?
> > > >> >
> > > >> > D.
> > > >> >
> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> > raul.asf@evosent.com <ma...@evosent.com>
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hey Dmitriy and all
> > > >> > >
> > > >> > > Also, since we have GA enabled for the website, we can track how
> > > many
> > > >> > times
> > > >> > > > this page was accessed, which will be equal to the number of
> > > starts.
> > > >> > This
> > > >> > > > way, the counter information is tracked and monitored by the
> > > Ignite
> > > >> > PMC.
> > > >> > >
> > > >> > >
> > > >> > > Unfortunately this won't work because GA is loaded via Javascript
> > on
> > > >> the
> > > >> > > browser, so Google will never receive the page hit.
> > > >> > >
> > > >> > > Given the constraints, the most viable solution is an HTTPS
> > endpoint
> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
> > request.
> > > >> The
> > > >> > > simplest thing is to write each request in a log file, along with
> > > the
> > > >> > > timestamp, the version reported by the client, maybe the IP (not
> > > sure
> > > >> > about
> > > >> > > the ASF rules about this concerning privacy, I guess it's OK if
> > you
> > > >> make
> > > >> > it
> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> > > creates
> > > >> on
> > > >> > > first startup or something.
> > > >> > >
> > > >> > > This endpoint would need some basic DDoS protection and
> > blacklisting
> > > >> to
> > > >> > > prevent data spoofing.
> > > >> > >
> > > >> > > If we'll be implementing this endpoint anyway, then there's no
> > point
> > > >> > > placing another file on Git or elsewhere for reporting the latest
> > > >> > versions:
> > > >> > > the endpoint itself can return them.
> > > >> > >
> > > >> > > WDYT?
> > > >> > >
> > > >> > > Cheers.
> > > >> > >
> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > > >> > dsetrakyan@apache.org <ma...@apache.org>>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Cos, Raul,
> > > >> > > >
> > > >> > > > Thanks for the feedback. I completely agree about Maven Central
> > > >> being a
> > > >> > > 3rd
> > > >> > > > party repo (did not think about that initially). All your
> > > >> suggestions
> > > >> > > make
> > > >> > > > sense, but I would like to keep it as simple as possible, and so
> > > far
> > > >> > > > everything suggested required GIT dependencies and extra work.
> > > >> > > >
> > > >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> > > >> > website
> > > >> > > > which will have only the latest version. Every time a node
> > starts,
> > > >> it
> > > >> > > > receives the latest version from the page and suggests that
> > users
> > > >> > upgrade
> > > >> > > > if needed.
> > > >> > > >
> > > >> > > > Also, since we have GA enabled for the website, we can track how
> > > >> many
> > > >> > > times
> > > >> > > > this page was accessed, which will be equal to the number of
> > > starts.
> > > >> > This
> > > >> > > > way, the counter information is tracked and monitored by the
> > > Ignite
> > > >> > PMC.
> > > >> > > >
> > > >> > > > This approach looks pretty innocent to me and everything is kept
> > > and
> > > >> > > > managed within Apache.
> > > >> > > >
> > > >> > > > Thoughts?
> > > >> > > >
> > > >> > > > D.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > > >> cos@apache.org <ma...@apache.org>>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > I agree with Raul.
> > > >> > > > > - providing a ping-back address to a 3rd party might be frown
> > > >> upon by
> > > >> > > > some.
> > > >> > > > >   And might have a consequences like stats collection about
> > > users'
> > > >> > > > >   infrastructure.
> > > >> > > > > - checking an ASF git-repo is easy and won't download any
> > binary
> > > >> > data:
> > > >> > > > >   everything is clear text and could be easily monitored by
> > any
> > > of
> > > >> > the
> > > >> > > > > network
> > > >> > > > >   diagnostic tools, shall it be required. But it involves a
> > bit
> > > of
> > > >> > the
> > > >> > > > > release
> > > >> > > > >   discipline.
> > > >> > > > > - the binary data download in the runtime is my main concern.
> > > >> This is
> > > >> > > the
> > > >> > > > >   vector of MMA. What if someone gains the control over the
> > > >> > repository
> > > >> > > > and
> > > >> > > > >   replaces the file with some malicious content.
> > > >> > > > >
> > > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
> > call
> > > >> to
> > > >> > an
> > > >> > > > > external script to check upon the atest software version
> > > available
> > > >> > for
> > > >> > > > > download. In the past, the endpoint was running on a 3rd party
> > > >> > server,
> > > >> > > I
> > > >> > > > > believe the best way would be to put this script on ASF infra
> > > and
> > > >> > have
> > > >> > > > the
> > > >> > > > > "update checker" running in a completely controlled
> > environment.
> > > >> > > > Actually,
> > > >> > > > > I
> > > >> > > > > recall we had this very discussion during the Incubation; I
> > can
> > > >> > > probably
> > > >> > > > > dig
> > > >> > > > > out the corresponding thread.
> > > >> > > > >
> > > >> > > > > Thoughts?
> > > >> > > > >   Cok
> > > >> > > > >
> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > >> > > > > > Hey guys
> > > >> > > > > >
> > > >> > > > > > In my opinion, maven.org <http://maven.org/> is still owned by a third party
> > > >> > (Sonatype).
> > > >> > > > We
> > > >> > > > > > don't know what kind of data analysis or intelligence
> > > extraction
> > > >> > they
> > > >> > > > > run.
> > > >> > > > > >
> > > >> > > > > > If Ignite servers all over the world were hitting maven.org <http://maven.org/>
> > > >> > > > periodically
> > > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
> > clear
> > > >> > > > indication
> > > >> > > > > > of who is running an Ignite server.
> > > >> > > > > >
> > > >> > > > > > They could reverse lookup the IP address and find out what
> > > >> > > corporation
> > > >> > > > it
> > > >> > > > > > is.
> > > >> > > > > >
> > > >> > > > > > How about having Ignite check the ASF Git directly?
> > > >> > > > > >
> > > >> > > > > > We could use the Git ssh API, but that would require a new
> > > >> > > dependency,
> > > >> > > > > > which I advise against.
> > > >> > > > > >
> > > >> > > > > > Alternatively, we could have it scrape this HTML for new Git
> > > >> tags:
> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git <https://git-wip-us.apache.org/repos/asf?p=ignite.git>
> > > >> > > > > >
> > > >> > > > > > Another option is to place a txt file in the root of the
> > > master
> > > >> > > branch
> > > >> > > > > (e.g
> > > >> > > > > > LATEST), containing a list of the latest GA versions for
> > each
> > > >> major
> > > >> > > > > version
> > > >> > > > > > line (1.x, 2.x).
> > > >> > > > > >
> > > >> > > > > > I would advocate this last option, but it requires somebody
> > > >> > > remembering
> > > >> > > > > to
> > > >> > > > > > update the file with every release, unless we automate it
> > > with a
> > > >> > > Maven
> > > >> > > > > > plugin.
> > > >> > > > > >
> > > >> > > > > > Hope that helps!
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org <ma...@apache.org>>
> > > wrote:
> > > >> > > > > >
> > > >> > > > > > I see nothing wrong with this approach.
> > > >> > > > > >
> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is
> > it
> > > >> good
> > > >> > > to
> > > >> > > > > go?
> > > >> > > > > >
> > > >> > > > > > —
> > > >> > > > > > Denis
> > > >> > > > > >
> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > > >> > > > dsetrakyan@apache.org <ma...@apache.org>
> > > >> > > > > >
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > Is everyone OK with this approach? Should I file a ticket
> > on
> > > >> it?
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > > >> > > > > dsetrakyan@apache.org <ma...@apache.org>>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Igniters,
> > > >> > > > > > >>
> > > >> > > > > > >> There has been lots of talk of proposals about various
> > > usage
> > > >> > > metrics
> > > >> > > > > for
> > > >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
> > > the
> > > >> > topic
> > > >> > > > and
> > > >> > > > > > >> propose something very simple and non-intrusive.
> > > >> > > > > > >>
> > > >> > > > > > >> 1. Update Checker
> > > >> > > > > > >> The main purpose of the update checker is not to collect
> > > >> > metrics,
> > > >> > > > but
> > > >> > > > > to
> > > >> > > > > > >> notify users about a new version of Ignite by accessing
> > > >> > maven.org <http://maven.org/>
> > > >> > > > and
> > > >> > > > > > >> getting the version out of the metadata file:
> > > >> > > > > > >> http://repo2.maven.org/maven2/ <http://repo2.maven.org/maven2/>
> > > org/apache/ignite/ignite-core/
> > > >> > > > > > >> maven-metadata.xml
> > > >> > > > > > >>
> > > >> > > > > > >> This way we do not send any information anywhere and, at
> > > the
> > > >> > same
> > > >> > > > > time,
> > > >> > > > > > >> urge our users to download and start using the latest
> > > >> version of
> > > >> > > > > Ignite.
> > > >> > > > > > >>
> > > >> > > > > > >> 2. Startup Counter
> > > >> > > > > > >> This piece is optional, but we can also get an insight in
> > > how
> > > >> > many
> > > >> > > > > times
> > > >> > > > > > a
> > > >> > > > > > >> certain Ignite release gets started. This is just a cool
> > > >> metric
> > > >> > > for
> > > >> > > > > the
> > > >> > > > > > >> community to gauge the project popularity. You can think
> > of
> > > >> it
> > > >> > as
> > > >> > > > of a
> > > >> > > > > > page
> > > >> > > > > > >> visit counter shown on many websites. We can even decide
> > to
> > > >> > > display
> > > >> > > > > this
> > > >> > > > > > >> counter on the Ignite website as well.
> > > >> > > > > > >>
> > > >> > > > > > >> To do this, we can simply add a JAR in maven for every
> > > >> release,
> > > >> > > e.g.
> > > >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> > > >> Every
> > > >> > > time
> > > >> > > > > an
> > > >> > > > > > >> Ignite node starts, it will download this JAR in the
> > > >> background.
> > > >> > > > Then
> > > >> > > > > we
> > > >> > > > > > >> will be able to view the number of the total downloads
> > for
> > > >> this
> > > >> > > JAR
> > > >> > > > in
> > > >> > > > > > >> Maven Central, which is essentially the number of starts
> > of
> > > >> > Ignite
> > > >> > > > > nodes.
> > > >> > > > > > >>
> > > >> > > > > > >> *Note that neither of the above suggestions require
> > Ignite
> > > to
> > > >> > send
> > > >> > > > or
> > > >> > > > > > >> track any user information whatsoever.*
> > > >> > > > > > >>
> > > >> > > > > > >> Please reply suggesting weather you are OK with this
> > > >> approach.
> > > >> > > > > > >>
> > > >> > > > > > >> D.
> > > >> > > > > > >>
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@apache.org>.

I don't see an issue with node version either.

Just one more, and it might be slightly irrelevant, issue though... I
looked at the Ignite's site and found the following ads and trackers (that
are indeed getting disabled by my browser).
Why are googleads or doubleclick are permitted?



Thanks,
  Cos


--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any company the author might be
affiliated with at the moment of writing.

On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Folks,
> >
> > Can we add version of current node to web request? This way we will
> better
> > understand version distribution, what might help us with certain API
> > update/deprecate decisions
> > E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0
>
>
> Vladimir, I personally do not see a problem with that, as sending the
> current version to the update checker seems very innocent to me. At the
> same time, it will allow us to examine the usage of each release and make
> decisions about dropping backward compatibility or spotting bugs for a
> certain release.
>
> Cos, Raul, any thoughts?
>
>
> >
> >
> > Vladimir.
> >
> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> > wrote:
> >
> > > I think it is safe to assume at this point that everyone is in general
> > > agreement, since there are no active objections.
> > >
> > > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> > > https://issues.apache.org/jira/browse/IGNITE-6305
> > >
> > > D.
> > >
> > > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > >
> > > >
> > > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> raul.asf@evosent.com>
> > > > wrote:
> > > >
> > > >> Yeah, I guess that's doable as well and requires less management
> > effort
> > > >> than my suggestion. We could use events [1] to store payload data
> > (e.g.
> > > >> IP,
> > > >> version, etc.)
> > > >
> > > >
> > > > Yes, we could use events or some other similar API provided by GA.
> > > >
> > > >
> > > >> What the download page CGI developed in? PHP?
> > > >>
> > > >
> > > > To be honest, no clue. I guess someone in the community can figure it
> > > out:
> > > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > > >
> > > >
> > > >> However, I'm not sure whether storing this data in a 3rd party
> > (Google)
> > > is
> > > >> compliant with the ASF policy. I guess it's no biggie, but if
> there's
> > > >> doubt
> > > >> in the PMC, it's better to ask legal@.
> > > >
> > > >
> > > > I am not sure there is anything to ask about. The whole Ignite
> website
> > is
> > > > GA enabled, and all we are doing is accessing a standard web page
> from
> > > the
> > > > Ignite web site. The information gathered from GA is available only
> to
> > > the
> > > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > > > question.
> > > >
> > > > Even ASF website itself uses GA: https://www.apache.org/
> > > > foundation/policies/privacy.html
> > > >
> > > >
> > > >> I think Cos said it's OK; maybe Roman can pitch in.
> > > >>
> > > >
> > > >  Sure, would be nice to hear from Roman as well.
> > > >
> > > >
> > > >> Cheers.
> > > >>
> > > >> [1]
> > > >> https://developers.google.com/analytics/devguides/collection
> > > >> /analyticsjs/events
> > > >>
> > > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > > >> dsetrakyan@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Raul,
> > > >> >
> > > >> > Could point about Javascript, it will not work because it executes
> > in
> > > >> the
> > > >> > browser. This means we need a server-side script, like CGI we are
> > > using
> > > >> on
> > > >> > our download page.
> > > >> >
> > > >> > How about this approach. We create something like
> ignite-version.cgi
> > > >> script
> > > >> > which will invoke a call to GA and then return the latest version.
> > > >> >
> > > >> > This should work, right?
> > > >> >
> > > >> > D.
> > > >> >
> > > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> > raul.asf@evosent.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hey Dmitriy and all
> > > >> > >
> > > >> > > Also, since we have GA enabled for the website, we can track how
> > > many
> > > >> > times
> > > >> > > > this page was accessed, which will be equal to the number of
> > > starts.
> > > >> > This
> > > >> > > > way, the counter information is tracked and monitored by the
> > > Ignite
> > > >> > PMC.
> > > >> > >
> > > >> > >
> > > >> > > Unfortunately this won't work because GA is loaded via
> Javascript
> > on
> > > >> the
> > > >> > > browser, so Google will never receive the page hit.
> > > >> > >
> > > >> > > Given the constraints, the most viable solution is an HTTPS
> > endpoint
> > > >> > > running on ASF infra that Ignite invokes via a GET or POST
> > request.
> > > >> The
> > > >> > > simplest thing is to write each request in a log file, along
> with
> > > the
> > > >> > > timestamp, the version reported by the client, maybe the IP (not
> > > sure
> > > >> > about
> > > >> > > the ASF rules about this concerning privacy, I guess it's OK if
> > you
> > > >> make
> > > >> > it
> > > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> > > creates
> > > >> on
> > > >> > > first startup or something.
> > > >> > >
> > > >> > > This endpoint would need some basic DDoS protection and
> > blacklisting
> > > >> to
> > > >> > > prevent data spoofing.
> > > >> > >
> > > >> > > If we'll be implementing this endpoint anyway, then there's no
> > point
> > > >> > > placing another file on Git or elsewhere for reporting the
> latest
> > > >> > versions:
> > > >> > > the endpoint itself can return them.
> > > >> > >
> > > >> > > WDYT?
> > > >> > >
> > > >> > > Cheers.
> > > >> > >
> > > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > > >> > dsetrakyan@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Cos, Raul,
> > > >> > > >
> > > >> > > > Thanks for the feedback. I completely agree about Maven
> Central
> > > >> being a
> > > >> > > 3rd
> > > >> > > > party repo (did not think about that initially). All your
> > > >> suggestions
> > > >> > > make
> > > >> > > > sense, but I would like to keep it as simple as possible, and
> so
> > > far
> > > >> > > > everything suggested required GIT dependencies and extra work.
> > > >> > > >
> > > >> > > > How about Yakov's suggestion. We simply add a page to the
> Ignite
> > > >> > website
> > > >> > > > which will have only the latest version. Every time a node
> > starts,
> > > >> it
> > > >> > > > receives the latest version from the page and suggests that
> > users
> > > >> > upgrade
> > > >> > > > if needed.
> > > >> > > >
> > > >> > > > Also, since we have GA enabled for the website, we can track
> how
> > > >> many
> > > >> > > times
> > > >> > > > this page was accessed, which will be equal to the number of
> > > starts.
> > > >> > This
> > > >> > > > way, the counter information is tracked and monitored by the
> > > Ignite
> > > >> > PMC.
> > > >> > > >
> > > >> > > > This approach looks pretty innocent to me and everything is
> kept
> > > and
> > > >> > > > managed within Apache.
> > > >> > > >
> > > >> > > > Thoughts?
> > > >> > > >
> > > >> > > > D.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > > >> cos@apache.org>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > I agree with Raul.
> > > >> > > > > - providing a ping-back address to a 3rd party might be
> frown
> > > >> upon by
> > > >> > > > some.
> > > >> > > > >   And might have a consequences like stats collection about
> > > users'
> > > >> > > > >   infrastructure.
> > > >> > > > > - checking an ASF git-repo is easy and won't download any
> > binary
> > > >> > data:
> > > >> > > > >   everything is clear text and could be easily monitored by
> > any
> > > of
> > > >> > the
> > > >> > > > > network
> > > >> > > > >   diagnostic tools, shall it be required. But it involves a
> > bit
> > > of
> > > >> > the
> > > >> > > > > release
> > > >> > > > >   discipline.
> > > >> > > > > - the binary data download in the runtime is my main
> concern.
> > > >> This is
> > > >> > > the
> > > >> > > > >   vector of MMA. What if someone gains the control over the
> > > >> > repository
> > > >> > > > and
> > > >> > > > >   replaces the file with some malicious content.
> > > >> > > > >
> > > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
> > call
> > > >> to
> > > >> > an
> > > >> > > > > external script to check upon the atest software version
> > > available
> > > >> > for
> > > >> > > > > download. In the past, the endpoint was running on a 3rd
> party
> > > >> > server,
> > > >> > > I
> > > >> > > > > believe the best way would be to put this script on ASF
> infra
> > > and
> > > >> > have
> > > >> > > > the
> > > >> > > > > "update checker" running in a completely controlled
> > environment.
> > > >> > > > Actually,
> > > >> > > > > I
> > > >> > > > > recall we had this very discussion during the Incubation; I
> > can
> > > >> > > probably
> > > >> > > > > dig
> > > >> > > > > out the corresponding thread.
> > > >> > > > >
> > > >> > > > > Thoughts?
> > > >> > > > >   Cok
> > > >> > > > >
> > > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > >> > > > > > Hey guys
> > > >> > > > > >
> > > >> > > > > > In my opinion, maven.org is still owned by a third party
> > > >> > (Sonatype).
> > > >> > > > We
> > > >> > > > > > don't know what kind of data analysis or intelligence
> > > extraction
> > > >> > they
> > > >> > > > > run.
> > > >> > > > > >
> > > >> > > > > > If Ignite servers all over the world were hitting
> maven.org
> > > >> > > > periodically
> > > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
> > clear
> > > >> > > > indication
> > > >> > > > > > of who is running an Ignite server.
> > > >> > > > > >
> > > >> > > > > > They could reverse lookup the IP address and find out what
> > > >> > > corporation
> > > >> > > > it
> > > >> > > > > > is.
> > > >> > > > > >
> > > >> > > > > > How about having Ignite check the ASF Git directly?
> > > >> > > > > >
> > > >> > > > > > We could use the Git ssh API, but that would require a new
> > > >> > > dependency,
> > > >> > > > > > which I advise against.
> > > >> > > > > >
> > > >> > > > > > Alternatively, we could have it scrape this HTML for new
> Git
> > > >> tags:
> > > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > > >> > > > > >
> > > >> > > > > > Another option is to place a txt file in the root of the
> > > master
> > > >> > > branch
> > > >> > > > > (e.g
> > > >> > > > > > LATEST), containing a list of the latest GA versions for
> > each
> > > >> major
> > > >> > > > > version
> > > >> > > > > > line (1.x, 2.x).
> > > >> > > > > >
> > > >> > > > > > I would advocate this last option, but it requires
> somebody
> > > >> > > remembering
> > > >> > > > > to
> > > >> > > > > > update the file with every release, unless we automate it
> > > with a
> > > >> > > Maven
> > > >> > > > > > plugin.
> > > >> > > > > >
> > > >> > > > > > Hope that helps!
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org>
> > > wrote:
> > > >> > > > > >
> > > >> > > > > > I see nothing wrong with this approach.
> > > >> > > > > >
> > > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think?
> Is
> > it
> > > >> good
> > > >> > > to
> > > >> > > > > go?
> > > >> > > > > >
> > > >> > > > > > —
> > > >> > > > > > Denis
> > > >> > > > > >
> > > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > > >> > > > dsetrakyan@apache.org
> > > >> > > > > >
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > Is everyone OK with this approach? Should I file a
> ticket
> > on
> > > >> it?
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > > >> > > > > dsetrakyan@apache.org>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Igniters,
> > > >> > > > > > >>
> > > >> > > > > > >> There has been lots of talk of proposals about various
> > > usage
> > > >> > > metrics
> > > >> > > > > for
> > > >> > > > > > >> Ignite and nothing came of it. I would like to
> resurrect
> > > the
> > > >> > topic
> > > >> > > > and
> > > >> > > > > > >> propose something very simple and non-intrusive.
> > > >> > > > > > >>
> > > >> > > > > > >> 1. Update Checker
> > > >> > > > > > >> The main purpose of the update checker is not to
> collect
> > > >> > metrics,
> > > >> > > > but
> > > >> > > > > to
> > > >> > > > > > >> notify users about a new version of Ignite by accessing
> > > >> > maven.org
> > > >> > > > and
> > > >> > > > > > >> getting the version out of the metadata file:
> > > >> > > > > > >> http://repo2.maven.org/maven2/
> > > org/apache/ignite/ignite-core/
> > > >> > > > > > >> maven-metadata.xml
> > > >> > > > > > >>
> > > >> > > > > > >> This way we do not send any information anywhere and,
> at
> > > the
> > > >> > same
> > > >> > > > > time,
> > > >> > > > > > >> urge our users to download and start using the latest
> > > >> version of
> > > >> > > > > Ignite.
> > > >> > > > > > >>
> > > >> > > > > > >> 2. Startup Counter
> > > >> > > > > > >> This piece is optional, but we can also get an insight
> in
> > > how
> > > >> > many
> > > >> > > > > times
> > > >> > > > > > a
> > > >> > > > > > >> certain Ignite release gets started. This is just a
> cool
> > > >> metric
> > > >> > > for
> > > >> > > > > the
> > > >> > > > > > >> community to gauge the project popularity. You can
> think
> > of
> > > >> it
> > > >> > as
> > > >> > > > of a
> > > >> > > > > > page
> > > >> > > > > > >> visit counter shown on many websites. We can even
> decide
> > to
> > > >> > > display
> > > >> > > > > this
> > > >> > > > > > >> counter on the Ignite website as well.
> > > >> > > > > > >>
> > > >> > > > > > >> To do this, we can simply add a JAR in maven for every
> > > >> release,
> > > >> > > e.g.
> > > >> > > > > > >> ignite-start-counter.jar, which will contain only 1
> byte.
> > > >> Every
> > > >> > > time
> > > >> > > > > an
> > > >> > > > > > >> Ignite node starts, it will download this JAR in the
> > > >> background.
> > > >> > > > Then
> > > >> > > > > we
> > > >> > > > > > >> will be able to view the number of the total downloads
> > for
> > > >> this
> > > >> > > JAR
> > > >> > > > in
> > > >> > > > > > >> Maven Central, which is essentially the number of
> starts
> > of
> > > >> > Ignite
> > > >> > > > > nodes.
> > > >> > > > > > >>
> > > >> > > > > > >> *Note that neither of the above suggestions require
> > Ignite
> > > to
> > > >> > send
> > > >> > > > or
> > > >> > > > > > >> track any user information whatsoever.*
> > > >> > > > > > >>
> > > >> > > > > > >> Please reply suggesting weather you are OK with this
> > > >> approach.
> > > >> > > > > > >>
> > > >> > > > > > >> D.
> > > >> > > > > > >>
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:

> Folks,
>
> Can we add version of current node to web request? This way we will better
> understand version distribution, what might help us with certain API
> update/deprecate decisions
> E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0


Vladimir, I personally do not see a problem with that, as sending the
current version to the update checker seems very innocent to me. At the
same time, it will allow us to examine the usage of each release and make
decisions about dropping backward compatibility or spotting bugs for a
certain release.

Cos, Raul, any thoughts?


>
>
> Vladimir.
>
> On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > I think it is safe to assume at this point that everyone is in general
> > agreement, since there are no active objections.
> >
> > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> > https://issues.apache.org/jira/browse/IGNITE-6305
> >
> > D.
> >
> > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > >
> > >
> > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <ra...@evosent.com>
> > > wrote:
> > >
> > >> Yeah, I guess that's doable as well and requires less management
> effort
> > >> than my suggestion. We could use events [1] to store payload data
> (e.g.
> > >> IP,
> > >> version, etc.)
> > >
> > >
> > > Yes, we could use events or some other similar API provided by GA.
> > >
> > >
> > >> What the download page CGI developed in? PHP?
> > >>
> > >
> > > To be honest, no clue. I guess someone in the community can figure it
> > out:
> > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >
> > >
> > >> However, I'm not sure whether storing this data in a 3rd party
> (Google)
> > is
> > >> compliant with the ASF policy. I guess it's no biggie, but if there's
> > >> doubt
> > >> in the PMC, it's better to ask legal@.
> > >
> > >
> > > I am not sure there is anything to ask about. The whole Ignite website
> is
> > > GA enabled, and all we are doing is accessing a standard web page from
> > the
> > > Ignite web site. The information gathered from GA is available only to
> > the
> > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > > question.
> > >
> > > Even ASF website itself uses GA: https://www.apache.org/
> > > foundation/policies/privacy.html
> > >
> > >
> > >> I think Cos said it's OK; maybe Roman can pitch in.
> > >>
> > >
> > >  Sure, would be nice to hear from Roman as well.
> > >
> > >
> > >> Cheers.
> > >>
> > >> [1]
> > >> https://developers.google.com/analytics/devguides/collection
> > >> /analyticsjs/events
> > >>
> > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >> dsetrakyan@apache.org>
> > >> wrote:
> > >>
> > >> > Raul,
> > >> >
> > >> > Could point about Javascript, it will not work because it executes
> in
> > >> the
> > >> > browser. This means we need a server-side script, like CGI we are
> > using
> > >> on
> > >> > our download page.
> > >> >
> > >> > How about this approach. We create something like ignite-version.cgi
> > >> script
> > >> > which will invoke a call to GA and then return the latest version.
> > >> >
> > >> > This should work, right?
> > >> >
> > >> > D.
> > >> >
> > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> raul.asf@evosent.com
> > >
> > >> > wrote:
> > >> >
> > >> > > Hey Dmitriy and all
> > >> > >
> > >> > > Also, since we have GA enabled for the website, we can track how
> > many
> > >> > times
> > >> > > > this page was accessed, which will be equal to the number of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by the
> > Ignite
> > >> > PMC.
> > >> > >
> > >> > >
> > >> > > Unfortunately this won't work because GA is loaded via Javascript
> on
> > >> the
> > >> > > browser, so Google will never receive the page hit.
> > >> > >
> > >> > > Given the constraints, the most viable solution is an HTTPS
> endpoint
> > >> > > running on ASF infra that Ignite invokes via a GET or POST
> request.
> > >> The
> > >> > > simplest thing is to write each request in a log file, along with
> > the
> > >> > > timestamp, the version reported by the client, maybe the IP (not
> > sure
> > >> > about
> > >> > > the ASF rules about this concerning privacy, I guess it's OK if
> you
> > >> make
> > >> > it
> > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> > creates
> > >> on
> > >> > > first startup or something.
> > >> > >
> > >> > > This endpoint would need some basic DDoS protection and
> blacklisting
> > >> to
> > >> > > prevent data spoofing.
> > >> > >
> > >> > > If we'll be implementing this endpoint anyway, then there's no
> point
> > >> > > placing another file on Git or elsewhere for reporting the latest
> > >> > versions:
> > >> > > the endpoint itself can return them.
> > >> > >
> > >> > > WDYT?
> > >> > >
> > >> > > Cheers.
> > >> > >
> > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > >> > dsetrakyan@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > Cos, Raul,
> > >> > > >
> > >> > > > Thanks for the feedback. I completely agree about Maven Central
> > >> being a
> > >> > > 3rd
> > >> > > > party repo (did not think about that initially). All your
> > >> suggestions
> > >> > > make
> > >> > > > sense, but I would like to keep it as simple as possible, and so
> > far
> > >> > > > everything suggested required GIT dependencies and extra work.
> > >> > > >
> > >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> > >> > website
> > >> > > > which will have only the latest version. Every time a node
> starts,
> > >> it
> > >> > > > receives the latest version from the page and suggests that
> users
> > >> > upgrade
> > >> > > > if needed.
> > >> > > >
> > >> > > > Also, since we have GA enabled for the website, we can track how
> > >> many
> > >> > > times
> > >> > > > this page was accessed, which will be equal to the number of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by the
> > Ignite
> > >> > PMC.
> > >> > > >
> > >> > > > This approach looks pretty innocent to me and everything is kept
> > and
> > >> > > > managed within Apache.
> > >> > > >
> > >> > > > Thoughts?
> > >> > > >
> > >> > > > D.
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > >> cos@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > I agree with Raul.
> > >> > > > > - providing a ping-back address to a 3rd party might be frown
> > >> upon by
> > >> > > > some.
> > >> > > > >   And might have a consequences like stats collection about
> > users'
> > >> > > > >   infrastructure.
> > >> > > > > - checking an ASF git-repo is easy and won't download any
> binary
> > >> > data:
> > >> > > > >   everything is clear text and could be easily monitored by
> any
> > of
> > >> > the
> > >> > > > > network
> > >> > > > >   diagnostic tools, shall it be required. But it involves a
> bit
> > of
> > >> > the
> > >> > > > > release
> > >> > > > >   discipline.
> > >> > > > > - the binary data download in the runtime is my main concern.
> > >> This is
> > >> > > the
> > >> > > > >   vector of MMA. What if someone gains the control over the
> > >> > repository
> > >> > > > and
> > >> > > > >   replaces the file with some malicious content.
> > >> > > > >
> > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
> call
> > >> to
> > >> > an
> > >> > > > > external script to check upon the atest software version
> > available
> > >> > for
> > >> > > > > download. In the past, the endpoint was running on a 3rd party
> > >> > server,
> > >> > > I
> > >> > > > > believe the best way would be to put this script on ASF infra
> > and
> > >> > have
> > >> > > > the
> > >> > > > > "update checker" running in a completely controlled
> environment.
> > >> > > > Actually,
> > >> > > > > I
> > >> > > > > recall we had this very discussion during the Incubation; I
> can
> > >> > > probably
> > >> > > > > dig
> > >> > > > > out the corresponding thread.
> > >> > > > >
> > >> > > > > Thoughts?
> > >> > > > >   Cok
> > >> > > > >
> > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > >> > > > > > Hey guys
> > >> > > > > >
> > >> > > > > > In my opinion, maven.org is still owned by a third party
> > >> > (Sonatype).
> > >> > > > We
> > >> > > > > > don't know what kind of data analysis or intelligence
> > extraction
> > >> > they
> > >> > > > > run.
> > >> > > > > >
> > >> > > > > > If Ignite servers all over the world were hitting maven.org
> > >> > > > periodically
> > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
> clear
> > >> > > > indication
> > >> > > > > > of who is running an Ignite server.
> > >> > > > > >
> > >> > > > > > They could reverse lookup the IP address and find out what
> > >> > > corporation
> > >> > > > it
> > >> > > > > > is.
> > >> > > > > >
> > >> > > > > > How about having Ignite check the ASF Git directly?
> > >> > > > > >
> > >> > > > > > We could use the Git ssh API, but that would require a new
> > >> > > dependency,
> > >> > > > > > which I advise against.
> > >> > > > > >
> > >> > > > > > Alternatively, we could have it scrape this HTML for new Git
> > >> tags:
> > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >> > > > > >
> > >> > > > > > Another option is to place a txt file in the root of the
> > master
> > >> > > branch
> > >> > > > > (e.g
> > >> > > > > > LATEST), containing a list of the latest GA versions for
> each
> > >> major
> > >> > > > > version
> > >> > > > > > line (1.x, 2.x).
> > >> > > > > >
> > >> > > > > > I would advocate this last option, but it requires somebody
> > >> > > remembering
> > >> > > > > to
> > >> > > > > > update the file with every release, unless we automate it
> > with a
> > >> > > Maven
> > >> > > > > > plugin.
> > >> > > > > >
> > >> > > > > > Hope that helps!
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org>
> > wrote:
> > >> > > > > >
> > >> > > > > > I see nothing wrong with this approach.
> > >> > > > > >
> > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is
> it
> > >> good
> > >> > > to
> > >> > > > > go?
> > >> > > > > >
> > >> > > > > > —
> > >> > > > > > Denis
> > >> > > > > >
> > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > >> > > > dsetrakyan@apache.org
> > >> > > > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Is everyone OK with this approach? Should I file a ticket
> on
> > >> it?
> > >> > > > > > >
> > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > >> > > > > dsetrakyan@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > >> Igniters,
> > >> > > > > > >>
> > >> > > > > > >> There has been lots of talk of proposals about various
> > usage
> > >> > > metrics
> > >> > > > > for
> > >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
> > the
> > >> > topic
> > >> > > > and
> > >> > > > > > >> propose something very simple and non-intrusive.
> > >> > > > > > >>
> > >> > > > > > >> 1. Update Checker
> > >> > > > > > >> The main purpose of the update checker is not to collect
> > >> > metrics,
> > >> > > > but
> > >> > > > > to
> > >> > > > > > >> notify users about a new version of Ignite by accessing
> > >> > maven.org
> > >> > > > and
> > >> > > > > > >> getting the version out of the metadata file:
> > >> > > > > > >> http://repo2.maven.org/maven2/
> > org/apache/ignite/ignite-core/
> > >> > > > > > >> maven-metadata.xml
> > >> > > > > > >>
> > >> > > > > > >> This way we do not send any information anywhere and, at
> > the
> > >> > same
> > >> > > > > time,
> > >> > > > > > >> urge our users to download and start using the latest
> > >> version of
> > >> > > > > Ignite.
> > >> > > > > > >>
> > >> > > > > > >> 2. Startup Counter
> > >> > > > > > >> This piece is optional, but we can also get an insight in
> > how
> > >> > many
> > >> > > > > times
> > >> > > > > > a
> > >> > > > > > >> certain Ignite release gets started. This is just a cool
> > >> metric
> > >> > > for
> > >> > > > > the
> > >> > > > > > >> community to gauge the project popularity. You can think
> of
> > >> it
> > >> > as
> > >> > > > of a
> > >> > > > > > page
> > >> > > > > > >> visit counter shown on many websites. We can even decide
> to
> > >> > > display
> > >> > > > > this
> > >> > > > > > >> counter on the Ignite website as well.
> > >> > > > > > >>
> > >> > > > > > >> To do this, we can simply add a JAR in maven for every
> > >> release,
> > >> > > e.g.
> > >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> > >> Every
> > >> > > time
> > >> > > > > an
> > >> > > > > > >> Ignite node starts, it will download this JAR in the
> > >> background.
> > >> > > > Then
> > >> > > > > we
> > >> > > > > > >> will be able to view the number of the total downloads
> for
> > >> this
> > >> > > JAR
> > >> > > > in
> > >> > > > > > >> Maven Central, which is essentially the number of starts
> of
> > >> > Ignite
> > >> > > > > nodes.
> > >> > > > > > >>
> > >> > > > > > >> *Note that neither of the above suggestions require
> Ignite
> > to
> > >> > send
> > >> > > > or
> > >> > > > > > >> track any user information whatsoever.*
> > >> > > > > > >>
> > >> > > > > > >> Please reply suggesting weather you are OK with this
> > >> approach.
> > >> > > > > > >>
> > >> > > > > > >> D.
> > >> > > > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Vladimir Ozerov <vo...@gridgain.com>.

Folks,

Can we add version of current node to web request? This way we will better
understand version distribution, what might help us with certain API
update/deprecate decisions
E.g. http://ignite.apache.org/latest.cgi&ver=2.2.0

Vladimir.

On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> I think it is safe to assume at this point that everyone is in general
> agreement, since there are no active objections.
>
> I have filed a ticket for the 2.3 release. Let's try to make it happen:
> https://issues.apache.org/jira/browse/IGNITE-6305
>
> D.
>
> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> >
> >
> > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <ra...@evosent.com>
> > wrote:
> >
> >> Yeah, I guess that's doable as well and requires less management effort
> >> than my suggestion. We could use events [1] to store payload data (e.g.
> >> IP,
> >> version, etc.)
> >
> >
> > Yes, we could use events or some other similar API provided by GA.
> >
> >
> >> What the download page CGI developed in? PHP?
> >>
> >
> > To be honest, no clue. I guess someone in the community can figure it
> out:
> > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >
> >
> >> However, I'm not sure whether storing this data in a 3rd party (Google)
> is
> >> compliant with the ASF policy. I guess it's no biggie, but if there's
> >> doubt
> >> in the PMC, it's better to ask legal@.
> >
> >
> > I am not sure there is anything to ask about. The whole Ignite website is
> > GA enabled, and all we are doing is accessing a standard web page from
> the
> > Ignite web site. The information gathered from GA is available only to
> the
> > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > question.
> >
> > Even ASF website itself uses GA: https://www.apache.org/
> > foundation/policies/privacy.html
> >
> >
> >> I think Cos said it's OK; maybe Roman can pitch in.
> >>
> >
> >  Sure, would be nice to hear from Roman as well.
> >
> >
> >> Cheers.
> >>
> >> [1]
> >> https://developers.google.com/analytics/devguides/collection
> >> /analyticsjs/events
> >>
> >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >> dsetrakyan@apache.org>
> >> wrote:
> >>
> >> > Raul,
> >> >
> >> > Could point about Javascript, it will not work because it executes in
> >> the
> >> > browser. This means we need a server-side script, like CGI we are
> using
> >> on
> >> > our download page.
> >> >
> >> > How about this approach. We create something like ignite-version.cgi
> >> script
> >> > which will invoke a call to GA and then return the latest version.
> >> >
> >> > This should work, right?
> >> >
> >> > D.
> >> >
> >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <raul.asf@evosent.com
> >
> >> > wrote:
> >> >
> >> > > Hey Dmitriy and all
> >> > >
> >> > > Also, since we have GA enabled for the website, we can track how
> many
> >> > times
> >> > > > this page was accessed, which will be equal to the number of
> starts.
> >> > This
> >> > > > way, the counter information is tracked and monitored by the
> Ignite
> >> > PMC.
> >> > >
> >> > >
> >> > > Unfortunately this won't work because GA is loaded via Javascript on
> >> the
> >> > > browser, so Google will never receive the page hit.
> >> > >
> >> > > Given the constraints, the most viable solution is an HTTPS endpoint
> >> > > running on ASF infra that Ignite invokes via a GET or POST request.
> >> The
> >> > > simplest thing is to write each request in a log file, along with
> the
> >> > > timestamp, the version reported by the client, maybe the IP (not
> sure
> >> > about
> >> > > the ASF rules about this concerning privacy, I guess it's OK if you
> >> make
> >> > it
> >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> creates
> >> on
> >> > > first startup or something.
> >> > >
> >> > > This endpoint would need some basic DDoS protection and blacklisting
> >> to
> >> > > prevent data spoofing.
> >> > >
> >> > > If we'll be implementing this endpoint anyway, then there's no point
> >> > > placing another file on Git or elsewhere for reporting the latest
> >> > versions:
> >> > > the endpoint itself can return them.
> >> > >
> >> > > WDYT?
> >> > >
> >> > > Cheers.
> >> > >
> >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >> > dsetrakyan@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Cos, Raul,
> >> > > >
> >> > > > Thanks for the feedback. I completely agree about Maven Central
> >> being a
> >> > > 3rd
> >> > > > party repo (did not think about that initially). All your
> >> suggestions
> >> > > make
> >> > > > sense, but I would like to keep it as simple as possible, and so
> far
> >> > > > everything suggested required GIT dependencies and extra work.
> >> > > >
> >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> >> > website
> >> > > > which will have only the latest version. Every time a node starts,
> >> it
> >> > > > receives the latest version from the page and suggests that users
> >> > upgrade
> >> > > > if needed.
> >> > > >
> >> > > > Also, since we have GA enabled for the website, we can track how
> >> many
> >> > > times
> >> > > > this page was accessed, which will be equal to the number of
> starts.
> >> > This
> >> > > > way, the counter information is tracked and monitored by the
> Ignite
> >> > PMC.
> >> > > >
> >> > > > This approach looks pretty innocent to me and everything is kept
> and
> >> > > > managed within Apache.
> >> > > >
> >> > > > Thoughts?
> >> > > >
> >> > > > D.
> >> > > >
> >> > > >
> >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >> cos@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > I agree with Raul.
> >> > > > > - providing a ping-back address to a 3rd party might be frown
> >> upon by
> >> > > > some.
> >> > > > >   And might have a consequences like stats collection about
> users'
> >> > > > >   infrastructure.
> >> > > > > - checking an ASF git-repo is easy and won't download any binary
> >> > data:
> >> > > > >   everything is clear text and could be easily monitored by any
> of
> >> > the
> >> > > > > network
> >> > > > >   diagnostic tools, shall it be required. But it involves a bit
> of
> >> > the
> >> > > > > release
> >> > > > >   discipline.
> >> > > > > - the binary data download in the runtime is my main concern.
> >> This is
> >> > > the
> >> > > > >   vector of MMA. What if someone gains the control over the
> >> > repository
> >> > > > and
> >> > > > >   replaces the file with some malicious content.
> >> > > > >
> >> > > > > As for the particular mechanism: IIRC Ignite used to make a call
> >> to
> >> > an
> >> > > > > external script to check upon the atest software version
> available
> >> > for
> >> > > > > download. In the past, the endpoint was running on a 3rd party
> >> > server,
> >> > > I
> >> > > > > believe the best way would be to put this script on ASF infra
> and
> >> > have
> >> > > > the
> >> > > > > "update checker" running in a completely controlled environment.
> >> > > > Actually,
> >> > > > > I
> >> > > > > recall we had this very discussion during the Incubation; I can
> >> > > probably
> >> > > > > dig
> >> > > > > out the corresponding thread.
> >> > > > >
> >> > > > > Thoughts?
> >> > > > >   Cok
> >> > > > >
> >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >> > > > > > Hey guys
> >> > > > > >
> >> > > > > > In my opinion, maven.org is still owned by a third party
> >> > (Sonatype).
> >> > > > We
> >> > > > > > don't know what kind of data analysis or intelligence
> extraction
> >> > they
> >> > > > > run.
> >> > > > > >
> >> > > > > > If Ignite servers all over the world were hitting maven.org
> >> > > > periodically
> >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> >> > > > indication
> >> > > > > > of who is running an Ignite server.
> >> > > > > >
> >> > > > > > They could reverse lookup the IP address and find out what
> >> > > corporation
> >> > > > it
> >> > > > > > is.
> >> > > > > >
> >> > > > > > How about having Ignite check the ASF Git directly?
> >> > > > > >
> >> > > > > > We could use the Git ssh API, but that would require a new
> >> > > dependency,
> >> > > > > > which I advise against.
> >> > > > > >
> >> > > > > > Alternatively, we could have it scrape this HTML for new Git
> >> tags:
> >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >> > > > > >
> >> > > > > > Another option is to place a txt file in the root of the
> master
> >> > > branch
> >> > > > > (e.g
> >> > > > > > LATEST), containing a list of the latest GA versions for each
> >> major
> >> > > > > version
> >> > > > > > line (1.x, 2.x).
> >> > > > > >
> >> > > > > > I would advocate this last option, but it requires somebody
> >> > > remembering
> >> > > > > to
> >> > > > > > update the file with every release, unless we automate it
> with a
> >> > > Maven
> >> > > > > > plugin.
> >> > > > > >
> >> > > > > > Hope that helps!
> >> > > > > >
> >> > > > > >
> >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org>
> wrote:
> >> > > > > >
> >> > > > > > I see nothing wrong with this approach.
> >> > > > > >
> >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it
> >> good
> >> > > to
> >> > > > > go?
> >> > > > > >
> >> > > > > > —
> >> > > > > > Denis
> >> > > > > >
> >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >> > > > dsetrakyan@apache.org
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > Is everyone OK with this approach? Should I file a ticket on
> >> it?
> >> > > > > > >
> >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >> > > > > dsetrakyan@apache.org>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > >> Igniters,
> >> > > > > > >>
> >> > > > > > >> There has been lots of talk of proposals about various
> usage
> >> > > metrics
> >> > > > > for
> >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
> the
> >> > topic
> >> > > > and
> >> > > > > > >> propose something very simple and non-intrusive.
> >> > > > > > >>
> >> > > > > > >> 1. Update Checker
> >> > > > > > >> The main purpose of the update checker is not to collect
> >> > metrics,
> >> > > > but
> >> > > > > to
> >> > > > > > >> notify users about a new version of Ignite by accessing
> >> > maven.org
> >> > > > and
> >> > > > > > >> getting the version out of the metadata file:
> >> > > > > > >> http://repo2.maven.org/maven2/
> org/apache/ignite/ignite-core/
> >> > > > > > >> maven-metadata.xml
> >> > > > > > >>
> >> > > > > > >> This way we do not send any information anywhere and, at
> the
> >> > same
> >> > > > > time,
> >> > > > > > >> urge our users to download and start using the latest
> >> version of
> >> > > > > Ignite.
> >> > > > > > >>
> >> > > > > > >> 2. Startup Counter
> >> > > > > > >> This piece is optional, but we can also get an insight in
> how
> >> > many
> >> > > > > times
> >> > > > > > a
> >> > > > > > >> certain Ignite release gets started. This is just a cool
> >> metric
> >> > > for
> >> > > > > the
> >> > > > > > >> community to gauge the project popularity. You can think of
> >> it
> >> > as
> >> > > > of a
> >> > > > > > page
> >> > > > > > >> visit counter shown on many websites. We can even decide to
> >> > > display
> >> > > > > this
> >> > > > > > >> counter on the Ignite website as well.
> >> > > > > > >>
> >> > > > > > >> To do this, we can simply add a JAR in maven for every
> >> release,
> >> > > e.g.
> >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> >> Every
> >> > > time
> >> > > > > an
> >> > > > > > >> Ignite node starts, it will download this JAR in the
> >> background.
> >> > > > Then
> >> > > > > we
> >> > > > > > >> will be able to view the number of the total downloads for
> >> this
> >> > > JAR
> >> > > > in
> >> > > > > > >> Maven Central, which is essentially the number of starts of
> >> > Ignite
> >> > > > > nodes.
> >> > > > > > >>
> >> > > > > > >> *Note that neither of the above suggestions require Ignite
> to
> >> > send
> >> > > > or
> >> > > > > > >> track any user information whatsoever.*
> >> > > > > > >>
> >> > > > > > >> Please reply suggesting weather you are OK with this
> >> approach.
> >> > > > > > >>
> >> > > > > > >> D.
> >> > > > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@apache.org>.

Cos,

I’m not sure the redirect will suit us. Remember the update notifier should do 2 things:

1. get the most recent version of Ignite stored in the file on the site [1].
2. Trigger GA.

The redirect does 2. but skips 1. Any technical tricks we can apply here?

[1] https://ignite.apache.org/latest

—
Denis

> On Oct 31, 2017, at 1:23 PM, Konstantin Boudnik <co...@boudnik.org> wrote:
> 
> Yes, that what I meant. It might not work per website (depends on how
> the server is configured), but server-wide redirects could be
> arranged.
> Also, there's a new discussion on the topic going on legal-discuss@
> right now [1]
> 
> 
> [1] https://is.gd/RPaWQt
> --
>  With regards,
> Konstantin (Cos) Boudnik
> 
> 
> On Tue, Oct 31, 2017 at 1:21 PM, Dmitriy Setrakyan
> <ds...@apache.org> wrote:
>> Cos, thanks for helping out! By redirect, do you mean changing the
>> .htaccess file to point a certain URL containing Ignite version to a GA
>> URL? In that case, do you know if it will be possible to still send the
>> latest Ignite version back as a response?
>> 
>> D.
>> 
>> On Tue, Oct 31, 2017 at 11:30 AM, Konstantin Boudnik <co...@apache.org> wrote:
>> 
>>> Looks like we are getting somewhere with this. Please check [1] for
>>> the latest comment from the Infra team. I guess we can start with
>>> asking for a server-side redirect to GA as the immediate step. And
>>> explore it further possibilities as per Greg's recommendations.
>>> 
>>> 
>>> [1] https://is.gd/CL88O6
>>> --
>>>  With regards,
>>> Konstantin (Cos) Boudnik
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>> 
>>> Disclaimer: Opinions expressed in this email are those of the author,
>>> and do not necessarily represent the views of any company the author
>>> might be affiliated with at the moment of writing.
>>> 
>>> 
>>> On Wed, Oct 25, 2017 at 8:21 AM, Denis Magda <dm...@apache.org> wrote:
>>>> Hi Cos,
>>>> 
>>>> Sure, will see you around there.
>>>> 
>>>> Anyway, a short summary is the following.
>>>> 
>>>> Starting Ignite 2.3 the nodes will be connecting to a *static* file [1]
>>> deployed on Ignite site to obtain the most recent version. If the file has
>>> a later version than the nodes will print out a message encouraging a user
>>> to do an upgrade.
>>>> 
>>>> On top of that, some community members proposed to trigger GA once the
>>> file [1] is loaded. But to achieve this the file has to become dynamic
>>> being able to execute simple CGI scripts. The INFRA turned this request
>>> down.
>>>> 
>>>> [1] https://ignite.apache.org/latest
>>>> [2] https://issues.apache.org/jira/browse/INFRA-15182
>>>> 
>>>> —
>>>> Denis
>>>> 
>>>>> On Oct 24, 2017, at 11:57 PM, Konstantin Boudnik <co...@apache.org>
>>> wrote:
>>>>> 
>>>>> Guys,
>>>>> 
>>>>> I had a quick chat with Nikita earlier today and it seems that we have
>>> came across of a blocker of some kind. With my member's hat on I would love
>>> to help to find a resolution that let us (and potentially other
>>> communities) to move forward.
>>>>> 
>>>>> Let's craft the message and circulate it with Infra and Greg. Denis,
>>> could you find me tomorrow at the IMCS and give me an insight into the
>>> technical side of it? I feel a bit at loss after going through this
>>> thread...
>>>>> --
>>>>> Regards,
>>>>> Cos
>>>>> 
>>>>> On October 2, 2017 7:31:48 AM PDT, Denis Magda <dm...@apache.org>
>>> wrote:
>>>>>> Dmitriy,
>>>>>> 
>>>>>> That’s the rule.  See replies in the ticket [1]
>>>>>> 
>>>>>> *Background: the TLP server is already pretty darned busy just serving
>>>>>> *static* sites. Dynamic operation for near-200 PMCs would bury the
>>>>>> machine. Our policy of "static-only websites" has been in place since
>>>>>> the Foundation started*
>>>>>> 
>>>>>> Download scripts seem to be the only exception and we as PMC don’t even
>>>>>> have access to them.
>>>>>> 
>>>>>> If you want to keep pushing this direction let’s craft a message to
>>>>>> Greg and Daniel directly. I don’t know what else to ask for here rather
>>>>>> than a virtual machine that’s conceivably to much for a single script
>>>>>> like that.
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/INFRA-15182
>>>>>> <https://issues.apache.org/jira/browse/INFRA-15182>
>>>>>> 
>>>>>> —
>>>>>> Denis
>>>>>> 
>>>>>>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
>>>>>> <vo...@gridgain.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I am not sure it is good idea to send requests to 3rd-party
>>>>>> addresses from
>>>>>>>> Ignite node. Let's do not make the same mistakes again.
>>>>>>>> 
>>>>>>> 
>>>>>>> Agree with Vladimir.
>>>>>>> 
>>>>>>> We obviously have CGI support on the website. Can someone explain why
>>>>>> CGI
>>>>>>> is not possible to use?
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
>>>>>> <an...@gridgain.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> We may directly send request to GA from Ignite node:
>>>>>>>>> https://developers.google.com/analytics/devguides/
>>>>>>>> collection/protocol/v1/
>>>>>>>>> <https://developers.google.com/analytics/devguides/
>>>>>>>> collection/protocol/v1/
>>>>>>>>>> 
>>>>>>>>> Latest version can be received from maven central:
>>>>>>>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>>>>>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>>>>>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
>>>>>>>>> написал(а):
>>>>>>>>>> 
>>>>>>>>>> Denis,
>>>>>>>>>> 
>>>>>>>>>> I am not sure I understand. We already do have CGI enabled for
>>>>>>>>>> download.cgi. Is there something else we need?
>>>>>>>>>> 
>>>>>>>>>> D.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> There is an obstacle. There is no way to execute the script using
>>>>>> PHP
>>>>>>>> or
>>>>>>>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>>>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>>>>>>>> 
>>>>>>>>>>> How else can we tackle this?
>>>>>>>>>>> 
>>>>>>>>>>> Denis
>>>>>>>>>>> 
>>>>>>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>>>>>>>> dsetrakyan@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I think it is safe to assume at this point that everyone is in
>>>>>>>> general
>>>>>>>>>>>> agreement, since there are no active objections.
>>>>>>>>>>>> 
>>>>>>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>>>>>>>> happen:
>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>>>>>>>> 
>>>>>>>>>>>> D.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>> <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>>>>>>> raul.asf@evosent.com
>>>>>>>>>>>> <javascript:;>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yeah, I guess that's doable as well and requires less
>>>>>> management
>>>>>>>>>>> effort
>>>>>>>>>>>>>> than my suggestion. We could use events [1] to store payload
>>>>>> data
>>>>>>>>>>> (e.g.
>>>>>>>>>>>>>> IP,
>>>>>>>>>>>>>> version, etc.)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes, we could use events or some other similar API provided by
>>>>>> GA.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> To be honest, no clue. I guess someone in the community can
>>>>>> figure
>>>>>>>> it
>>>>>>>>>>>> out:
>>>>>>>>>>>>> 
>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
>>>>>>>>>>> (Google)
>>>>>>>>>>>> is
>>>>>>>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
>>>>>>>> there's
>>>>>>>>>>>>>> doubt
>>>>>>>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>>>>>>>> website
>>>>>>>>>>> is
>>>>>>>>>>>>> GA enabled, and all we are doing is accessing a standard web
>>>>>> page
>>>>>>>> from
>>>>>>>>>>>> the
>>>>>>>>>>>>> Ignite web site. The information gathered from GA is available
>>>>>> only
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
>>>>>> this
>>>>>>>>>>>>> question.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>>>>>>>> foundation/policies/privacy.html
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>>>>>>>> /analyticsjs/events
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Raul,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Could point about Javascript, it will not work because it
>>>>>> executes
>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> browser. This means we need a server-side script, like CGI we
>>>>>> are
>>>>>>>>>>>> using
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> our download page.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> How about this approach. We create something like
>>>>>>>> ignite-version.cgi
>>>>>>>>>>>>>> script
>>>>>>>>>>>>>>> which will invoke a call to GA and then return the latest
>>>>>> version.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This should work, right?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>>>>>>>>> raul.asf@evosent.com
>>>>>>>>>>>> <javascript:;>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can track
>>>>>> how
>>>>>>>>>>>> many
>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>>>>>> of
>>>>>>>>>>>> starts.
>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>>>>>> the
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>> PMC.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Unfortunately this won't work because GA is loaded via
>>>>>> Javascript
>>>>>>>>>>> on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> browser, so Google will never receive the page hit.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>>>>>>>>>>> request.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> simplest thing is to write each request in a log file, along
>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> timestamp, the version reported by the client, maybe the IP
>>>>>> (not
>>>>>>>>>>>> sure
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
>>>>>> if
>>>>>>>>>>> you
>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
>>>>>> node
>>>>>>>>>>>> creates
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> first startup or something.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This endpoint would need some basic DDoS protection and
>>>>>>>>>>> blacklisting
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> prevent data spoofing.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's
>>>>>> no
>>>>>>>>>>> point
>>>>>>>>>>>>>>>> placing another file on Git or elsewhere for reporting the
>>>>>> latest
>>>>>>>>>>>>>>> versions:
>>>>>>>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
>>>>>> Central
>>>>>>>>>>>>>> being a
>>>>>>>>>>>>>>>> 3rd
>>>>>>>>>>>>>>>>> party repo (did not think about that initially). All your
>>>>>>>>>>>>>> suggestions
>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>> sense, but I would like to keep it as simple as possible,
>>>>>> and so
>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>> everything suggested required GIT dependencies and extra
>>>>>> work.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
>>>>>> Ignite
>>>>>>>>>>>>>>> website
>>>>>>>>>>>>>>>>> which will have only the latest version. Every time a node
>>>>>>>>>>> starts,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> receives the latest version from the page and suggests that
>>>>>>>>>>> users
>>>>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>>>>> if needed.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can
>>>>>> track how
>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>>>>>> of
>>>>>>>>>>>> starts.
>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>>>>>> the
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>> PMC.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This approach looks pretty innocent to me and everything is
>>>>>> kept
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> managed within Apache.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>>>>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
>>>>>> frown
>>>>>>>>>>>>>> upon by
>>>>>>>>>>>>>>>>> some.
>>>>>>>>>>>>>>>>>> And might have a consequences like stats collection about
>>>>>>>>>>>> users'
>>>>>>>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>>>>>>>>>>> binary
>>>>>>>>>>>>>>> data:
>>>>>>>>>>>>>>>>>> everything is clear text and could be easily monitored by
>>>>>>>>>>> any
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> network
>>>>>>>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
>>>>>>>>>>> bit
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>> discipline.
>>>>>>>>>>>>>>>>>> - the binary data download in the runtime is my main
>>>>>> concern.
>>>>>>>>>>>>>> This is
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
>>>>>>>>>>>>>>> repository
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> replaces the file with some malicious content.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make
>>>>>> a
>>>>>>>>>>> call
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> external script to check upon the atest software version
>>>>>>>>>>>> available
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
>>>>>> party
>>>>>>>>>>>>>>> server,
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> believe the best way would be to put this script on ASF
>>>>>> infra
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> "update checker" running in a completely controlled
>>>>>>>>>>> environment.
>>>>>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> recall we had this very discussion during the Incubation;
>>>>>> I
>>>>>>>>>>> can
>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>>> dig
>>>>>>>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>> Cok
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>>>>>>>>>>>>>>> (Sonatype).
>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
>>>>>>>>>>>> extraction
>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If Ignite servers all over the world were hitting
>>>>>> maven.org
>>>>>>>>>>>>>>>>> periodically
>>>>>>>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>>>>>>>>>>> clear
>>>>>>>>>>>>>>>>> indication
>>>>>>>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> They could reverse lookup the IP address and find out
>>>>>> what
>>>>>>>>>>>>>>>> corporation
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> is.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> We could use the Git ssh API, but that would require a
>>>>>> new
>>>>>>>>>>>>>>>> dependency,
>>>>>>>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
>>>>>> Git
>>>>>>>>>>>>>> tags:
>>>>>>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Another option is to place a txt file in the root of the
>>>>>>>>>>>> master
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>> (e.g
>>>>>>>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>>>>>>>>>>> each
>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would advocate this last option, but it requires
>>>>>> somebody
>>>>>>>>>>>>>>>> remembering
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> update the file with every release, unless we automate it
>>>>>>>>>>>> with a
>>>>>>>>>>>>>>>> Maven
>>>>>>>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>>>>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think?
>>>>>> Is
>>>>>>>>>>> it
>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> go?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a
>>>>>> ticket
>>>>>>>>>>> on
>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
>>>>>>>>>>>> usage
>>>>>>>>>>>>>>>> metrics
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to
>>>>>> resurrect
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>>>>>>>> The main purpose of the update checker is not to
>>>>>> collect
>>>>>>>>>>>>>>> metrics,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>>>>>>>>>>>>>>> maven.org
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>>>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> This way we do not send any information anywhere and,
>>>>>> at
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>> urge our users to download and start using the latest
>>>>>>>>>>>>>> version of
>>>>>>>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight
>>>>>> in
>>>>>>>>>>>> how
>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a
>>>>>> cool
>>>>>>>>>>>>>> metric
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> community to gauge the project popularity. You can
>>>>>> think
>>>>>>>>>>> of
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>> page
>>>>>>>>>>>>>>>>>>>>> visit counter shown on many websites. We can even
>>>>>> decide
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> display
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1
>>>>>> byte.
>>>>>>>>>>>>>> Every
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>>>>>>>>>>>>>> background.
>>>>>>>>>>>>>>>>> Then
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> will be able to view the number of the total downloads
>>>>>>>>>>> for
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> JAR
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> Maven Central, which is essentially the number of
>>>>>> starts
>>>>>>>>>>> of
>>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
>>>>>>>>>>> Ignite
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>>>>>>>>>>>>>> approach.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>> 
>>>

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@boudnik.org>.

Yes, that what I meant. It might not work per website (depends on how
the server is configured), but server-wide redirects could be
arranged.
Also, there's a new discussion on the topic going on legal-discuss@
right now [1]


[1] https://is.gd/RPaWQt
--
  With regards,
Konstantin (Cos) Boudnik


On Tue, Oct 31, 2017 at 1:21 PM, Dmitriy Setrakyan
<ds...@apache.org> wrote:
> Cos, thanks for helping out! By redirect, do you mean changing the
> .htaccess file to point a certain URL containing Ignite version to a GA
> URL? In that case, do you know if it will be possible to still send the
> latest Ignite version back as a response?
>
> D.
>
> On Tue, Oct 31, 2017 at 11:30 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> Looks like we are getting somewhere with this. Please check [1] for
>> the latest comment from the Infra team. I guess we can start with
>> asking for a server-side redirect to GA as the immediate step. And
>> explore it further possibilities as per Greg's recommendations.
>>
>>
>> [1] https://is.gd/CL88O6
>> --
>>   With regards,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Wed, Oct 25, 2017 at 8:21 AM, Denis Magda <dm...@apache.org> wrote:
>> > Hi Cos,
>> >
>> > Sure, will see you around there.
>> >
>> > Anyway, a short summary is the following.
>> >
>> > Starting Ignite 2.3 the nodes will be connecting to a *static* file [1]
>> deployed on Ignite site to obtain the most recent version. If the file has
>> a later version than the nodes will print out a message encouraging a user
>> to do an upgrade.
>> >
>> > On top of that, some community members proposed to trigger GA once the
>> file [1] is loaded. But to achieve this the file has to become dynamic
>> being able to execute simple CGI scripts. The INFRA turned this request
>> down.
>> >
>> > [1] https://ignite.apache.org/latest
>> > [2] https://issues.apache.org/jira/browse/INFRA-15182
>> >
>> > —
>> > Denis
>> >
>> >> On Oct 24, 2017, at 11:57 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> >>
>> >> Guys,
>> >>
>> >> I had a quick chat with Nikita earlier today and it seems that we have
>> came across of a blocker of some kind. With my member's hat on I would love
>> to help to find a resolution that let us (and potentially other
>> communities) to move forward.
>> >>
>> >> Let's craft the message and circulate it with Infra and Greg. Denis,
>> could you find me tomorrow at the IMCS and give me an insight into the
>> technical side of it? I feel a bit at loss after going through this
>> thread...
>> >> --
>> >> Regards,
>> >>  Cos
>> >>
>> >> On October 2, 2017 7:31:48 AM PDT, Denis Magda <dm...@apache.org>
>> wrote:
>> >>> Dmitriy,
>> >>>
>> >>> That’s the rule.  See replies in the ticket [1]
>> >>>
>> >>> *Background: the TLP server is already pretty darned busy just serving
>> >>> *static* sites. Dynamic operation for near-200 PMCs would bury the
>> >>> machine. Our policy of "static-only websites" has been in place since
>> >>> the Foundation started*
>> >>>
>> >>> Download scripts seem to be the only exception and we as PMC don’t even
>> >>> have access to them.
>> >>>
>> >>> If you want to keep pushing this direction let’s craft a message to
>> >>> Greg and Daniel directly. I don’t know what else to ask for here rather
>> >>> than a virtual machine that’s conceivably to much for a single script
>> >>> like that.
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/INFRA-15182
>> >>> <https://issues.apache.org/jira/browse/INFRA-15182>
>> >>>
>> >>> —
>> >>> Denis
>> >>>
>> >>>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
>> >>> wrote:
>> >>>>
>> >>>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
>> >>> <vo...@gridgain.com>
>> >>>> wrote:
>> >>>>
>> >>>>> I am not sure it is good idea to send requests to 3rd-party
>> >>> addresses from
>> >>>>> Ignite node. Let's do not make the same mistakes again.
>> >>>>>
>> >>>>
>> >>>> Agree with Vladimir.
>> >>>>
>> >>>> We obviously have CGI support on the website. Can someone explain why
>> >>> CGI
>> >>>> is not possible to use?
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
>> >>> <an...@gridgain.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> We may directly send request to GA from Ignite node:
>> >>>>>> https://developers.google.com/analytics/devguides/
>> >>>>> collection/protocol/v1/
>> >>>>>> <https://developers.google.com/analytics/devguides/
>> >>>>> collection/protocol/v1/
>> >>>>>>>
>> >>>>>> Latest version can be received from maven central:
>> >>>>>> https://repo1.maven.org/maven2/org/apache/ignite/
>> >>>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>> >>>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>> >>>>>>
>> >>>>>>
>> >>>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
>> >>>>>> написал(а):
>> >>>>>>>
>> >>>>>>> Denis,
>> >>>>>>>
>> >>>>>>> I am not sure I understand. We already do have CGI enabled for
>> >>>>>>> download.cgi. Is there something else we need?
>> >>>>>>>
>> >>>>>>> D.
>> >>>>>>>
>> >>>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
>> >>>>> wrote:
>> >>>>>>>
>> >>>>>>>> There is an obstacle. There is no way to execute the script using
>> >>> PHP
>> >>>>> or
>> >>>>>>>> similar sever side language and trigger GA as discussed earlier:
>> >>>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>> >>>>>>>>
>> >>>>>>>> How else can we tackle this?
>> >>>>>>>>
>> >>>>>>>> Denis
>> >>>>>>>>
>> >>>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>> >>>>>> dsetrakyan@apache.org>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> I think it is safe to assume at this point that everyone is in
>> >>>>> general
>> >>>>>>>>> agreement, since there are no active objections.
>> >>>>>>>>>
>> >>>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>> >>>>> happen:
>> >>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>> >>>>>>>>>
>> >>>>>>>>> D.
>> >>>>>>>>>
>> >>>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>> >>>>>>>> dsetrakyan@apache.org
>> >>>>>>>>> <javascript:;>>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>> >>>>> raul.asf@evosent.com
>> >>>>>>>>> <javascript:;>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Yeah, I guess that's doable as well and requires less
>> >>> management
>> >>>>>>>> effort
>> >>>>>>>>>>> than my suggestion. We could use events [1] to store payload
>> >>> data
>> >>>>>>>> (e.g.
>> >>>>>>>>>>> IP,
>> >>>>>>>>>>> version, etc.)
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Yes, we could use events or some other similar API provided by
>> >>> GA.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>> What the download page CGI developed in? PHP?
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> To be honest, no clue. I guess someone in the community can
>> >>> figure
>> >>>>> it
>> >>>>>>>>> out:
>> >>>>>>>>>>
>> >>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
>> >>>>>>>> (Google)
>> >>>>>>>>> is
>> >>>>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
>> >>>>> there's
>> >>>>>>>>>>> doubt
>> >>>>>>>>>>> in the PMC, it's better to ask legal@.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>> >>>>> website
>> >>>>>>>> is
>> >>>>>>>>>> GA enabled, and all we are doing is accessing a standard web
>> >>> page
>> >>>>> from
>> >>>>>>>>> the
>> >>>>>>>>>> Ignite web site. The information gathered from GA is available
>> >>> only
>> >>>>> to
>> >>>>>>>>> the
>> >>>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
>> >>> this
>> >>>>>>>>>> question.
>> >>>>>>>>>>
>> >>>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>> >>>>>>>>>> foundation/policies/privacy.html
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Sure, would be nice to hear from Roman as well.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>> Cheers.
>> >>>>>>>>>>>
>> >>>>>>>>>>> [1]
>> >>>>>>>>>>> https://developers.google.com/analytics/devguides/collection
>> >>>>>>>>>>> /analyticsjs/events
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>> >>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Raul,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Could point about Javascript, it will not work because it
>> >>> executes
>> >>>>>>>> in
>> >>>>>>>>>>> the
>> >>>>>>>>>>>> browser. This means we need a server-side script, like CGI we
>> >>> are
>> >>>>>>>>> using
>> >>>>>>>>>>> on
>> >>>>>>>>>>>> our download page.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> How about this approach. We create something like
>> >>>>> ignite-version.cgi
>> >>>>>>>>>>> script
>> >>>>>>>>>>>> which will invoke a call to GA and then return the latest
>> >>> version.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> This should work, right?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> D.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>> >>>>>>>> raul.asf@evosent.com
>> >>>>>>>>> <javascript:;>>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Hey Dmitriy and all
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Also, since we have GA enabled for the website, we can track
>> >>> how
>> >>>>>>>>> many
>> >>>>>>>>>>>> times
>> >>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>> >>> of
>> >>>>>>>>> starts.
>> >>>>>>>>>>>> This
>> >>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>> >>> the
>> >>>>>>>>> Ignite
>> >>>>>>>>>>>> PMC.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Unfortunately this won't work because GA is loaded via
>> >>> Javascript
>> >>>>>>>> on
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>> browser, so Google will never receive the page hit.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
>> >>>>>>>> endpoint
>> >>>>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>> >>>>>>>> request.
>> >>>>>>>>>>> The
>> >>>>>>>>>>>>> simplest thing is to write each request in a log file, along
>> >>> with
>> >>>>>>>>> the
>> >>>>>>>>>>>>> timestamp, the version reported by the client, maybe the IP
>> >>> (not
>> >>>>>>>>> sure
>> >>>>>>>>>>>> about
>> >>>>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
>> >>> if
>> >>>>>>>> you
>> >>>>>>>>>>> make
>> >>>>>>>>>>>> it
>> >>>>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
>> >>> node
>> >>>>>>>>> creates
>> >>>>>>>>>>> on
>> >>>>>>>>>>>>> first startup or something.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> This endpoint would need some basic DDoS protection and
>> >>>>>>>> blacklisting
>> >>>>>>>>>>> to
>> >>>>>>>>>>>>> prevent data spoofing.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's
>> >>> no
>> >>>>>>>> point
>> >>>>>>>>>>>>> placing another file on Git or elsewhere for reporting the
>> >>> latest
>> >>>>>>>>>>>> versions:
>> >>>>>>>>>>>>> the endpoint itself can return them.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> WDYT?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Cheers.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>> >>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Cos, Raul,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
>> >>> Central
>> >>>>>>>>>>> being a
>> >>>>>>>>>>>>> 3rd
>> >>>>>>>>>>>>>> party repo (did not think about that initially). All your
>> >>>>>>>>>>> suggestions
>> >>>>>>>>>>>>> make
>> >>>>>>>>>>>>>> sense, but I would like to keep it as simple as possible,
>> >>> and so
>> >>>>>>>>> far
>> >>>>>>>>>>>>>> everything suggested required GIT dependencies and extra
>> >>> work.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
>> >>> Ignite
>> >>>>>>>>>>>> website
>> >>>>>>>>>>>>>> which will have only the latest version. Every time a node
>> >>>>>>>> starts,
>> >>>>>>>>>>> it
>> >>>>>>>>>>>>>> receives the latest version from the page and suggests that
>> >>>>>>>> users
>> >>>>>>>>>>>> upgrade
>> >>>>>>>>>>>>>> if needed.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can
>> >>> track how
>> >>>>>>>>>>> many
>> >>>>>>>>>>>>> times
>> >>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>> >>> of
>> >>>>>>>>> starts.
>> >>>>>>>>>>>> This
>> >>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>> >>> the
>> >>>>>>>>> Ignite
>> >>>>>>>>>>>> PMC.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> This approach looks pretty innocent to me and everything is
>> >>> kept
>> >>>>>>>>> and
>> >>>>>>>>>>>>>> managed within Apache.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thoughts?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> D.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>> >>>>>>>>>>> cos@apache.org <javascript:;>>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I agree with Raul.
>> >>>>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
>> >>> frown
>> >>>>>>>>>>> upon by
>> >>>>>>>>>>>>>> some.
>> >>>>>>>>>>>>>>> And might have a consequences like stats collection about
>> >>>>>>>>> users'
>> >>>>>>>>>>>>>>> infrastructure.
>> >>>>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>> >>>>>>>> binary
>> >>>>>>>>>>>> data:
>> >>>>>>>>>>>>>>> everything is clear text and could be easily monitored by
>> >>>>>>>> any
>> >>>>>>>>> of
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> network
>> >>>>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
>> >>>>>>>> bit
>> >>>>>>>>> of
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> release
>> >>>>>>>>>>>>>>> discipline.
>> >>>>>>>>>>>>>>> - the binary data download in the runtime is my main
>> >>> concern.
>> >>>>>>>>>>> This is
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
>> >>>>>>>>>>>> repository
>> >>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>> replaces the file with some malicious content.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make
>> >>> a
>> >>>>>>>> call
>> >>>>>>>>>>> to
>> >>>>>>>>>>>> an
>> >>>>>>>>>>>>>>> external script to check upon the atest software version
>> >>>>>>>>> available
>> >>>>>>>>>>>> for
>> >>>>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
>> >>> party
>> >>>>>>>>>>>> server,
>> >>>>>>>>>>>>> I
>> >>>>>>>>>>>>>>> believe the best way would be to put this script on ASF
>> >>> infra
>> >>>>>>>>> and
>> >>>>>>>>>>>> have
>> >>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> "update checker" running in a completely controlled
>> >>>>>>>> environment.
>> >>>>>>>>>>>>>> Actually,
>> >>>>>>>>>>>>>>> I
>> >>>>>>>>>>>>>>> recall we had this very discussion during the Incubation;
>> >>> I
>> >>>>>>>> can
>> >>>>>>>>>>>>> probably
>> >>>>>>>>>>>>>>> dig
>> >>>>>>>>>>>>>>> out the corresponding thread.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thoughts?
>> >>>>>>>>>>>>>>> Cok
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> >>>>>>>>>>>>>>>> Hey guys
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>> >>>>>>>>>>>> (Sonatype).
>> >>>>>>>>>>>>>> We
>> >>>>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
>> >>>>>>>>> extraction
>> >>>>>>>>>>>> they
>> >>>>>>>>>>>>>>> run.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> If Ignite servers all over the world were hitting
>> >>> maven.org
>> >>>>>>>>>>>>>> periodically
>> >>>>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>> >>>>>>>> clear
>> >>>>>>>>>>>>>> indication
>> >>>>>>>>>>>>>>>> of who is running an Ignite server.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> They could reverse lookup the IP address and find out
>> >>> what
>> >>>>>>>>>>>>> corporation
>> >>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>> is.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> We could use the Git ssh API, but that would require a
>> >>> new
>> >>>>>>>>>>>>> dependency,
>> >>>>>>>>>>>>>>>> which I advise against.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
>> >>> Git
>> >>>>>>>>>>> tags:
>> >>>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Another option is to place a txt file in the root of the
>> >>>>>>>>> master
>> >>>>>>>>>>>>> branch
>> >>>>>>>>>>>>>>> (e.g
>> >>>>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>> >>>>>>>> each
>> >>>>>>>>>>> major
>> >>>>>>>>>>>>>>> version
>> >>>>>>>>>>>>>>>> line (1.x, 2.x).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I would advocate this last option, but it requires
>> >>> somebody
>> >>>>>>>>>>>>> remembering
>> >>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>> update the file with every release, unless we automate it
>> >>>>>>>>> with a
>> >>>>>>>>>>>>> Maven
>> >>>>>>>>>>>>>>>> plugin.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Hope that helps!
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>> >>>>>>>>> <javascript:;>> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I see nothing wrong with this approach.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think?
>> >>> Is
>> >>>>>>>> it
>> >>>>>>>>>>> good
>> >>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>> go?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> —
>> >>>>>>>>>>>>>>>> Denis
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>> >>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a
>> >>> ticket
>> >>>>>>>> on
>> >>>>>>>>>>> it?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> >>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>> >>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Igniters,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
>> >>>>>>>>> usage
>> >>>>>>>>>>>>> metrics
>> >>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to
>> >>> resurrect
>> >>>>>>>>> the
>> >>>>>>>>>>>> topic
>> >>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> 1. Update Checker
>> >>>>>>>>>>>>>>>>>> The main purpose of the update checker is not to
>> >>> collect
>> >>>>>>>>>>>> metrics,
>> >>>>>>>>>>>>>> but
>> >>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>> >>>>>>>>>>>> maven.org
>> >>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>> getting the version out of the metadata file:
>> >>>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>> >>>>>>>>> org/apache/ignite/ignite-core/
>> >>>>>>>>>>>>>>>>>> maven-metadata.xml
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> This way we do not send any information anywhere and,
>> >>> at
>> >>>>>>>>> the
>> >>>>>>>>>>>> same
>> >>>>>>>>>>>>>>> time,
>> >>>>>>>>>>>>>>>>>> urge our users to download and start using the latest
>> >>>>>>>>>>> version of
>> >>>>>>>>>>>>>>> Ignite.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> 2. Startup Counter
>> >>>>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight
>> >>> in
>> >>>>>>>>> how
>> >>>>>>>>>>>> many
>> >>>>>>>>>>>>>>> times
>> >>>>>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a
>> >>> cool
>> >>>>>>>>>>> metric
>> >>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> community to gauge the project popularity. You can
>> >>> think
>> >>>>>>>> of
>> >>>>>>>>>>> it
>> >>>>>>>>>>>> as
>> >>>>>>>>>>>>>> of a
>> >>>>>>>>>>>>>>>> page
>> >>>>>>>>>>>>>>>>>> visit counter shown on many websites. We can even
>> >>> decide
>> >>>>>>>> to
>> >>>>>>>>>>>>> display
>> >>>>>>>>>>>>>>> this
>> >>>>>>>>>>>>>>>>>> counter on the Ignite website as well.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>> >>>>>>>>>>> release,
>> >>>>>>>>>>>>> e.g.
>> >>>>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1
>> >>> byte.
>> >>>>>>>>>>> Every
>> >>>>>>>>>>>>> time
>> >>>>>>>>>>>>>>> an
>> >>>>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>> >>>>>>>>>>> background.
>> >>>>>>>>>>>>>> Then
>> >>>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>>>> will be able to view the number of the total downloads
>> >>>>>>>> for
>> >>>>>>>>>>> this
>> >>>>>>>>>>>>> JAR
>> >>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>> Maven Central, which is essentially the number of
>> >>> starts
>> >>>>>>>> of
>> >>>>>>>>>>>> Ignite
>> >>>>>>>>>>>>>>> nodes.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
>> >>>>>>>> Ignite
>> >>>>>>>>> to
>> >>>>>>>>>>>> send
>> >>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>>> track any user information whatsoever.*
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>> >>>>>>>>>>> approach.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> D.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >
>>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

Cos, thanks for helping out! By redirect, do you mean changing the
.htaccess file to point a certain URL containing Ignite version to a GA
URL? In that case, do you know if it will be possible to still send the
latest Ignite version back as a response?

D.

On Tue, Oct 31, 2017 at 11:30 AM, Konstantin Boudnik <co...@apache.org> wrote:

> Looks like we are getting somewhere with this. Please check [1] for
> the latest comment from the Infra team. I guess we can start with
> asking for a server-side redirect to GA as the immediate step. And
> explore it further possibilities as per Greg's recommendations.
>
>
> [1] https://is.gd/CL88O6
> --
>   With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Wed, Oct 25, 2017 at 8:21 AM, Denis Magda <dm...@apache.org> wrote:
> > Hi Cos,
> >
> > Sure, will see you around there.
> >
> > Anyway, a short summary is the following.
> >
> > Starting Ignite 2.3 the nodes will be connecting to a *static* file [1]
> deployed on Ignite site to obtain the most recent version. If the file has
> a later version than the nodes will print out a message encouraging a user
> to do an upgrade.
> >
> > On top of that, some community members proposed to trigger GA once the
> file [1] is loaded. But to achieve this the file has to become dynamic
> being able to execute simple CGI scripts. The INFRA turned this request
> down.
> >
> > [1] https://ignite.apache.org/latest
> > [2] https://issues.apache.org/jira/browse/INFRA-15182
> >
> > —
> > Denis
> >
> >> On Oct 24, 2017, at 11:57 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>
> >> Guys,
> >>
> >> I had a quick chat with Nikita earlier today and it seems that we have
> came across of a blocker of some kind. With my member's hat on I would love
> to help to find a resolution that let us (and potentially other
> communities) to move forward.
> >>
> >> Let's craft the message and circulate it with Infra and Greg. Denis,
> could you find me tomorrow at the IMCS and give me an insight into the
> technical side of it? I feel a bit at loss after going through this
> thread...
> >> --
> >> Regards,
> >>  Cos
> >>
> >> On October 2, 2017 7:31:48 AM PDT, Denis Magda <dm...@apache.org>
> wrote:
> >>> Dmitriy,
> >>>
> >>> That’s the rule.  See replies in the ticket [1]
> >>>
> >>> *Background: the TLP server is already pretty darned busy just serving
> >>> *static* sites. Dynamic operation for near-200 PMCs would bury the
> >>> machine. Our policy of "static-only websites" has been in place since
> >>> the Foundation started*
> >>>
> >>> Download scripts seem to be the only exception and we as PMC don’t even
> >>> have access to them.
> >>>
> >>> If you want to keep pushing this direction let’s craft a message to
> >>> Greg and Daniel directly. I don’t know what else to ask for here rather
> >>> than a virtual machine that’s conceivably to much for a single script
> >>> like that.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/INFRA-15182
> >>> <https://issues.apache.org/jira/browse/INFRA-15182>
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
> >>> wrote:
> >>>>
> >>>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
> >>> <vo...@gridgain.com>
> >>>> wrote:
> >>>>
> >>>>> I am not sure it is good idea to send requests to 3rd-party
> >>> addresses from
> >>>>> Ignite node. Let's do not make the same mistakes again.
> >>>>>
> >>>>
> >>>> Agree with Vladimir.
> >>>>
> >>>> We obviously have CGI support on the website. Can someone explain why
> >>> CGI
> >>>> is not possible to use?
> >>>>
> >>>>
> >>>>>
> >>>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
> >>> <an...@gridgain.com>
> >>>>> wrote:
> >>>>>
> >>>>>> We may directly send request to GA from Ignite node:
> >>>>>> https://developers.google.com/analytics/devguides/
> >>>>> collection/protocol/v1/
> >>>>>> <https://developers.google.com/analytics/devguides/
> >>>>> collection/protocol/v1/
> >>>>>>>
> >>>>>> Latest version can be received from maven central:
> >>>>>> https://repo1.maven.org/maven2/org/apache/ignite/
> >>>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
> >>>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
> >>>>>>
> >>>>>>
> >>>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
> >>>>>> написал(а):
> >>>>>>>
> >>>>>>> Denis,
> >>>>>>>
> >>>>>>> I am not sure I understand. We already do have CGI enabled for
> >>>>>>> download.cgi. Is there something else we need?
> >>>>>>>
> >>>>>>> D.
> >>>>>>>
> >>>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> There is an obstacle. There is no way to execute the script using
> >>> PHP
> >>>>> or
> >>>>>>>> similar sever side language and trigger GA as discussed earlier:
> >>>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
> >>>>>>>>
> >>>>>>>> How else can we tackle this?
> >>>>>>>>
> >>>>>>>> Denis
> >>>>>>>>
> >>>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> >>>>>> dsetrakyan@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I think it is safe to assume at this point that everyone is in
> >>>>> general
> >>>>>>>>> agreement, since there are no active objections.
> >>>>>>>>>
> >>>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
> >>>>> happen:
> >>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >>>>>>>> dsetrakyan@apache.org
> >>>>>>>>> <javascript:;>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> >>>>> raul.asf@evosent.com
> >>>>>>>>> <javascript:;>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Yeah, I guess that's doable as well and requires less
> >>> management
> >>>>>>>> effort
> >>>>>>>>>>> than my suggestion. We could use events [1] to store payload
> >>> data
> >>>>>>>> (e.g.
> >>>>>>>>>>> IP,
> >>>>>>>>>>> version, etc.)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Yes, we could use events or some other similar API provided by
> >>> GA.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> What the download page CGI developed in? PHP?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> To be honest, no clue. I guess someone in the community can
> >>> figure
> >>>>> it
> >>>>>>>>> out:
> >>>>>>>>>>
> >>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
> >>>>>>>> (Google)
> >>>>>>>>> is
> >>>>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
> >>>>> there's
> >>>>>>>>>>> doubt
> >>>>>>>>>>> in the PMC, it's better to ask legal@.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
> >>>>> website
> >>>>>>>> is
> >>>>>>>>>> GA enabled, and all we are doing is accessing a standard web
> >>> page
> >>>>> from
> >>>>>>>>> the
> >>>>>>>>>> Ignite web site. The information gathered from GA is available
> >>> only
> >>>>> to
> >>>>>>>>> the
> >>>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
> >>> this
> >>>>>>>>>> question.
> >>>>>>>>>>
> >>>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
> >>>>>>>>>> foundation/policies/privacy.html
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sure, would be nice to hear from Roman as well.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Cheers.
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>> https://developers.google.com/analytics/devguides/collection
> >>>>>>>>>>> /analyticsjs/events
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Raul,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Could point about Javascript, it will not work because it
> >>> executes
> >>>>>>>> in
> >>>>>>>>>>> the
> >>>>>>>>>>>> browser. This means we need a server-side script, like CGI we
> >>> are
> >>>>>>>>> using
> >>>>>>>>>>> on
> >>>>>>>>>>>> our download page.
> >>>>>>>>>>>>
> >>>>>>>>>>>> How about this approach. We create something like
> >>>>> ignite-version.cgi
> >>>>>>>>>>> script
> >>>>>>>>>>>> which will invoke a call to GA and then return the latest
> >>> version.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This should work, right?
> >>>>>>>>>>>>
> >>>>>>>>>>>> D.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> >>>>>>>> raul.asf@evosent.com
> >>>>>>>>> <javascript:;>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey Dmitriy and all
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, since we have GA enabled for the website, we can track
> >>> how
> >>>>>>>>> many
> >>>>>>>>>>>> times
> >>>>>>>>>>>>>> this page was accessed, which will be equal to the number
> >>> of
> >>>>>>>>> starts.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>> way, the counter information is tracked and monitored by
> >>> the
> >>>>>>>>> Ignite
> >>>>>>>>>>>> PMC.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Unfortunately this won't work because GA is loaded via
> >>> Javascript
> >>>>>>>> on
> >>>>>>>>>>> the
> >>>>>>>>>>>>> browser, so Google will never receive the page hit.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
> >>>>>>>> endpoint
> >>>>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
> >>>>>>>> request.
> >>>>>>>>>>> The
> >>>>>>>>>>>>> simplest thing is to write each request in a log file, along
> >>> with
> >>>>>>>>> the
> >>>>>>>>>>>>> timestamp, the version reported by the client, maybe the IP
> >>> (not
> >>>>>>>>> sure
> >>>>>>>>>>>> about
> >>>>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
> >>> if
> >>>>>>>> you
> >>>>>>>>>>> make
> >>>>>>>>>>>> it
> >>>>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
> >>> node
> >>>>>>>>> creates
> >>>>>>>>>>> on
> >>>>>>>>>>>>> first startup or something.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This endpoint would need some basic DDoS protection and
> >>>>>>>> blacklisting
> >>>>>>>>>>> to
> >>>>>>>>>>>>> prevent data spoofing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's
> >>> no
> >>>>>>>> point
> >>>>>>>>>>>>> placing another file on Git or elsewhere for reporting the
> >>> latest
> >>>>>>>>>>>> versions:
> >>>>>>>>>>>>> the endpoint itself can return them.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cos, Raul,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
> >>> Central
> >>>>>>>>>>> being a
> >>>>>>>>>>>>> 3rd
> >>>>>>>>>>>>>> party repo (did not think about that initially). All your
> >>>>>>>>>>> suggestions
> >>>>>>>>>>>>> make
> >>>>>>>>>>>>>> sense, but I would like to keep it as simple as possible,
> >>> and so
> >>>>>>>>> far
> >>>>>>>>>>>>>> everything suggested required GIT dependencies and extra
> >>> work.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
> >>> Ignite
> >>>>>>>>>>>> website
> >>>>>>>>>>>>>> which will have only the latest version. Every time a node
> >>>>>>>> starts,
> >>>>>>>>>>> it
> >>>>>>>>>>>>>> receives the latest version from the page and suggests that
> >>>>>>>> users
> >>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>> if needed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can
> >>> track how
> >>>>>>>>>>> many
> >>>>>>>>>>>>> times
> >>>>>>>>>>>>>> this page was accessed, which will be equal to the number
> >>> of
> >>>>>>>>> starts.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>> way, the counter information is tracked and monitored by
> >>> the
> >>>>>>>>> Ignite
> >>>>>>>>>>>> PMC.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This approach looks pretty innocent to me and everything is
> >>> kept
> >>>>>>>>> and
> >>>>>>>>>>>>>> managed within Apache.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >>>>>>>>>>> cos@apache.org <javascript:;>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I agree with Raul.
> >>>>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
> >>> frown
> >>>>>>>>>>> upon by
> >>>>>>>>>>>>>> some.
> >>>>>>>>>>>>>>> And might have a consequences like stats collection about
> >>>>>>>>> users'
> >>>>>>>>>>>>>>> infrastructure.
> >>>>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
> >>>>>>>> binary
> >>>>>>>>>>>> data:
> >>>>>>>>>>>>>>> everything is clear text and could be easily monitored by
> >>>>>>>> any
> >>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
> >>>>>>>> bit
> >>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>> discipline.
> >>>>>>>>>>>>>>> - the binary data download in the runtime is my main
> >>> concern.
> >>>>>>>>>>> This is
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
> >>>>>>>>>>>> repository
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> replaces the file with some malicious content.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make
> >>> a
> >>>>>>>> call
> >>>>>>>>>>> to
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>> external script to check upon the atest software version
> >>>>>>>>> available
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
> >>> party
> >>>>>>>>>>>> server,
> >>>>>>>>>>>>> I
> >>>>>>>>>>>>>>> believe the best way would be to put this script on ASF
> >>> infra
> >>>>>>>>> and
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> "update checker" running in a completely controlled
> >>>>>>>> environment.
> >>>>>>>>>>>>>> Actually,
> >>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>> recall we had this very discussion during the Incubation;
> >>> I
> >>>>>>>> can
> >>>>>>>>>>>>> probably
> >>>>>>>>>>>>>>> dig
> >>>>>>>>>>>>>>> out the corresponding thread.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>>>>> Cok
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >>>>>>>>>>>>>>>> Hey guys
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
> >>>>>>>>>>>> (Sonatype).
> >>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
> >>>>>>>>> extraction
> >>>>>>>>>>>> they
> >>>>>>>>>>>>>>> run.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If Ignite servers all over the world were hitting
> >>> maven.org
> >>>>>>>>>>>>>> periodically
> >>>>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
> >>>>>>>> clear
> >>>>>>>>>>>>>> indication
> >>>>>>>>>>>>>>>> of who is running an Ignite server.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> They could reverse lookup the IP address and find out
> >>> what
> >>>>>>>>>>>>> corporation
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> is.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> We could use the Git ssh API, but that would require a
> >>> new
> >>>>>>>>>>>>> dependency,
> >>>>>>>>>>>>>>>> which I advise against.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
> >>> Git
> >>>>>>>>>>> tags:
> >>>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Another option is to place a txt file in the root of the
> >>>>>>>>> master
> >>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>> (e.g
> >>>>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
> >>>>>>>> each
> >>>>>>>>>>> major
> >>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>> line (1.x, 2.x).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would advocate this last option, but it requires
> >>> somebody
> >>>>>>>>>>>>> remembering
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> update the file with every release, unless we automate it
> >>>>>>>>> with a
> >>>>>>>>>>>>> Maven
> >>>>>>>>>>>>>>>> plugin.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hope that helps!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> >>>>>>>>> <javascript:;>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I see nothing wrong with this approach.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think?
> >>> Is
> >>>>>>>> it
> >>>>>>>>>>> good
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> go?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> —
> >>>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a
> >>> ticket
> >>>>>>>> on
> >>>>>>>>>>> it?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
> >>>>>>>>> usage
> >>>>>>>>>>>>> metrics
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to
> >>> resurrect
> >>>>>>>>> the
> >>>>>>>>>>>> topic
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1. Update Checker
> >>>>>>>>>>>>>>>>>> The main purpose of the update checker is not to
> >>> collect
> >>>>>>>>>>>> metrics,
> >>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
> >>>>>>>>>>>> maven.org
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> getting the version out of the metadata file:
> >>>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
> >>>>>>>>> org/apache/ignite/ignite-core/
> >>>>>>>>>>>>>>>>>> maven-metadata.xml
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This way we do not send any information anywhere and,
> >>> at
> >>>>>>>>> the
> >>>>>>>>>>>> same
> >>>>>>>>>>>>>>> time,
> >>>>>>>>>>>>>>>>>> urge our users to download and start using the latest
> >>>>>>>>>>> version of
> >>>>>>>>>>>>>>> Ignite.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 2. Startup Counter
> >>>>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight
> >>> in
> >>>>>>>>> how
> >>>>>>>>>>>> many
> >>>>>>>>>>>>>>> times
> >>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a
> >>> cool
> >>>>>>>>>>> metric
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> community to gauge the project popularity. You can
> >>> think
> >>>>>>>> of
> >>>>>>>>>>> it
> >>>>>>>>>>>> as
> >>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>> page
> >>>>>>>>>>>>>>>>>> visit counter shown on many websites. We can even
> >>> decide
> >>>>>>>> to
> >>>>>>>>>>>>> display
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> counter on the Ignite website as well.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
> >>>>>>>>>>> release,
> >>>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1
> >>> byte.
> >>>>>>>>>>> Every
> >>>>>>>>>>>>> time
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
> >>>>>>>>>>> background.
> >>>>>>>>>>>>>> Then
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> will be able to view the number of the total downloads
> >>>>>>>> for
> >>>>>>>>>>> this
> >>>>>>>>>>>>> JAR
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> Maven Central, which is essentially the number of
> >>> starts
> >>>>>>>> of
> >>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>>> nodes.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
> >>>>>>>> Ignite
> >>>>>>>>> to
> >>>>>>>>>>>> send
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> track any user information whatsoever.*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
> >>>>>>>>>>> approach.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@apache.org>.

Looks like we are getting somewhere with this. Please check [1] for
the latest comment from the Infra team. I guess we can start with
asking for a server-side redirect to GA as the immediate step. And
explore it further possibilities as per Greg's recommendations.


[1] https://is.gd/CL88O6
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Wed, Oct 25, 2017 at 8:21 AM, Denis Magda <dm...@apache.org> wrote:
> Hi Cos,
>
> Sure, will see you around there.
>
> Anyway, a short summary is the following.
>
> Starting Ignite 2.3 the nodes will be connecting to a *static* file [1] deployed on Ignite site to obtain the most recent version. If the file has a later version than the nodes will print out a message encouraging a user to do an upgrade.
>
> On top of that, some community members proposed to trigger GA once the file [1] is loaded. But to achieve this the file has to become dynamic being able to execute simple CGI scripts. The INFRA turned this request down.
>
> [1] https://ignite.apache.org/latest
> [2] https://issues.apache.org/jira/browse/INFRA-15182
>
> —
> Denis
>
>> On Oct 24, 2017, at 11:57 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>> Guys,
>>
>> I had a quick chat with Nikita earlier today and it seems that we have came across of a blocker of some kind. With my member's hat on I would love to help to find a resolution that let us (and potentially other communities) to move forward.
>>
>> Let's craft the message and circulate it with Infra and Greg. Denis, could you find me tomorrow at the IMCS and give me an insight into the technical side of it? I feel a bit at loss after going through this thread...
>> --
>> Regards,
>>  Cos
>>
>> On October 2, 2017 7:31:48 AM PDT, Denis Magda <dm...@apache.org> wrote:
>>> Dmitriy,
>>>
>>> That’s the rule.  See replies in the ticket [1]
>>>
>>> *Background: the TLP server is already pretty darned busy just serving
>>> *static* sites. Dynamic operation for near-200 PMCs would bury the
>>> machine. Our policy of "static-only websites" has been in place since
>>> the Foundation started*
>>>
>>> Download scripts seem to be the only exception and we as PMC don’t even
>>> have access to them.
>>>
>>> If you want to keep pushing this direction let’s craft a message to
>>> Greg and Daniel directly. I don’t know what else to ask for here rather
>>> than a virtual machine that’s conceivably to much for a single script
>>> like that.
>>>
>>> [1] https://issues.apache.org/jira/browse/INFRA-15182
>>> <https://issues.apache.org/jira/browse/INFRA-15182>
>>>
>>> —
>>> Denis
>>>
>>>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
>>> wrote:
>>>>
>>>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
>>> <vo...@gridgain.com>
>>>> wrote:
>>>>
>>>>> I am not sure it is good idea to send requests to 3rd-party
>>> addresses from
>>>>> Ignite node. Let's do not make the same mistakes again.
>>>>>
>>>>
>>>> Agree with Vladimir.
>>>>
>>>> We obviously have CGI support on the website. Can someone explain why
>>> CGI
>>>> is not possible to use?
>>>>
>>>>
>>>>>
>>>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
>>> <an...@gridgain.com>
>>>>> wrote:
>>>>>
>>>>>> We may directly send request to GA from Ignite node:
>>>>>> https://developers.google.com/analytics/devguides/
>>>>> collection/protocol/v1/
>>>>>> <https://developers.google.com/analytics/devguides/
>>>>> collection/protocol/v1/
>>>>>>>
>>>>>> Latest version can be received from maven central:
>>>>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>>>>>
>>>>>>
>>>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
>>>>>> написал(а):
>>>>>>>
>>>>>>> Denis,
>>>>>>>
>>>>>>> I am not sure I understand. We already do have CGI enabled for
>>>>>>> download.cgi. Is there something else we need?
>>>>>>>
>>>>>>> D.
>>>>>>>
>>>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
>>>>> wrote:
>>>>>>>
>>>>>>>> There is an obstacle. There is no way to execute the script using
>>> PHP
>>>>> or
>>>>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>>>>>
>>>>>>>> How else can we tackle this?
>>>>>>>>
>>>>>>>> Denis
>>>>>>>>
>>>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I think it is safe to assume at this point that everyone is in
>>>>> general
>>>>>>>>> agreement, since there are no active objections.
>>>>>>>>>
>>>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>>>>> happen:
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>>>>> dsetrakyan@apache.org
>>>>>>>>> <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>>>> raul.asf@evosent.com
>>>>>>>>> <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yeah, I guess that's doable as well and requires less
>>> management
>>>>>>>> effort
>>>>>>>>>>> than my suggestion. We could use events [1] to store payload
>>> data
>>>>>>>> (e.g.
>>>>>>>>>>> IP,
>>>>>>>>>>> version, etc.)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, we could use events or some other similar API provided by
>>> GA.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> To be honest, no clue. I guess someone in the community can
>>> figure
>>>>> it
>>>>>>>>> out:
>>>>>>>>>>
>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
>>>>>>>> (Google)
>>>>>>>>> is
>>>>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
>>>>> there's
>>>>>>>>>>> doubt
>>>>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>>>>> website
>>>>>>>> is
>>>>>>>>>> GA enabled, and all we are doing is accessing a standard web
>>> page
>>>>> from
>>>>>>>>> the
>>>>>>>>>> Ignite web site. The information gathered from GA is available
>>> only
>>>>> to
>>>>>>>>> the
>>>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
>>> this
>>>>>>>>>> question.
>>>>>>>>>>
>>>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>>>>> foundation/policies/privacy.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Cheers.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>>>>> /analyticsjs/events
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Raul,
>>>>>>>>>>>>
>>>>>>>>>>>> Could point about Javascript, it will not work because it
>>> executes
>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>> browser. This means we need a server-side script, like CGI we
>>> are
>>>>>>>>> using
>>>>>>>>>>> on
>>>>>>>>>>>> our download page.
>>>>>>>>>>>>
>>>>>>>>>>>> How about this approach. We create something like
>>>>> ignite-version.cgi
>>>>>>>>>>> script
>>>>>>>>>>>> which will invoke a call to GA and then return the latest
>>> version.
>>>>>>>>>>>>
>>>>>>>>>>>> This should work, right?
>>>>>>>>>>>>
>>>>>>>>>>>> D.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>>>>>> raul.asf@evosent.com
>>>>>>>>> <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can track
>>> how
>>>>>>>>> many
>>>>>>>>>>>> times
>>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>>> of
>>>>>>>>> starts.
>>>>>>>>>>>> This
>>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>>> the
>>>>>>>>> Ignite
>>>>>>>>>>>> PMC.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unfortunately this won't work because GA is loaded via
>>> Javascript
>>>>>>>> on
>>>>>>>>>>> the
>>>>>>>>>>>>> browser, so Google will never receive the page hit.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
>>>>>>>> endpoint
>>>>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>>>>>>>> request.
>>>>>>>>>>> The
>>>>>>>>>>>>> simplest thing is to write each request in a log file, along
>>> with
>>>>>>>>> the
>>>>>>>>>>>>> timestamp, the version reported by the client, maybe the IP
>>> (not
>>>>>>>>> sure
>>>>>>>>>>>> about
>>>>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
>>> if
>>>>>>>> you
>>>>>>>>>>> make
>>>>>>>>>>>> it
>>>>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
>>> node
>>>>>>>>> creates
>>>>>>>>>>> on
>>>>>>>>>>>>> first startup or something.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This endpoint would need some basic DDoS protection and
>>>>>>>> blacklisting
>>>>>>>>>>> to
>>>>>>>>>>>>> prevent data spoofing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's
>>> no
>>>>>>>> point
>>>>>>>>>>>>> placing another file on Git or elsewhere for reporting the
>>> latest
>>>>>>>>>>>> versions:
>>>>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
>>> Central
>>>>>>>>>>> being a
>>>>>>>>>>>>> 3rd
>>>>>>>>>>>>>> party repo (did not think about that initially). All your
>>>>>>>>>>> suggestions
>>>>>>>>>>>>> make
>>>>>>>>>>>>>> sense, but I would like to keep it as simple as possible,
>>> and so
>>>>>>>>> far
>>>>>>>>>>>>>> everything suggested required GIT dependencies and extra
>>> work.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
>>> Ignite
>>>>>>>>>>>> website
>>>>>>>>>>>>>> which will have only the latest version. Every time a node
>>>>>>>> starts,
>>>>>>>>>>> it
>>>>>>>>>>>>>> receives the latest version from the page and suggests that
>>>>>>>> users
>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>> if needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can
>>> track how
>>>>>>>>>>> many
>>>>>>>>>>>>> times
>>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>>> of
>>>>>>>>> starts.
>>>>>>>>>>>> This
>>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>>> the
>>>>>>>>> Ignite
>>>>>>>>>>>> PMC.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This approach looks pretty innocent to me and everything is
>>> kept
>>>>>>>>> and
>>>>>>>>>>>>>> managed within Apache.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
>>> frown
>>>>>>>>>>> upon by
>>>>>>>>>>>>>> some.
>>>>>>>>>>>>>>> And might have a consequences like stats collection about
>>>>>>>>> users'
>>>>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>>>>>>>> binary
>>>>>>>>>>>> data:
>>>>>>>>>>>>>>> everything is clear text and could be easily monitored by
>>>>>>>> any
>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> network
>>>>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
>>>>>>>> bit
>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>> discipline.
>>>>>>>>>>>>>>> - the binary data download in the runtime is my main
>>> concern.
>>>>>>>>>>> This is
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
>>>>>>>>>>>> repository
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> replaces the file with some malicious content.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make
>>> a
>>>>>>>> call
>>>>>>>>>>> to
>>>>>>>>>>>> an
>>>>>>>>>>>>>>> external script to check upon the atest software version
>>>>>>>>> available
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
>>> party
>>>>>>>>>>>> server,
>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> believe the best way would be to put this script on ASF
>>> infra
>>>>>>>>> and
>>>>>>>>>>>> have
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> "update checker" running in a completely controlled
>>>>>>>> environment.
>>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> recall we had this very discussion during the Incubation;
>>> I
>>>>>>>> can
>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>> dig
>>>>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>> Cok
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>>>>>>>>>>>> (Sonatype).
>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
>>>>>>>>> extraction
>>>>>>>>>>>> they
>>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If Ignite servers all over the world were hitting
>>> maven.org
>>>>>>>>>>>>>> periodically
>>>>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>>>>>>>> clear
>>>>>>>>>>>>>> indication
>>>>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> They could reverse lookup the IP address and find out
>>> what
>>>>>>>>>>>>> corporation
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> is.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We could use the Git ssh API, but that would require a
>>> new
>>>>>>>>>>>>> dependency,
>>>>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
>>> Git
>>>>>>>>>>> tags:
>>>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Another option is to place a txt file in the root of the
>>>>>>>>> master
>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>> (e.g
>>>>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>>>>>>>> each
>>>>>>>>>>> major
>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would advocate this last option, but it requires
>>> somebody
>>>>>>>>>>>>> remembering
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> update the file with every release, unless we automate it
>>>>>>>>> with a
>>>>>>>>>>>>> Maven
>>>>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think?
>>> Is
>>>>>>>> it
>>>>>>>>>>> good
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> go?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a
>>> ticket
>>>>>>>> on
>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
>>>>>>>>> usage
>>>>>>>>>>>>> metrics
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to
>>> resurrect
>>>>>>>>> the
>>>>>>>>>>>> topic
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>>>>> The main purpose of the update checker is not to
>>> collect
>>>>>>>>>>>> metrics,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>>>>>>>>>>>> maven.org
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This way we do not send any information anywhere and,
>>> at
>>>>>>>>> the
>>>>>>>>>>>> same
>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>> urge our users to download and start using the latest
>>>>>>>>>>> version of
>>>>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight
>>> in
>>>>>>>>> how
>>>>>>>>>>>> many
>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a
>>> cool
>>>>>>>>>>> metric
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> community to gauge the project popularity. You can
>>> think
>>>>>>>> of
>>>>>>>>>>> it
>>>>>>>>>>>> as
>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>> page
>>>>>>>>>>>>>>>>>> visit counter shown on many websites. We can even
>>> decide
>>>>>>>> to
>>>>>>>>>>>>> display
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>>>>>>>>>>> release,
>>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1
>>> byte.
>>>>>>>>>>> Every
>>>>>>>>>>>>> time
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>>>>>>>>>>> background.
>>>>>>>>>>>>>> Then
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> will be able to view the number of the total downloads
>>>>>>>> for
>>>>>>>>>>> this
>>>>>>>>>>>>> JAR
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> Maven Central, which is essentially the number of
>>> starts
>>>>>>>> of
>>>>>>>>>>>> Ignite
>>>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
>>>>>>>> Ignite
>>>>>>>>> to
>>>>>>>>>>>> send
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>>>>>>>>>>> approach.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@apache.org>.

Hi Cos,

Sure, will see you around there. 

Anyway, a short summary is the following.

Starting Ignite 2.3 the nodes will be connecting to a *static* file [1] deployed on Ignite site to obtain the most recent version. If the file has a later version than the nodes will print out a message encouraging a user to do an upgrade. 

On top of that, some community members proposed to trigger GA once the file [1] is loaded. But to achieve this the file has to become dynamic being able to execute simple CGI scripts. The INFRA turned this request down.

[1] https://ignite.apache.org/latest
[2] https://issues.apache.org/jira/browse/INFRA-15182

—
Denis

> On Oct 24, 2017, at 11:57 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> Guys,
> 
> I had a quick chat with Nikita earlier today and it seems that we have came across of a blocker of some kind. With my member's hat on I would love to help to find a resolution that let us (and potentially other communities) to move forward.
> 
> Let's craft the message and circulate it with Infra and Greg. Denis, could you find me tomorrow at the IMCS and give me an insight into the technical side of it? I feel a bit at loss after going through this thread...
> --
> Regards,
>  Cos
> 
> On October 2, 2017 7:31:48 AM PDT, Denis Magda <dm...@apache.org> wrote:
>> Dmitriy, 
>> 
>> That’s the rule.  See replies in the ticket [1]
>> 
>> *Background: the TLP server is already pretty darned busy just serving
>> *static* sites. Dynamic operation for near-200 PMCs would bury the
>> machine. Our policy of "static-only websites" has been in place since
>> the Foundation started*
>> 
>> Download scripts seem to be the only exception and we as PMC don’t even
>> have access to them.
>> 
>> If you want to keep pushing this direction let’s craft a message to
>> Greg and Daniel directly. I don’t know what else to ask for here rather
>> than a virtual machine that’s conceivably to much for a single script
>> like that.
>> 
>> [1] https://issues.apache.org/jira/browse/INFRA-15182
>> <https://issues.apache.org/jira/browse/INFRA-15182>
>> 
>> —
>> Denis
>> 
>>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>>> 
>>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
>> <vo...@gridgain.com>
>>> wrote:
>>> 
>>>> I am not sure it is good idea to send requests to 3rd-party
>> addresses from
>>>> Ignite node. Let's do not make the same mistakes again.
>>>> 
>>> 
>>> Agree with Vladimir.
>>> 
>>> We obviously have CGI support on the website. Can someone explain why
>> CGI
>>> is not possible to use?
>>> 
>>> 
>>>> 
>>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
>> <an...@gridgain.com>
>>>> wrote:
>>>> 
>>>>> We may directly send request to GA from Ignite node:
>>>>> https://developers.google.com/analytics/devguides/
>>>> collection/protocol/v1/
>>>>> <https://developers.google.com/analytics/devguides/
>>>> collection/protocol/v1/
>>>>>> 
>>>>> Latest version can be received from maven central:
>>>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>>>> 
>>>>> 
>>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
>>>>> написал(а):
>>>>>> 
>>>>>> Denis,
>>>>>> 
>>>>>> I am not sure I understand. We already do have CGI enabled for
>>>>>> download.cgi. Is there something else we need?
>>>>>> 
>>>>>> D.
>>>>>> 
>>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
>>>> wrote:
>>>>>> 
>>>>>>> There is an obstacle. There is no way to execute the script using
>> PHP
>>>> or
>>>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>>>> 
>>>>>>> How else can we tackle this?
>>>>>>> 
>>>>>>> Denis
>>>>>>> 
>>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I think it is safe to assume at this point that everyone is in
>>>> general
>>>>>>>> agreement, since there are no active objections.
>>>>>>>> 
>>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>>>> happen:
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>>>> 
>>>>>>>> D.
>>>>>>>> 
>>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>>>> dsetrakyan@apache.org
>>>>>>>> <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>>> raul.asf@evosent.com
>>>>>>>> <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Yeah, I guess that's doable as well and requires less
>> management
>>>>>>> effort
>>>>>>>>>> than my suggestion. We could use events [1] to store payload
>> data
>>>>>>> (e.g.
>>>>>>>>>> IP,
>>>>>>>>>> version, etc.)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Yes, we could use events or some other similar API provided by
>> GA.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> To be honest, no clue. I guess someone in the community can
>> figure
>>>> it
>>>>>>>> out:
>>>>>>>>> 
>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
>>>>>>> (Google)
>>>>>>>> is
>>>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
>>>> there's
>>>>>>>>>> doubt
>>>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>>>> website
>>>>>>> is
>>>>>>>>> GA enabled, and all we are doing is accessing a standard web
>> page
>>>> from
>>>>>>>> the
>>>>>>>>> Ignite web site. The information gathered from GA is available
>> only
>>>> to
>>>>>>>> the
>>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
>> this
>>>>>>>>> question.
>>>>>>>>> 
>>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>>>> foundation/policies/privacy.html
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Cheers.
>>>>>>>>>> 
>>>>>>>>>> [1]
>>>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>>>> /analyticsjs/events
>>>>>>>>>> 
>>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Raul,
>>>>>>>>>>> 
>>>>>>>>>>> Could point about Javascript, it will not work because it
>> executes
>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>> browser. This means we need a server-side script, like CGI we
>> are
>>>>>>>> using
>>>>>>>>>> on
>>>>>>>>>>> our download page.
>>>>>>>>>>> 
>>>>>>>>>>> How about this approach. We create something like
>>>> ignite-version.cgi
>>>>>>>>>> script
>>>>>>>>>>> which will invoke a call to GA and then return the latest
>> version.
>>>>>>>>>>> 
>>>>>>>>>>> This should work, right?
>>>>>>>>>>> 
>>>>>>>>>>> D.
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>>>>> raul.asf@evosent.com
>>>>>>>> <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>>>> 
>>>>>>>>>>>> Also, since we have GA enabled for the website, we can track
>> how
>>>>>>>> many
>>>>>>>>>>> times
>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>> of
>>>>>>>> starts.
>>>>>>>>>>> This
>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>> the
>>>>>>>> Ignite
>>>>>>>>>>> PMC.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Unfortunately this won't work because GA is loaded via
>> Javascript
>>>>>>> on
>>>>>>>>>> the
>>>>>>>>>>>> browser, so Google will never receive the page hit.
>>>>>>>>>>>> 
>>>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
>>>>>>> endpoint
>>>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>>>>>>> request.
>>>>>>>>>> The
>>>>>>>>>>>> simplest thing is to write each request in a log file, along
>> with
>>>>>>>> the
>>>>>>>>>>>> timestamp, the version reported by the client, maybe the IP
>> (not
>>>>>>>> sure
>>>>>>>>>>> about
>>>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
>> if
>>>>>>> you
>>>>>>>>>> make
>>>>>>>>>>> it
>>>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
>> node
>>>>>>>> creates
>>>>>>>>>> on
>>>>>>>>>>>> first startup or something.
>>>>>>>>>>>> 
>>>>>>>>>>>> This endpoint would need some basic DDoS protection and
>>>>>>> blacklisting
>>>>>>>>>> to
>>>>>>>>>>>> prevent data spoofing.
>>>>>>>>>>>> 
>>>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's
>> no
>>>>>>> point
>>>>>>>>>>>> placing another file on Git or elsewhere for reporting the
>> latest
>>>>>>>>>>> versions:
>>>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>>>> 
>>>>>>>>>>>> WDYT?
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
>> Central
>>>>>>>>>> being a
>>>>>>>>>>>> 3rd
>>>>>>>>>>>>> party repo (did not think about that initially). All your
>>>>>>>>>> suggestions
>>>>>>>>>>>> make
>>>>>>>>>>>>> sense, but I would like to keep it as simple as possible,
>> and so
>>>>>>>> far
>>>>>>>>>>>>> everything suggested required GIT dependencies and extra
>> work.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
>> Ignite
>>>>>>>>>>> website
>>>>>>>>>>>>> which will have only the latest version. Every time a node
>>>>>>> starts,
>>>>>>>>>> it
>>>>>>>>>>>>> receives the latest version from the page and suggests that
>>>>>>> users
>>>>>>>>>>> upgrade
>>>>>>>>>>>>> if needed.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Also, since we have GA enabled for the website, we can
>> track how
>>>>>>>>>> many
>>>>>>>>>>>> times
>>>>>>>>>>>>> this page was accessed, which will be equal to the number
>> of
>>>>>>>> starts.
>>>>>>>>>>> This
>>>>>>>>>>>>> way, the counter information is tracked and monitored by
>> the
>>>>>>>> Ignite
>>>>>>>>>>> PMC.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This approach looks pretty innocent to me and everything is
>> kept
>>>>>>>> and
>>>>>>>>>>>>> managed within Apache.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> D.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
>> frown
>>>>>>>>>> upon by
>>>>>>>>>>>>> some.
>>>>>>>>>>>>>> And might have a consequences like stats collection about
>>>>>>>> users'
>>>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>>>>>>> binary
>>>>>>>>>>> data:
>>>>>>>>>>>>>> everything is clear text and could be easily monitored by
>>>>>>> any
>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>> network
>>>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
>>>>>>> bit
>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>> release
>>>>>>>>>>>>>> discipline.
>>>>>>>>>>>>>> - the binary data download in the runtime is my main
>> concern.
>>>>>>>>>> This is
>>>>>>>>>>>> the
>>>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
>>>>>>>>>>> repository
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> replaces the file with some malicious content.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make
>> a
>>>>>>> call
>>>>>>>>>> to
>>>>>>>>>>> an
>>>>>>>>>>>>>> external script to check upon the atest software version
>>>>>>>> available
>>>>>>>>>>> for
>>>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
>> party
>>>>>>>>>>> server,
>>>>>>>>>>>> I
>>>>>>>>>>>>>> believe the best way would be to put this script on ASF
>> infra
>>>>>>>> and
>>>>>>>>>>> have
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> "update checker" running in a completely controlled
>>>>>>> environment.
>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>> recall we had this very discussion during the Incubation;
>> I
>>>>>>> can
>>>>>>>>>>>> probably
>>>>>>>>>>>>>> dig
>>>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>> Cok
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>>>>>>>>>>> (Sonatype).
>>>>>>>>>>>>> We
>>>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
>>>>>>>> extraction
>>>>>>>>>>> they
>>>>>>>>>>>>>> run.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If Ignite servers all over the world were hitting
>> maven.org
>>>>>>>>>>>>> periodically
>>>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>>>>>>> clear
>>>>>>>>>>>>> indication
>>>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> They could reverse lookup the IP address and find out
>> what
>>>>>>>>>>>> corporation
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> is.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We could use the Git ssh API, but that would require a
>> new
>>>>>>>>>>>> dependency,
>>>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
>> Git
>>>>>>>>>> tags:
>>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Another option is to place a txt file in the root of the
>>>>>>>> master
>>>>>>>>>>>> branch
>>>>>>>>>>>>>> (e.g
>>>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>>>>>>> each
>>>>>>>>>> major
>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I would advocate this last option, but it requires
>> somebody
>>>>>>>>>>>> remembering
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> update the file with every release, unless we automate it
>>>>>>>> with a
>>>>>>>>>>>> Maven
>>>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think?
>> Is
>>>>>>> it
>>>>>>>>>> good
>>>>>>>>>>>> to
>>>>>>>>>>>>>> go?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a
>> ticket
>>>>>>> on
>>>>>>>>>> it?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
>>>>>>>> usage
>>>>>>>>>>>> metrics
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to
>> resurrect
>>>>>>>> the
>>>>>>>>>>> topic
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>>>> The main purpose of the update checker is not to
>> collect
>>>>>>>>>>> metrics,
>>>>>>>>>>>>> but
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>>>>>>>>>>> maven.org
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This way we do not send any information anywhere and,
>> at
>>>>>>>> the
>>>>>>>>>>> same
>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>> urge our users to download and start using the latest
>>>>>>>>>> version of
>>>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight
>> in
>>>>>>>> how
>>>>>>>>>>> many
>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a
>> cool
>>>>>>>>>> metric
>>>>>>>>>>>> for
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> community to gauge the project popularity. You can
>> think
>>>>>>> of
>>>>>>>>>> it
>>>>>>>>>>> as
>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>> page
>>>>>>>>>>>>>>>>> visit counter shown on many websites. We can even
>> decide
>>>>>>> to
>>>>>>>>>>>> display
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>>>>>>>>>> release,
>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1
>> byte.
>>>>>>>>>> Every
>>>>>>>>>>>> time
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>>>>>>>>>> background.
>>>>>>>>>>>>> Then
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> will be able to view the number of the total downloads
>>>>>>> for
>>>>>>>>>> this
>>>>>>>>>>>> JAR
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> Maven Central, which is essentially the number of
>> starts
>>>>>>> of
>>>>>>>>>>> Ignite
>>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
>>>>>>> Ignite
>>>>>>>> to
>>>>>>>>>>> send
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>>>>>>>>>> approach.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>>

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@apache.org>.

Guys,

I had a quick chat with Nikita earlier today and it seems that we have came across of a blocker of some kind. With my member's hat on I would love to help to find a resolution that let us (and potentially other communities) to move forward.

Let's craft the message and circulate it with Infra and Greg. Denis, could you find me tomorrow at the IMCS and give me an insight into the technical side of it? I feel a bit at loss after going through this thread...
--
Regards,
  Cos

On October 2, 2017 7:31:48 AM PDT, Denis Magda <dm...@apache.org> wrote:
>Dmitriy, 
>
>That’s the rule.  See replies in the ticket [1]
>
>*Background: the TLP server is already pretty darned busy just serving
>*static* sites. Dynamic operation for near-200 PMCs would bury the
>machine. Our policy of "static-only websites" has been in place since
>the Foundation started*
>
>Download scripts seem to be the only exception and we as PMC don’t even
>have access to them.
>
>If you want to keep pushing this direction let’s craft a message to
>Greg and Daniel directly. I don’t know what else to ask for here rather
>than a virtual machine that’s conceivably to much for a single script
>like that.
>
>[1] https://issues.apache.org/jira/browse/INFRA-15182
><https://issues.apache.org/jira/browse/INFRA-15182>
>
>—
>Denis
>
>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
>wrote:
>> 
>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
><vo...@gridgain.com>
>> wrote:
>> 
>>> I am not sure it is good idea to send requests to 3rd-party
>addresses from
>>> Ignite node. Let's do not make the same mistakes again.
>>> 
>> 
>> Agree with Vladimir.
>> 
>> We obviously have CGI support on the website. Can someone explain why
>CGI
>> is not possible to use?
>> 
>> 
>>> 
>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
><an...@gridgain.com>
>>> wrote:
>>> 
>>>> We may directly send request to GA from Ignite node:
>>>> https://developers.google.com/analytics/devguides/
>>> collection/protocol/v1/
>>>> <https://developers.google.com/analytics/devguides/
>>> collection/protocol/v1/
>>>>> 
>>>> Latest version can be received from maven central:
>>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>>> 
>>>> 
>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
>>>> написал(а):
>>>>> 
>>>>> Denis,
>>>>> 
>>>>> I am not sure I understand. We already do have CGI enabled for
>>>>> download.cgi. Is there something else we need?
>>>>> 
>>>>> D.
>>>>> 
>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
>>> wrote:
>>>>> 
>>>>>> There is an obstacle. There is no way to execute the script using
>PHP
>>> or
>>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>>> 
>>>>>> How else can we tackle this?
>>>>>> 
>>>>>> Denis
>>>>>> 
>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>>> dsetrakyan@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> I think it is safe to assume at this point that everyone is in
>>> general
>>>>>>> agreement, since there are no active objections.
>>>>>>> 
>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>>> happen:
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>>> 
>>>>>>> D.
>>>>>>> 
>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org
>>>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>> raul.asf@evosent.com
>>>>>>> <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Yeah, I guess that's doable as well and requires less
>management
>>>>>> effort
>>>>>>>>> than my suggestion. We could use events [1] to store payload
>data
>>>>>> (e.g.
>>>>>>>>> IP,
>>>>>>>>> version, etc.)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Yes, we could use events or some other similar API provided by
>GA.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> To be honest, no clue. I guess someone in the community can
>figure
>>> it
>>>>>>> out:
>>>>>>>>
>https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
>>>>>> (Google)
>>>>>>> is
>>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
>>> there's
>>>>>>>>> doubt
>>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>>> website
>>>>>> is
>>>>>>>> GA enabled, and all we are doing is accessing a standard web
>page
>>> from
>>>>>>> the
>>>>>>>> Ignite web site. The information gathered from GA is available
>only
>>> to
>>>>>>> the
>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
>this
>>>>>>>> question.
>>>>>>>> 
>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>>> foundation/policies/privacy.html
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Cheers.
>>>>>>>>> 
>>>>>>>>> [1]
>>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>>> /analyticsjs/events
>>>>>>>>> 
>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Raul,
>>>>>>>>>> 
>>>>>>>>>> Could point about Javascript, it will not work because it
>executes
>>>>>> in
>>>>>>>>> the
>>>>>>>>>> browser. This means we need a server-side script, like CGI we
>are
>>>>>>> using
>>>>>>>>> on
>>>>>>>>>> our download page.
>>>>>>>>>> 
>>>>>>>>>> How about this approach. We create something like
>>> ignite-version.cgi
>>>>>>>>> script
>>>>>>>>>> which will invoke a call to GA and then return the latest
>version.
>>>>>>>>>> 
>>>>>>>>>> This should work, right?
>>>>>>>>>> 
>>>>>>>>>> D.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>>>> raul.asf@evosent.com
>>>>>>> <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>>> 
>>>>>>>>>>> Also, since we have GA enabled for the website, we can track
>how
>>>>>>> many
>>>>>>>>>> times
>>>>>>>>>>>> this page was accessed, which will be equal to the number
>of
>>>>>>> starts.
>>>>>>>>>> This
>>>>>>>>>>>> way, the counter information is tracked and monitored by
>the
>>>>>>> Ignite
>>>>>>>>>> PMC.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Unfortunately this won't work because GA is loaded via
>Javascript
>>>>>> on
>>>>>>>>> the
>>>>>>>>>>> browser, so Google will never receive the page hit.
>>>>>>>>>>> 
>>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
>>>>>> endpoint
>>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>>>>>> request.
>>>>>>>>> The
>>>>>>>>>>> simplest thing is to write each request in a log file, along
>with
>>>>>>> the
>>>>>>>>>>> timestamp, the version reported by the client, maybe the IP
>(not
>>>>>>> sure
>>>>>>>>>> about
>>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
>if
>>>>>> you
>>>>>>>>> make
>>>>>>>>>> it
>>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
>node
>>>>>>> creates
>>>>>>>>> on
>>>>>>>>>>> first startup or something.
>>>>>>>>>>> 
>>>>>>>>>>> This endpoint would need some basic DDoS protection and
>>>>>> blacklisting
>>>>>>>>> to
>>>>>>>>>>> prevent data spoofing.
>>>>>>>>>>> 
>>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's
>no
>>>>>> point
>>>>>>>>>>> placing another file on Git or elsewhere for reporting the
>latest
>>>>>>>>>> versions:
>>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>>> 
>>>>>>>>>>> WDYT?
>>>>>>>>>>> 
>>>>>>>>>>> Cheers.
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
>Central
>>>>>>>>> being a
>>>>>>>>>>> 3rd
>>>>>>>>>>>> party repo (did not think about that initially). All your
>>>>>>>>> suggestions
>>>>>>>>>>> make
>>>>>>>>>>>> sense, but I would like to keep it as simple as possible,
>and so
>>>>>>> far
>>>>>>>>>>>> everything suggested required GIT dependencies and extra
>work.
>>>>>>>>>>>> 
>>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
>Ignite
>>>>>>>>>> website
>>>>>>>>>>>> which will have only the latest version. Every time a node
>>>>>> starts,
>>>>>>>>> it
>>>>>>>>>>>> receives the latest version from the page and suggests that
>>>>>> users
>>>>>>>>>> upgrade
>>>>>>>>>>>> if needed.
>>>>>>>>>>>> 
>>>>>>>>>>>> Also, since we have GA enabled for the website, we can
>track how
>>>>>>>>> many
>>>>>>>>>>> times
>>>>>>>>>>>> this page was accessed, which will be equal to the number
>of
>>>>>>> starts.
>>>>>>>>>> This
>>>>>>>>>>>> way, the counter information is tracked and monitored by
>the
>>>>>>> Ignite
>>>>>>>>>> PMC.
>>>>>>>>>>>> 
>>>>>>>>>>>> This approach looks pretty innocent to me and everything is
>kept
>>>>>>> and
>>>>>>>>>>>> managed within Apache.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>> 
>>>>>>>>>>>> D.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
>frown
>>>>>>>>> upon by
>>>>>>>>>>>> some.
>>>>>>>>>>>>> And might have a consequences like stats collection about
>>>>>>> users'
>>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>>>>>> binary
>>>>>>>>>> data:
>>>>>>>>>>>>> everything is clear text and could be easily monitored by
>>>>>> any
>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>> network
>>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
>>>>>> bit
>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>> release
>>>>>>>>>>>>> discipline.
>>>>>>>>>>>>> - the binary data download in the runtime is my main
>concern.
>>>>>>>>> This is
>>>>>>>>>>> the
>>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
>>>>>>>>>> repository
>>>>>>>>>>>> and
>>>>>>>>>>>>> replaces the file with some malicious content.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make
>a
>>>>>> call
>>>>>>>>> to
>>>>>>>>>> an
>>>>>>>>>>>>> external script to check upon the atest software version
>>>>>>> available
>>>>>>>>>> for
>>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
>party
>>>>>>>>>> server,
>>>>>>>>>>> I
>>>>>>>>>>>>> believe the best way would be to put this script on ASF
>infra
>>>>>>> and
>>>>>>>>>> have
>>>>>>>>>>>> the
>>>>>>>>>>>>> "update checker" running in a completely controlled
>>>>>> environment.
>>>>>>>>>>>> Actually,
>>>>>>>>>>>>> I
>>>>>>>>>>>>> recall we had this very discussion during the Incubation;
>I
>>>>>> can
>>>>>>>>>>> probably
>>>>>>>>>>>>> dig
>>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>> Cok
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>>>>>>>>>> (Sonatype).
>>>>>>>>>>>> We
>>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
>>>>>>> extraction
>>>>>>>>>> they
>>>>>>>>>>>>> run.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If Ignite servers all over the world were hitting
>maven.org
>>>>>>>>>>>> periodically
>>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>>>>>> clear
>>>>>>>>>>>> indication
>>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> They could reverse lookup the IP address and find out
>what
>>>>>>>>>>> corporation
>>>>>>>>>>>> it
>>>>>>>>>>>>>> is.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We could use the Git ssh API, but that would require a
>new
>>>>>>>>>>> dependency,
>>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
>Git
>>>>>>>>> tags:
>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Another option is to place a txt file in the root of the
>>>>>>> master
>>>>>>>>>>> branch
>>>>>>>>>>>>> (e.g
>>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>>>>>> each
>>>>>>>>> major
>>>>>>>>>>>>> version
>>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would advocate this last option, but it requires
>somebody
>>>>>>>>>>> remembering
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> update the file with every release, unless we automate it
>>>>>>> with a
>>>>>>>>>>> Maven
>>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think?
>Is
>>>>>> it
>>>>>>>>> good
>>>>>>>>>>> to
>>>>>>>>>>>>> go?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> —
>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a
>ticket
>>>>>> on
>>>>>>>>> it?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
>>>>>>> usage
>>>>>>>>>>> metrics
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to
>resurrect
>>>>>>> the
>>>>>>>>>> topic
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>>> The main purpose of the update checker is not to
>collect
>>>>>>>>>> metrics,
>>>>>>>>>>>> but
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>>>>>>>>>> maven.org
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This way we do not send any information anywhere and,
>at
>>>>>>> the
>>>>>>>>>> same
>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>> urge our users to download and start using the latest
>>>>>>>>> version of
>>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight
>in
>>>>>>> how
>>>>>>>>>> many
>>>>>>>>>>>>> times
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a
>cool
>>>>>>>>> metric
>>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> community to gauge the project popularity. You can
>think
>>>>>> of
>>>>>>>>> it
>>>>>>>>>> as
>>>>>>>>>>>> of a
>>>>>>>>>>>>>> page
>>>>>>>>>>>>>>>> visit counter shown on many websites. We can even
>decide
>>>>>> to
>>>>>>>>>>> display
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>>>>>>>>> release,
>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1
>byte.
>>>>>>>>> Every
>>>>>>>>>>> time
>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>>>>>>>>> background.
>>>>>>>>>>>> Then
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> will be able to view the number of the total downloads
>>>>>> for
>>>>>>>>> this
>>>>>>>>>>> JAR
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> Maven Central, which is essentially the number of
>starts
>>>>>> of
>>>>>>>>>> Ignite
>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
>>>>>> Ignite
>>>>>>> to
>>>>>>>>>> send
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>>>>>>>>> approach.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@gridgain.com>.

I can’t see it. Could you share a direct link to it?

If we can add one more script like that and it will work then the question
is closed.

Denis

On Monday, October 2, 2017, Dmitriy Setrakyan <ds...@apache.org> wrote:

> Denis, I can see the download.cgi script in SVN. Am I missing something?
>
> On Mon, Oct 2, 2017 at 5:31 PM, Denis Magda <dmagda@apache.org
> <javascript:;>> wrote:
>
> > Dmitriy,
> >
> > That’s the rule.  See replies in the ticket [1]
> >
> > *Background: the TLP server is already pretty darned busy just serving
> > *static* sites. Dynamic operation for near-200 PMCs would bury the
> machine.
> > Our policy of "static-only websites" has been in place since the
> Foundation
> > started*
> >
> > Download scripts seem to be the only exception and we as PMC don’t even
> > have access to them.
> >
> > If you want to keep pushing this direction let’s craft a message to Greg
> > and Daniel directly. I don’t know what else to ask for here rather than a
> > virtual machine that’s conceivably to much for a single script like that.
> >
> > [1] https://issues.apache.org/jira/browse/INFRA-15182 <
> > https://issues.apache.org/jira/browse/INFRA-15182>
> >
> > —
> > Denis
> >
> > > On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> <javascript:;>>
> > wrote:
> > >
> > > On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vozerov@gridgain.com
> <javascript:;>>
> > > wrote:
> > >
> > >> I am not sure it is good idea to send requests to 3rd-party addresses
> > from
> > >> Ignite node. Let's do not make the same mistakes again.
> > >>
> > >
> > > Agree with Vladimir.
> > >
> > > We obviously have CGI support on the website. Can someone explain why
> CGI
> > > is not possible to use?
> > >
> > >
> > >>
> > >> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <
> anovikov@gridgain.com <javascript:;>>
> > >> wrote:
> > >>
> > >>> We may directly send request to GA from Ignite node:
> > >>> https://developers.google.com/analytics/devguides/
> > >> collection/protocol/v1/
> > >>> <https://developers.google.com/analytics/devguides/
> > >> collection/protocol/v1/
> > >>>>
> > >>> Latest version can be received from maven central:
> > >>> https://repo1.maven.org/maven2/org/apache/ignite/
> > >>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
> > >>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
> > >>>
> > >>>
> > >>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org
> <javascript:;>>
> > >>> написал(а):
> > >>>>
> > >>>> Denis,
> > >>>>
> > >>>> I am not sure I understand. We already do have CGI enabled for
> > >>>> download.cgi. Is there something else we need?
> > >>>>
> > >>>> D.
> > >>>>
> > >>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com
> <javascript:;>>
> > >> wrote:
> > >>>>
> > >>>>> There is an obstacle. There is no way to execute the script using
> PHP
> > >> or
> > >>>>> similar sever side language and trigger GA as discussed earlier:
> > >>>>> https://issues.apache.org/jira/browse/INFRA-15182
> > >>>>>
> > >>>>> How else can we tackle this?
> > >>>>>
> > >>>>> Denis
> > >>>>>
> > >>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> > >>> dsetrakyan@apache.org <javascript:;>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> I think it is safe to assume at this point that everyone is in
> > >> general
> > >>>>>> agreement, since there are no active objections.
> > >>>>>>
> > >>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
> > >> happen:
> > >>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
> > >>>>>>
> > >>>>>> D.
> > >>>>>>
> > >>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> > >>>>> dsetrakyan@apache.org <javascript:;>
> > >>>>>> <javascript:;>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> > >> raul.asf@evosent.com <javascript:;>
> > >>>>>> <javascript:;>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Yeah, I guess that's doable as well and requires less management
> > >>>>> effort
> > >>>>>>>> than my suggestion. We could use events [1] to store payload
> data
> > >>>>> (e.g.
> > >>>>>>>> IP,
> > >>>>>>>> version, etc.)
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Yes, we could use events or some other similar API provided by
> GA.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> What the download page CGI developed in? PHP?
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> To be honest, no clue. I guess someone in the community can
> figure
> > >> it
> > >>>>>> out:
> > >>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> However, I'm not sure whether storing this data in a 3rd party
> > >>>>> (Google)
> > >>>>>> is
> > >>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
> > >> there's
> > >>>>>>>> doubt
> > >>>>>>>> in the PMC, it's better to ask legal@.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I am not sure there is anything to ask about. The whole Ignite
> > >> website
> > >>>>> is
> > >>>>>>> GA enabled, and all we are doing is accessing a standard web page
> > >> from
> > >>>>>> the
> > >>>>>>> Ignite web site. The information gathered from GA is available
> only
> > >> to
> > >>>>>> the
> > >>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
> this
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> Even ASF website itself uses GA: https://www.apache.org/
> > >>>>>>> foundation/policies/privacy.html
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Sure, would be nice to hear from Roman as well.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Cheers.
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>> https://developers.google.com/analytics/devguides/collection
> > >>>>>>>> /analyticsjs/events
> > >>>>>>>>
> > >>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >>>>>>>> dsetrakyan@apache.org <javascript:;> <javascript:;>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Raul,
> > >>>>>>>>>
> > >>>>>>>>> Could point about Javascript, it will not work because it
> > executes
> > >>>>> in
> > >>>>>>>> the
> > >>>>>>>>> browser. This means we need a server-side script, like CGI we
> are
> > >>>>>> using
> > >>>>>>>> on
> > >>>>>>>>> our download page.
> > >>>>>>>>>
> > >>>>>>>>> How about this approach. We create something like
> > >> ignite-version.cgi
> > >>>>>>>> script
> > >>>>>>>>> which will invoke a call to GA and then return the latest
> > version.
> > >>>>>>>>>
> > >>>>>>>>> This should work, right?
> > >>>>>>>>>
> > >>>>>>>>> D.
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> > >>>>> raul.asf@evosent.com <javascript:;>
> > >>>>>> <javascript:;>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hey Dmitriy and all
> > >>>>>>>>>>
> > >>>>>>>>>> Also, since we have GA enabled for the website, we can track
> how
> > >>>>>> many
> > >>>>>>>>> times
> > >>>>>>>>>>> this page was accessed, which will be equal to the number of
> > >>>>>> starts.
> > >>>>>>>>> This
> > >>>>>>>>>>> way, the counter information is tracked and monitored by the
> > >>>>>> Ignite
> > >>>>>>>>> PMC.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Unfortunately this won't work because GA is loaded via
> > Javascript
> > >>>>> on
> > >>>>>>>> the
> > >>>>>>>>>> browser, so Google will never receive the page hit.
> > >>>>>>>>>>
> > >>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
> > >>>>> endpoint
> > >>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
> > >>>>> request.
> > >>>>>>>> The
> > >>>>>>>>>> simplest thing is to write each request in a log file, along
> > with
> > >>>>>> the
> > >>>>>>>>>> timestamp, the version reported by the client, maybe the IP
> (not
> > >>>>>> sure
> > >>>>>>>>> about
> > >>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
> if
> > >>>>> you
> > >>>>>>>> make
> > >>>>>>>>> it
> > >>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
> > >>>>>> creates
> > >>>>>>>> on
> > >>>>>>>>>> first startup or something.
> > >>>>>>>>>>
> > >>>>>>>>>> This endpoint would need some basic DDoS protection and
> > >>>>> blacklisting
> > >>>>>>>> to
> > >>>>>>>>>> prevent data spoofing.
> > >>>>>>>>>>
> > >>>>>>>>>> If we'll be implementing this endpoint anyway, then there's no
> > >>>>> point
> > >>>>>>>>>> placing another file on Git or elsewhere for reporting the
> > latest
> > >>>>>>>>> versions:
> > >>>>>>>>>> the endpoint itself can return them.
> > >>>>>>>>>>
> > >>>>>>>>>> WDYT?
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers.
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > >>>>>>>>> dsetrakyan@apache.org <javascript:;> <javascript:;>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Cos, Raul,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the feedback. I completely agree about Maven
> Central
> > >>>>>>>> being a
> > >>>>>>>>>> 3rd
> > >>>>>>>>>>> party repo (did not think about that initially). All your
> > >>>>>>>> suggestions
> > >>>>>>>>>> make
> > >>>>>>>>>>> sense, but I would like to keep it as simple as possible, and
> > so
> > >>>>>> far
> > >>>>>>>>>>> everything suggested required GIT dependencies and extra
> work.
> > >>>>>>>>>>>
> > >>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
> > Ignite
> > >>>>>>>>> website
> > >>>>>>>>>>> which will have only the latest version. Every time a node
> > >>>>> starts,
> > >>>>>>>> it
> > >>>>>>>>>>> receives the latest version from the page and suggests that
> > >>>>> users
> > >>>>>>>>> upgrade
> > >>>>>>>>>>> if needed.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also, since we have GA enabled for the website, we can track
> > how
> > >>>>>>>> many
> > >>>>>>>>>> times
> > >>>>>>>>>>> this page was accessed, which will be equal to the number of
> > >>>>>> starts.
> > >>>>>>>>> This
> > >>>>>>>>>>> way, the counter information is tracked and monitored by the
> > >>>>>> Ignite
> > >>>>>>>>> PMC.
> > >>>>>>>>>>>
> > >>>>>>>>>>> This approach looks pretty innocent to me and everything is
> > kept
> > >>>>>> and
> > >>>>>>>>>>> managed within Apache.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thoughts?
> > >>>>>>>>>>>
> > >>>>>>>>>>> D.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > >>>>>>>> cos@apache.org <javascript:;> <javascript:;>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> I agree with Raul.
> > >>>>>>>>>>>> - providing a ping-back address to a 3rd party might be
> frown
> > >>>>>>>> upon by
> > >>>>>>>>>>> some.
> > >>>>>>>>>>>> And might have a consequences like stats collection about
> > >>>>>> users'
> > >>>>>>>>>>>> infrastructure.
> > >>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
> > >>>>> binary
> > >>>>>>>>> data:
> > >>>>>>>>>>>> everything is clear text and could be easily monitored by
> > >>>>> any
> > >>>>>> of
> > >>>>>>>>> the
> > >>>>>>>>>>>> network
> > >>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
> > >>>>> bit
> > >>>>>> of
> > >>>>>>>>> the
> > >>>>>>>>>>>> release
> > >>>>>>>>>>>> discipline.
> > >>>>>>>>>>>> - the binary data download in the runtime is my main
> concern.
> > >>>>>>>> This is
> > >>>>>>>>>> the
> > >>>>>>>>>>>> vector of MMA. What if someone gains the control over the
> > >>>>>>>>> repository
> > >>>>>>>>>>> and
> > >>>>>>>>>>>> replaces the file with some malicious content.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a
> > >>>>> call
> > >>>>>>>> to
> > >>>>>>>>> an
> > >>>>>>>>>>>> external script to check upon the atest software version
> > >>>>>> available
> > >>>>>>>>> for
> > >>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd
> party
> > >>>>>>>>> server,
> > >>>>>>>>>> I
> > >>>>>>>>>>>> believe the best way would be to put this script on ASF
> infra
> > >>>>>> and
> > >>>>>>>>> have
> > >>>>>>>>>>> the
> > >>>>>>>>>>>> "update checker" running in a completely controlled
> > >>>>> environment.
> > >>>>>>>>>>> Actually,
> > >>>>>>>>>>>> I
> > >>>>>>>>>>>> recall we had this very discussion during the Incubation; I
> > >>>>> can
> > >>>>>>>>>> probably
> > >>>>>>>>>>>> dig
> > >>>>>>>>>>>> out the corresponding thread.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thoughts?
> > >>>>>>>>>>>> Cok
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > >>>>>>>>>>>>> Hey guys
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
> > >>>>>>>>> (Sonatype).
> > >>>>>>>>>>> We
> > >>>>>>>>>>>>> don't know what kind of data analysis or intelligence
> > >>>>>> extraction
> > >>>>>>>>> they
> > >>>>>>>>>>>> run.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> If Ignite servers all over the world were hitting
> maven.org
> > >>>>>>>>>>> periodically
> > >>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
> > >>>>> clear
> > >>>>>>>>>>> indication
> > >>>>>>>>>>>>> of who is running an Ignite server.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> They could reverse lookup the IP address and find out what
> > >>>>>>>>>> corporation
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>> is.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We could use the Git ssh API, but that would require a new
> > >>>>>>>>>> dependency,
> > >>>>>>>>>>>>> which I advise against.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new
> Git
> > >>>>>>>> tags:
> > >>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Another option is to place a txt file in the root of the
> > >>>>>> master
> > >>>>>>>>>> branch
> > >>>>>>>>>>>> (e.g
> > >>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
> > >>>>> each
> > >>>>>>>> major
> > >>>>>>>>>>>> version
> > >>>>>>>>>>>>> line (1.x, 2.x).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I would advocate this last option, but it requires somebody
> > >>>>>>>>>> remembering
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>> update the file with every release, unless we automate it
> > >>>>>> with a
> > >>>>>>>>>> Maven
> > >>>>>>>>>>>>> plugin.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hope that helps!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> <javascript:;>
> > >>>>>> <javascript:;>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I see nothing wrong with this approach.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is
> > >>>>> it
> > >>>>>>>> good
> > >>>>>>>>>> to
> > >>>>>>>>>>>> go?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> —
> > >>>>>>>>>>>>> Denis
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > >>>>>>>>>>> dsetrakyan@apache.org <javascript:;> <javascript:;>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket
> > >>>>> on
> > >>>>>>>> it?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > >>>>>>>>>>>> dsetrakyan@apache.org <javascript:;> <javascript:;>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> There has been lots of talk of proposals about various
> > >>>>>> usage
> > >>>>>>>>>> metrics
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect
> > >>>>>> the
> > >>>>>>>>> topic
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>> propose something very simple and non-intrusive.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. Update Checker
> > >>>>>>>>>>>>>>> The main purpose of the update checker is not to collect
> > >>>>>>>>> metrics,
> > >>>>>>>>>>> but
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
> > >>>>>>>>> maven.org
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>> getting the version out of the metadata file:
> > >>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
> > >>>>>> org/apache/ignite/ignite-core/
> > >>>>>>>>>>>>>>> maven-metadata.xml
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This way we do not send any information anywhere and, at
> > >>>>>> the
> > >>>>>>>>> same
> > >>>>>>>>>>>> time,
> > >>>>>>>>>>>>>>> urge our users to download and start using the latest
> > >>>>>>>> version of
> > >>>>>>>>>>>> Ignite.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2. Startup Counter
> > >>>>>>>>>>>>>>> This piece is optional, but we can also get an insight in
> > >>>>>> how
> > >>>>>>>>> many
> > >>>>>>>>>>>> times
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>> certain Ignite release gets started. This is just a cool
> > >>>>>>>> metric
> > >>>>>>>>>> for
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> community to gauge the project popularity. You can think
> > >>>>> of
> > >>>>>>>> it
> > >>>>>>>>> as
> > >>>>>>>>>>> of a
> > >>>>>>>>>>>>> page
> > >>>>>>>>>>>>>>> visit counter shown on many websites. We can even decide
> > >>>>> to
> > >>>>>>>>>> display
> > >>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> counter on the Ignite website as well.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
> > >>>>>>>> release,
> > >>>>>>>>>> e.g.
> > >>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte.
> > >>>>>>>> Every
> > >>>>>>>>>> time
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
> > >>>>>>>> background.
> > >>>>>>>>>>> Then
> > >>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> will be able to view the number of the total downloads
> > >>>>> for
> > >>>>>>>> this
> > >>>>>>>>>> JAR
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>>> Maven Central, which is essentially the number of starts
> > >>>>> of
> > >>>>>>>>> Ignite
> > >>>>>>>>>>>> nodes.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Note that neither of the above suggestions require
> > >>>>> Ignite
> > >>>>>> to
> > >>>>>>>>> send
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>>> track any user information whatsoever.*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
> > >>>>>>>> approach.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> D.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

Denis, I can see the download.cgi script in SVN. Am I missing something?

On Mon, Oct 2, 2017 at 5:31 PM, Denis Magda <dm...@apache.org> wrote:

> Dmitriy,
>
> That’s the rule.  See replies in the ticket [1]
>
> *Background: the TLP server is already pretty darned busy just serving
> *static* sites. Dynamic operation for near-200 PMCs would bury the machine.
> Our policy of "static-only websites" has been in place since the Foundation
> started*
>
> Download scripts seem to be the only exception and we as PMC don’t even
> have access to them.
>
> If you want to keep pushing this direction let’s craft a message to Greg
> and Daniel directly. I don’t know what else to ask for here rather than a
> virtual machine that’s conceivably to much for a single script like that.
>
> [1] https://issues.apache.org/jira/browse/INFRA-15182 <
> https://issues.apache.org/jira/browse/INFRA-15182>
>
> —
> Denis
>
> > On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> >> I am not sure it is good idea to send requests to 3rd-party addresses
> from
> >> Ignite node. Let's do not make the same mistakes again.
> >>
> >
> > Agree with Vladimir.
> >
> > We obviously have CGI support on the website. Can someone explain why CGI
> > is not possible to use?
> >
> >
> >>
> >> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <an...@gridgain.com>
> >> wrote:
> >>
> >>> We may directly send request to GA from Ignite node:
> >>> https://developers.google.com/analytics/devguides/
> >> collection/protocol/v1/
> >>> <https://developers.google.com/analytics/devguides/
> >> collection/protocol/v1/
> >>>>
> >>> Latest version can be received from maven central:
> >>> https://repo1.maven.org/maven2/org/apache/ignite/
> >>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
> >>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
> >>>
> >>>
> >>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
> >>> написал(а):
> >>>>
> >>>> Denis,
> >>>>
> >>>> I am not sure I understand. We already do have CGI enabled for
> >>>> download.cgi. Is there something else we need?
> >>>>
> >>>> D.
> >>>>
> >>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
> >> wrote:
> >>>>
> >>>>> There is an obstacle. There is no way to execute the script using PHP
> >> or
> >>>>> similar sever side language and trigger GA as discussed earlier:
> >>>>> https://issues.apache.org/jira/browse/INFRA-15182
> >>>>>
> >>>>> How else can we tackle this?
> >>>>>
> >>>>> Denis
> >>>>>
> >>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> >>> dsetrakyan@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> I think it is safe to assume at this point that everyone is in
> >> general
> >>>>>> agreement, since there are no active objections.
> >>>>>>
> >>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
> >> happen:
> >>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >>>>> dsetrakyan@apache.org
> >>>>>> <javascript:;>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> >> raul.asf@evosent.com
> >>>>>> <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Yeah, I guess that's doable as well and requires less management
> >>>>> effort
> >>>>>>>> than my suggestion. We could use events [1] to store payload data
> >>>>> (e.g.
> >>>>>>>> IP,
> >>>>>>>> version, etc.)
> >>>>>>>
> >>>>>>>
> >>>>>>> Yes, we could use events or some other similar API provided by GA.
> >>>>>>>
> >>>>>>>
> >>>>>>>> What the download page CGI developed in? PHP?
> >>>>>>>>
> >>>>>>>
> >>>>>>> To be honest, no clue. I guess someone in the community can figure
> >> it
> >>>>>> out:
> >>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >>>>>>>
> >>>>>>>
> >>>>>>>> However, I'm not sure whether storing this data in a 3rd party
> >>>>> (Google)
> >>>>>> is
> >>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
> >> there's
> >>>>>>>> doubt
> >>>>>>>> in the PMC, it's better to ask legal@.
> >>>>>>>
> >>>>>>>
> >>>>>>> I am not sure there is anything to ask about. The whole Ignite
> >> website
> >>>>> is
> >>>>>>> GA enabled, and all we are doing is accessing a standard web page
> >> from
> >>>>>> the
> >>>>>>> Ignite web site. The information gathered from GA is available only
> >> to
> >>>>>> the
> >>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
> >>>>>>> question.
> >>>>>>>
> >>>>>>> Even ASF website itself uses GA: https://www.apache.org/
> >>>>>>> foundation/policies/privacy.html
> >>>>>>>
> >>>>>>>
> >>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Sure, would be nice to hear from Roman as well.
> >>>>>>>
> >>>>>>>
> >>>>>>>> Cheers.
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>> https://developers.google.com/analytics/devguides/collection
> >>>>>>>> /analyticsjs/events
> >>>>>>>>
> >>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Raul,
> >>>>>>>>>
> >>>>>>>>> Could point about Javascript, it will not work because it
> executes
> >>>>> in
> >>>>>>>> the
> >>>>>>>>> browser. This means we need a server-side script, like CGI we are
> >>>>>> using
> >>>>>>>> on
> >>>>>>>>> our download page.
> >>>>>>>>>
> >>>>>>>>> How about this approach. We create something like
> >> ignite-version.cgi
> >>>>>>>> script
> >>>>>>>>> which will invoke a call to GA and then return the latest
> version.
> >>>>>>>>>
> >>>>>>>>> This should work, right?
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> >>>>> raul.asf@evosent.com
> >>>>>> <javascript:;>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hey Dmitriy and all
> >>>>>>>>>>
> >>>>>>>>>> Also, since we have GA enabled for the website, we can track how
> >>>>>> many
> >>>>>>>>> times
> >>>>>>>>>>> this page was accessed, which will be equal to the number of
> >>>>>> starts.
> >>>>>>>>> This
> >>>>>>>>>>> way, the counter information is tracked and monitored by the
> >>>>>> Ignite
> >>>>>>>>> PMC.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately this won't work because GA is loaded via
> Javascript
> >>>>> on
> >>>>>>>> the
> >>>>>>>>>> browser, so Google will never receive the page hit.
> >>>>>>>>>>
> >>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
> >>>>> endpoint
> >>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
> >>>>> request.
> >>>>>>>> The
> >>>>>>>>>> simplest thing is to write each request in a log file, along
> with
> >>>>>> the
> >>>>>>>>>> timestamp, the version reported by the client, maybe the IP (not
> >>>>>> sure
> >>>>>>>>> about
> >>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK if
> >>>>> you
> >>>>>>>> make
> >>>>>>>>> it
> >>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
> >>>>>> creates
> >>>>>>>> on
> >>>>>>>>>> first startup or something.
> >>>>>>>>>>
> >>>>>>>>>> This endpoint would need some basic DDoS protection and
> >>>>> blacklisting
> >>>>>>>> to
> >>>>>>>>>> prevent data spoofing.
> >>>>>>>>>>
> >>>>>>>>>> If we'll be implementing this endpoint anyway, then there's no
> >>>>> point
> >>>>>>>>>> placing another file on Git or elsewhere for reporting the
> latest
> >>>>>>>>> versions:
> >>>>>>>>>> the endpoint itself can return them.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>>
> >>>>>>>>>> Cheers.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Cos, Raul,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the feedback. I completely agree about Maven Central
> >>>>>>>> being a
> >>>>>>>>>> 3rd
> >>>>>>>>>>> party repo (did not think about that initially). All your
> >>>>>>>> suggestions
> >>>>>>>>>> make
> >>>>>>>>>>> sense, but I would like to keep it as simple as possible, and
> so
> >>>>>> far
> >>>>>>>>>>> everything suggested required GIT dependencies and extra work.
> >>>>>>>>>>>
> >>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the
> Ignite
> >>>>>>>>> website
> >>>>>>>>>>> which will have only the latest version. Every time a node
> >>>>> starts,
> >>>>>>>> it
> >>>>>>>>>>> receives the latest version from the page and suggests that
> >>>>> users
> >>>>>>>>> upgrade
> >>>>>>>>>>> if needed.
> >>>>>>>>>>>
> >>>>>>>>>>> Also, since we have GA enabled for the website, we can track
> how
> >>>>>>>> many
> >>>>>>>>>> times
> >>>>>>>>>>> this page was accessed, which will be equal to the number of
> >>>>>> starts.
> >>>>>>>>> This
> >>>>>>>>>>> way, the counter information is tracked and monitored by the
> >>>>>> Ignite
> >>>>>>>>> PMC.
> >>>>>>>>>>>
> >>>>>>>>>>> This approach looks pretty innocent to me and everything is
> kept
> >>>>>> and
> >>>>>>>>>>> managed within Apache.
> >>>>>>>>>>>
> >>>>>>>>>>> Thoughts?
> >>>>>>>>>>>
> >>>>>>>>>>> D.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >>>>>>>> cos@apache.org <javascript:;>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I agree with Raul.
> >>>>>>>>>>>> - providing a ping-back address to a 3rd party might be frown
> >>>>>>>> upon by
> >>>>>>>>>>> some.
> >>>>>>>>>>>> And might have a consequences like stats collection about
> >>>>>> users'
> >>>>>>>>>>>> infrastructure.
> >>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
> >>>>> binary
> >>>>>>>>> data:
> >>>>>>>>>>>> everything is clear text and could be easily monitored by
> >>>>> any
> >>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> network
> >>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
> >>>>> bit
> >>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> release
> >>>>>>>>>>>> discipline.
> >>>>>>>>>>>> - the binary data download in the runtime is my main concern.
> >>>>>>>> This is
> >>>>>>>>>> the
> >>>>>>>>>>>> vector of MMA. What if someone gains the control over the
> >>>>>>>>> repository
> >>>>>>>>>>> and
> >>>>>>>>>>>> replaces the file with some malicious content.
> >>>>>>>>>>>>
> >>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a
> >>>>> call
> >>>>>>>> to
> >>>>>>>>> an
> >>>>>>>>>>>> external script to check upon the atest software version
> >>>>>> available
> >>>>>>>>> for
> >>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd party
> >>>>>>>>> server,
> >>>>>>>>>> I
> >>>>>>>>>>>> believe the best way would be to put this script on ASF infra
> >>>>>> and
> >>>>>>>>> have
> >>>>>>>>>>> the
> >>>>>>>>>>>> "update checker" running in a completely controlled
> >>>>> environment.
> >>>>>>>>>>> Actually,
> >>>>>>>>>>>> I
> >>>>>>>>>>>> recall we had this very discussion during the Incubation; I
> >>>>> can
> >>>>>>>>>> probably
> >>>>>>>>>>>> dig
> >>>>>>>>>>>> out the corresponding thread.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>> Cok
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >>>>>>>>>>>>> Hey guys
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
> >>>>>>>>> (Sonatype).
> >>>>>>>>>>> We
> >>>>>>>>>>>>> don't know what kind of data analysis or intelligence
> >>>>>> extraction
> >>>>>>>>> they
> >>>>>>>>>>>> run.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If Ignite servers all over the world were hitting maven.org
> >>>>>>>>>>> periodically
> >>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
> >>>>> clear
> >>>>>>>>>>> indication
> >>>>>>>>>>>>> of who is running an Ignite server.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> They could reverse lookup the IP address and find out what
> >>>>>>>>>> corporation
> >>>>>>>>>>> it
> >>>>>>>>>>>>> is.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We could use the Git ssh API, but that would require a new
> >>>>>>>>>> dependency,
> >>>>>>>>>>>>> which I advise against.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git
> >>>>>>>> tags:
> >>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Another option is to place a txt file in the root of the
> >>>>>> master
> >>>>>>>>>> branch
> >>>>>>>>>>>> (e.g
> >>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
> >>>>> each
> >>>>>>>> major
> >>>>>>>>>>>> version
> >>>>>>>>>>>>> line (1.x, 2.x).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would advocate this last option, but it requires somebody
> >>>>>>>>>> remembering
> >>>>>>>>>>>> to
> >>>>>>>>>>>>> update the file with every release, unless we automate it
> >>>>>> with a
> >>>>>>>>>> Maven
> >>>>>>>>>>>>> plugin.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hope that helps!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> >>>>>> <javascript:;>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I see nothing wrong with this approach.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is
> >>>>> it
> >>>>>>>> good
> >>>>>>>>>> to
> >>>>>>>>>>>> go?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> —
> >>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket
> >>>>> on
> >>>>>>>> it?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There has been lots of talk of proposals about various
> >>>>>> usage
> >>>>>>>>>> metrics
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect
> >>>>>> the
> >>>>>>>>> topic
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> propose something very simple and non-intrusive.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. Update Checker
> >>>>>>>>>>>>>>> The main purpose of the update checker is not to collect
> >>>>>>>>> metrics,
> >>>>>>>>>>> but
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
> >>>>>>>>> maven.org
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> getting the version out of the metadata file:
> >>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
> >>>>>> org/apache/ignite/ignite-core/
> >>>>>>>>>>>>>>> maven-metadata.xml
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This way we do not send any information anywhere and, at
> >>>>>> the
> >>>>>>>>> same
> >>>>>>>>>>>> time,
> >>>>>>>>>>>>>>> urge our users to download and start using the latest
> >>>>>>>> version of
> >>>>>>>>>>>> Ignite.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. Startup Counter
> >>>>>>>>>>>>>>> This piece is optional, but we can also get an insight in
> >>>>>> how
> >>>>>>>>> many
> >>>>>>>>>>>> times
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>> certain Ignite release gets started. This is just a cool
> >>>>>>>> metric
> >>>>>>>>>> for
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> community to gauge the project popularity. You can think
> >>>>> of
> >>>>>>>> it
> >>>>>>>>> as
> >>>>>>>>>>> of a
> >>>>>>>>>>>>> page
> >>>>>>>>>>>>>>> visit counter shown on many websites. We can even decide
> >>>>> to
> >>>>>>>>>> display
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>> counter on the Ignite website as well.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
> >>>>>>>> release,
> >>>>>>>>>> e.g.
> >>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte.
> >>>>>>>> Every
> >>>>>>>>>> time
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
> >>>>>>>> background.
> >>>>>>>>>>> Then
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>> will be able to view the number of the total downloads
> >>>>> for
> >>>>>>>> this
> >>>>>>>>>> JAR
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> Maven Central, which is essentially the number of starts
> >>>>> of
> >>>>>>>>> Ignite
> >>>>>>>>>>>> nodes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *Note that neither of the above suggestions require
> >>>>> Ignite
> >>>>>> to
> >>>>>>>>> send
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>> track any user information whatsoever.*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
> >>>>>>>> approach.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@apache.org>.

Dmitriy, 

That’s the rule.  See replies in the ticket [1]

*Background: the TLP server is already pretty darned busy just serving *static* sites. Dynamic operation for near-200 PMCs would bury the machine. Our policy of "static-only websites" has been in place since the Foundation started*

Download scripts seem to be the only exception and we as PMC don’t even have access to them.

If you want to keep pushing this direction let’s craft a message to Greg and Daniel directly. I don’t know what else to ask for here rather than a virtual machine that’s conceivably to much for a single script like that.

[1] https://issues.apache.org/jira/browse/INFRA-15182 <https://issues.apache.org/jira/browse/INFRA-15182>

—
Denis

> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <ds...@apache.org> wrote:
> 
> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
> 
>> I am not sure it is good idea to send requests to 3rd-party addresses from
>> Ignite node. Let's do not make the same mistakes again.
>> 
> 
> Agree with Vladimir.
> 
> We obviously have CGI support on the website. Can someone explain why CGI
> is not possible to use?
> 
> 
>> 
>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <an...@gridgain.com>
>> wrote:
>> 
>>> We may directly send request to GA from Ignite node:
>>> https://developers.google.com/analytics/devguides/
>> collection/protocol/v1/
>>> <https://developers.google.com/analytics/devguides/
>> collection/protocol/v1/
>>>> 
>>> Latest version can be received from maven central:
>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>> 
>>> 
>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
>>> написал(а):
>>>> 
>>>> Denis,
>>>> 
>>>> I am not sure I understand. We already do have CGI enabled for
>>>> download.cgi. Is there something else we need?
>>>> 
>>>> D.
>>>> 
>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
>> wrote:
>>>> 
>>>>> There is an obstacle. There is no way to execute the script using PHP
>> or
>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>> 
>>>>> How else can we tackle this?
>>>>> 
>>>>> Denis
>>>>> 
>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> I think it is safe to assume at this point that everyone is in
>> general
>>>>>> agreement, since there are no active objections.
>>>>>> 
>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>> happen:
>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>> 
>>>>>> D.
>>>>>> 
>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org
>>>>>> <javascript:;>>
>>>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>> raul.asf@evosent.com
>>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Yeah, I guess that's doable as well and requires less management
>>>>> effort
>>>>>>>> than my suggestion. We could use events [1] to store payload data
>>>>> (e.g.
>>>>>>>> IP,
>>>>>>>> version, etc.)
>>>>>>> 
>>>>>>> 
>>>>>>> Yes, we could use events or some other similar API provided by GA.
>>>>>>> 
>>>>>>> 
>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>> 
>>>>>>> 
>>>>>>> To be honest, no clue. I guess someone in the community can figure
>> it
>>>>>> out:
>>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>> 
>>>>>>> 
>>>>>>>> However, I'm not sure whether storing this data in a 3rd party
>>>>> (Google)
>>>>>> is
>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if
>> there's
>>>>>>>> doubt
>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>> 
>>>>>>> 
>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>> website
>>>>> is
>>>>>>> GA enabled, and all we are doing is accessing a standard web page
>> from
>>>>>> the
>>>>>>> Ignite web site. The information gathered from GA is available only
>> to
>>>>>> the
>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
>>>>>>> question.
>>>>>>> 
>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>> foundation/policies/privacy.html
>>>>>>> 
>>>>>>> 
>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>> 
>>>>>>> 
>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>> 
>>>>>>> 
>>>>>>>> Cheers.
>>>>>>>> 
>>>>>>>> [1]
>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>> /analyticsjs/events
>>>>>>>> 
>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Raul,
>>>>>>>>> 
>>>>>>>>> Could point about Javascript, it will not work because it executes
>>>>> in
>>>>>>>> the
>>>>>>>>> browser. This means we need a server-side script, like CGI we are
>>>>>> using
>>>>>>>> on
>>>>>>>>> our download page.
>>>>>>>>> 
>>>>>>>>> How about this approach. We create something like
>> ignite-version.cgi
>>>>>>>> script
>>>>>>>>> which will invoke a call to GA and then return the latest version.
>>>>>>>>> 
>>>>>>>>> This should work, right?
>>>>>>>>> 
>>>>>>>>> D.
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>>> raul.asf@evosent.com
>>>>>> <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>> 
>>>>>>>>>> Also, since we have GA enabled for the website, we can track how
>>>>>> many
>>>>>>>>> times
>>>>>>>>>>> this page was accessed, which will be equal to the number of
>>>>>> starts.
>>>>>>>>> This
>>>>>>>>>>> way, the counter information is tracked and monitored by the
>>>>>> Ignite
>>>>>>>>> PMC.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Unfortunately this won't work because GA is loaded via Javascript
>>>>> on
>>>>>>>> the
>>>>>>>>>> browser, so Google will never receive the page hit.
>>>>>>>>>> 
>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS
>>>>> endpoint
>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>>>>> request.
>>>>>>>> The
>>>>>>>>>> simplest thing is to write each request in a log file, along with
>>>>>> the
>>>>>>>>>> timestamp, the version reported by the client, maybe the IP (not
>>>>>> sure
>>>>>>>>> about
>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK if
>>>>> you
>>>>>>>> make
>>>>>>>>> it
>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
>>>>>> creates
>>>>>>>> on
>>>>>>>>>> first startup or something.
>>>>>>>>>> 
>>>>>>>>>> This endpoint would need some basic DDoS protection and
>>>>> blacklisting
>>>>>>>> to
>>>>>>>>>> prevent data spoofing.
>>>>>>>>>> 
>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's no
>>>>> point
>>>>>>>>>> placing another file on Git or elsewhere for reporting the latest
>>>>>>>>> versions:
>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>> 
>>>>>>>>>> WDYT?
>>>>>>>>>> 
>>>>>>>>>> Cheers.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven Central
>>>>>>>> being a
>>>>>>>>>> 3rd
>>>>>>>>>>> party repo (did not think about that initially). All your
>>>>>>>> suggestions
>>>>>>>>>> make
>>>>>>>>>>> sense, but I would like to keep it as simple as possible, and so
>>>>>> far
>>>>>>>>>>> everything suggested required GIT dependencies and extra work.
>>>>>>>>>>> 
>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the Ignite
>>>>>>>>> website
>>>>>>>>>>> which will have only the latest version. Every time a node
>>>>> starts,
>>>>>>>> it
>>>>>>>>>>> receives the latest version from the page and suggests that
>>>>> users
>>>>>>>>> upgrade
>>>>>>>>>>> if needed.
>>>>>>>>>>> 
>>>>>>>>>>> Also, since we have GA enabled for the website, we can track how
>>>>>>>> many
>>>>>>>>>> times
>>>>>>>>>>> this page was accessed, which will be equal to the number of
>>>>>> starts.
>>>>>>>>> This
>>>>>>>>>>> way, the counter information is tracked and monitored by the
>>>>>> Ignite
>>>>>>>>> PMC.
>>>>>>>>>>> 
>>>>>>>>>>> This approach looks pretty innocent to me and everything is kept
>>>>>> and
>>>>>>>>>>> managed within Apache.
>>>>>>>>>>> 
>>>>>>>>>>> Thoughts?
>>>>>>>>>>> 
>>>>>>>>>>> D.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be frown
>>>>>>>> upon by
>>>>>>>>>>> some.
>>>>>>>>>>>> And might have a consequences like stats collection about
>>>>>> users'
>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>>>>> binary
>>>>>>>>> data:
>>>>>>>>>>>> everything is clear text and could be easily monitored by
>>>>> any
>>>>>> of
>>>>>>>>> the
>>>>>>>>>>>> network
>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a
>>>>> bit
>>>>>> of
>>>>>>>>> the
>>>>>>>>>>>> release
>>>>>>>>>>>> discipline.
>>>>>>>>>>>> - the binary data download in the runtime is my main concern.
>>>>>>>> This is
>>>>>>>>>> the
>>>>>>>>>>>> vector of MMA. What if someone gains the control over the
>>>>>>>>> repository
>>>>>>>>>>> and
>>>>>>>>>>>> replaces the file with some malicious content.
>>>>>>>>>>>> 
>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a
>>>>> call
>>>>>>>> to
>>>>>>>>> an
>>>>>>>>>>>> external script to check upon the atest software version
>>>>>> available
>>>>>>>>> for
>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd party
>>>>>>>>> server,
>>>>>>>>>> I
>>>>>>>>>>>> believe the best way would be to put this script on ASF infra
>>>>>> and
>>>>>>>>> have
>>>>>>>>>>> the
>>>>>>>>>>>> "update checker" running in a completely controlled
>>>>> environment.
>>>>>>>>>>> Actually,
>>>>>>>>>>>> I
>>>>>>>>>>>> recall we had this very discussion during the Incubation; I
>>>>> can
>>>>>>>>>> probably
>>>>>>>>>>>> dig
>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>> Cok
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>>>>>>>>> (Sonatype).
>>>>>>>>>>> We
>>>>>>>>>>>>> don't know what kind of data analysis or intelligence
>>>>>> extraction
>>>>>>>>> they
>>>>>>>>>>>> run.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If Ignite servers all over the world were hitting maven.org
>>>>>>>>>>> periodically
>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>>>>> clear
>>>>>>>>>>> indication
>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> They could reverse lookup the IP address and find out what
>>>>>>>>>> corporation
>>>>>>>>>>> it
>>>>>>>>>>>>> is.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We could use the Git ssh API, but that would require a new
>>>>>>>>>> dependency,
>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git
>>>>>>>> tags:
>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Another option is to place a txt file in the root of the
>>>>>> master
>>>>>>>>>> branch
>>>>>>>>>>>> (e.g
>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>>>>> each
>>>>>>>> major
>>>>>>>>>>>> version
>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would advocate this last option, but it requires somebody
>>>>>>>>>> remembering
>>>>>>>>>>>> to
>>>>>>>>>>>>> update the file with every release, unless we automate it
>>>>>> with a
>>>>>>>>>> Maven
>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is
>>>>> it
>>>>>>>> good
>>>>>>>>>> to
>>>>>>>>>>>> go?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> —
>>>>>>>>>>>>> Denis
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket
>>>>> on
>>>>>>>> it?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> There has been lots of talk of proposals about various
>>>>>> usage
>>>>>>>>>> metrics
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect
>>>>>> the
>>>>>>>>> topic
>>>>>>>>>>> and
>>>>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>> The main purpose of the update checker is not to collect
>>>>>>>>> metrics,
>>>>>>>>>>> but
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>>>>>>>>> maven.org
>>>>>>>>>>> and
>>>>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This way we do not send any information anywhere and, at
>>>>>> the
>>>>>>>>> same
>>>>>>>>>>>> time,
>>>>>>>>>>>>>>> urge our users to download and start using the latest
>>>>>>>> version of
>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight in
>>>>>> how
>>>>>>>>> many
>>>>>>>>>>>> times
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a cool
>>>>>>>> metric
>>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> community to gauge the project popularity. You can think
>>>>> of
>>>>>>>> it
>>>>>>>>> as
>>>>>>>>>>> of a
>>>>>>>>>>>>> page
>>>>>>>>>>>>>>> visit counter shown on many websites. We can even decide
>>>>> to
>>>>>>>>>> display
>>>>>>>>>>>> this
>>>>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>>>>>>>> release,
>>>>>>>>>> e.g.
>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte.
>>>>>>>> Every
>>>>>>>>>> time
>>>>>>>>>>>> an
>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>>>>>>>> background.
>>>>>>>>>>> Then
>>>>>>>>>>>> we
>>>>>>>>>>>>>>> will be able to view the number of the total downloads
>>>>> for
>>>>>>>> this
>>>>>>>>>> JAR
>>>>>>>>>>> in
>>>>>>>>>>>>>>> Maven Central, which is essentially the number of starts
>>>>> of
>>>>>>>>> Ignite
>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Note that neither of the above suggestions require
>>>>> Ignite
>>>>>> to
>>>>>>>>> send
>>>>>>>>>>> or
>>>>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>>>>>>>> approach.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:

> I am not sure it is good idea to send requests to 3rd-party addresses from
> Ignite node. Let's do not make the same mistakes again.
>

Agree with Vladimir.

We obviously have CGI support on the website. Can someone explain why CGI
is not possible to use?


>
> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <an...@gridgain.com>
> wrote:
>
> > We may directly send request to GA from Ignite node:
> > https://developers.google.com/analytics/devguides/
> collection/protocol/v1/
> > <https://developers.google.com/analytics/devguides/
> collection/protocol/v1/
> > >
> > Latest version can be received from maven central:
> > https://repo1.maven.org/maven2/org/apache/ignite/
> > ignite-core/maven-metadata.xml <https://repo1.maven.org/
> > maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
> >
> >
> > > 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
> > написал(а):
> > >
> > > Denis,
> > >
> > > I am not sure I understand. We already do have CGI enabled for
> > > download.cgi. Is there something else we need?
> > >
> > > D.
> > >
> > > On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com>
> wrote:
> > >
> > >> There is an obstacle. There is no way to execute the script using PHP
> or
> > >> similar sever side language and trigger GA as discussed earlier:
> > >> https://issues.apache.org/jira/browse/INFRA-15182
> > >>
> > >> How else can we tackle this?
> > >>
> > >> Denis
> > >>
> > >> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > >> wrote:
> > >>
> > >>> I think it is safe to assume at this point that everyone is in
> general
> > >>> agreement, since there are no active objections.
> > >>>
> > >>> I have filed a ticket for the 2.3 release. Let's try to make it
> happen:
> > >>> https://issues.apache.org/jira/browse/IGNITE-6305
> > >>>
> > >>> D.
> > >>>
> > >>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> > >> dsetrakyan@apache.org
> > >>> <javascript:;>>
> > >>> wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> raul.asf@evosent.com
> > >>> <javascript:;>>
> > >>>> wrote:
> > >>>>
> > >>>>> Yeah, I guess that's doable as well and requires less management
> > >> effort
> > >>>>> than my suggestion. We could use events [1] to store payload data
> > >> (e.g.
> > >>>>> IP,
> > >>>>> version, etc.)
> > >>>>
> > >>>>
> > >>>> Yes, we could use events or some other similar API provided by GA.
> > >>>>
> > >>>>
> > >>>>> What the download page CGI developed in? PHP?
> > >>>>>
> > >>>>
> > >>>> To be honest, no clue. I guess someone in the community can figure
> it
> > >>> out:
> > >>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >>>>
> > >>>>
> > >>>>> However, I'm not sure whether storing this data in a 3rd party
> > >> (Google)
> > >>> is
> > >>>>> compliant with the ASF policy. I guess it's no biggie, but if
> there's
> > >>>>> doubt
> > >>>>> in the PMC, it's better to ask legal@.
> > >>>>
> > >>>>
> > >>>> I am not sure there is anything to ask about. The whole Ignite
> website
> > >> is
> > >>>> GA enabled, and all we are doing is accessing a standard web page
> from
> > >>> the
> > >>>> Ignite web site. The information gathered from GA is available only
> to
> > >>> the
> > >>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
> > >>>> question.
> > >>>>
> > >>>> Even ASF website itself uses GA: https://www.apache.org/
> > >>>> foundation/policies/privacy.html
> > >>>>
> > >>>>
> > >>>>> I think Cos said it's OK; maybe Roman can pitch in.
> > >>>>>
> > >>>>
> > >>>> Sure, would be nice to hear from Roman as well.
> > >>>>
> > >>>>
> > >>>>> Cheers.
> > >>>>>
> > >>>>> [1]
> > >>>>> https://developers.google.com/analytics/devguides/collection
> > >>>>> /analyticsjs/events
> > >>>>>
> > >>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >>>>> dsetrakyan@apache.org <javascript:;>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Raul,
> > >>>>>>
> > >>>>>> Could point about Javascript, it will not work because it executes
> > >> in
> > >>>>> the
> > >>>>>> browser. This means we need a server-side script, like CGI we are
> > >>> using
> > >>>>> on
> > >>>>>> our download page.
> > >>>>>>
> > >>>>>> How about this approach. We create something like
> ignite-version.cgi
> > >>>>> script
> > >>>>>> which will invoke a call to GA and then return the latest version.
> > >>>>>>
> > >>>>>> This should work, right?
> > >>>>>>
> > >>>>>> D.
> > >>>>>>
> > >>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> > >> raul.asf@evosent.com
> > >>> <javascript:;>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hey Dmitriy and all
> > >>>>>>>
> > >>>>>>> Also, since we have GA enabled for the website, we can track how
> > >>> many
> > >>>>>> times
> > >>>>>>>> this page was accessed, which will be equal to the number of
> > >>> starts.
> > >>>>>> This
> > >>>>>>>> way, the counter information is tracked and monitored by the
> > >>> Ignite
> > >>>>>> PMC.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Unfortunately this won't work because GA is loaded via Javascript
> > >> on
> > >>>>> the
> > >>>>>>> browser, so Google will never receive the page hit.
> > >>>>>>>
> > >>>>>>> Given the constraints, the most viable solution is an HTTPS
> > >> endpoint
> > >>>>>>> running on ASF infra that Ignite invokes via a GET or POST
> > >> request.
> > >>>>> The
> > >>>>>>> simplest thing is to write each request in a log file, along with
> > >>> the
> > >>>>>>> timestamp, the version reported by the client, maybe the IP (not
> > >>> sure
> > >>>>>> about
> > >>>>>>> the ASF rules about this concerning privacy, I guess it's OK if
> > >> you
> > >>>>> make
> > >>>>>> it
> > >>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
> > >>> creates
> > >>>>> on
> > >>>>>>> first startup or something.
> > >>>>>>>
> > >>>>>>> This endpoint would need some basic DDoS protection and
> > >> blacklisting
> > >>>>> to
> > >>>>>>> prevent data spoofing.
> > >>>>>>>
> > >>>>>>> If we'll be implementing this endpoint anyway, then there's no
> > >> point
> > >>>>>>> placing another file on Git or elsewhere for reporting the latest
> > >>>>>> versions:
> > >>>>>>> the endpoint itself can return them.
> > >>>>>>>
> > >>>>>>> WDYT?
> > >>>>>>>
> > >>>>>>> Cheers.
> > >>>>>>>
> > >>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > >>>>>> dsetrakyan@apache.org <javascript:;>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Cos, Raul,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback. I completely agree about Maven Central
> > >>>>> being a
> > >>>>>>> 3rd
> > >>>>>>>> party repo (did not think about that initially). All your
> > >>>>> suggestions
> > >>>>>>> make
> > >>>>>>>> sense, but I would like to keep it as simple as possible, and so
> > >>> far
> > >>>>>>>> everything suggested required GIT dependencies and extra work.
> > >>>>>>>>
> > >>>>>>>> How about Yakov's suggestion. We simply add a page to the Ignite
> > >>>>>> website
> > >>>>>>>> which will have only the latest version. Every time a node
> > >> starts,
> > >>>>> it
> > >>>>>>>> receives the latest version from the page and suggests that
> > >> users
> > >>>>>> upgrade
> > >>>>>>>> if needed.
> > >>>>>>>>
> > >>>>>>>> Also, since we have GA enabled for the website, we can track how
> > >>>>> many
> > >>>>>>> times
> > >>>>>>>> this page was accessed, which will be equal to the number of
> > >>> starts.
> > >>>>>> This
> > >>>>>>>> way, the counter information is tracked and monitored by the
> > >>> Ignite
> > >>>>>> PMC.
> > >>>>>>>>
> > >>>>>>>> This approach looks pretty innocent to me and everything is kept
> > >>> and
> > >>>>>>>> managed within Apache.
> > >>>>>>>>
> > >>>>>>>> Thoughts?
> > >>>>>>>>
> > >>>>>>>> D.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > >>>>> cos@apache.org <javascript:;>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I agree with Raul.
> > >>>>>>>>> - providing a ping-back address to a 3rd party might be frown
> > >>>>> upon by
> > >>>>>>>> some.
> > >>>>>>>>>  And might have a consequences like stats collection about
> > >>> users'
> > >>>>>>>>>  infrastructure.
> > >>>>>>>>> - checking an ASF git-repo is easy and won't download any
> > >> binary
> > >>>>>> data:
> > >>>>>>>>>  everything is clear text and could be easily monitored by
> > >> any
> > >>> of
> > >>>>>> the
> > >>>>>>>>> network
> > >>>>>>>>>  diagnostic tools, shall it be required. But it involves a
> > >> bit
> > >>> of
> > >>>>>> the
> > >>>>>>>>> release
> > >>>>>>>>>  discipline.
> > >>>>>>>>> - the binary data download in the runtime is my main concern.
> > >>>>> This is
> > >>>>>>> the
> > >>>>>>>>>  vector of MMA. What if someone gains the control over the
> > >>>>>> repository
> > >>>>>>>> and
> > >>>>>>>>>  replaces the file with some malicious content.
> > >>>>>>>>>
> > >>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a
> > >> call
> > >>>>> to
> > >>>>>> an
> > >>>>>>>>> external script to check upon the atest software version
> > >>> available
> > >>>>>> for
> > >>>>>>>>> download. In the past, the endpoint was running on a 3rd party
> > >>>>>> server,
> > >>>>>>> I
> > >>>>>>>>> believe the best way would be to put this script on ASF infra
> > >>> and
> > >>>>>> have
> > >>>>>>>> the
> > >>>>>>>>> "update checker" running in a completely controlled
> > >> environment.
> > >>>>>>>> Actually,
> > >>>>>>>>> I
> > >>>>>>>>> recall we had this very discussion during the Incubation; I
> > >> can
> > >>>>>>> probably
> > >>>>>>>>> dig
> > >>>>>>>>> out the corresponding thread.
> > >>>>>>>>>
> > >>>>>>>>> Thoughts?
> > >>>>>>>>>  Cok
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > >>>>>>>>>> Hey guys
> > >>>>>>>>>>
> > >>>>>>>>>> In my opinion, maven.org is still owned by a third party
> > >>>>>> (Sonatype).
> > >>>>>>>> We
> > >>>>>>>>>> don't know what kind of data analysis or intelligence
> > >>> extraction
> > >>>>>> they
> > >>>>>>>>> run.
> > >>>>>>>>>>
> > >>>>>>>>>> If Ignite servers all over the world were hitting maven.org
> > >>>>>>>> periodically
> > >>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
> > >> clear
> > >>>>>>>> indication
> > >>>>>>>>>> of who is running an Ignite server.
> > >>>>>>>>>>
> > >>>>>>>>>> They could reverse lookup the IP address and find out what
> > >>>>>>> corporation
> > >>>>>>>> it
> > >>>>>>>>>> is.
> > >>>>>>>>>>
> > >>>>>>>>>> How about having Ignite check the ASF Git directly?
> > >>>>>>>>>>
> > >>>>>>>>>> We could use the Git ssh API, but that would require a new
> > >>>>>>> dependency,
> > >>>>>>>>>> which I advise against.
> > >>>>>>>>>>
> > >>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git
> > >>>>> tags:
> > >>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >>>>>>>>>>
> > >>>>>>>>>> Another option is to place a txt file in the root of the
> > >>> master
> > >>>>>>> branch
> > >>>>>>>>> (e.g
> > >>>>>>>>>> LATEST), containing a list of the latest GA versions for
> > >> each
> > >>>>> major
> > >>>>>>>>> version
> > >>>>>>>>>> line (1.x, 2.x).
> > >>>>>>>>>>
> > >>>>>>>>>> I would advocate this last option, but it requires somebody
> > >>>>>>> remembering
> > >>>>>>>>> to
> > >>>>>>>>>> update the file with every release, unless we automate it
> > >>> with a
> > >>>>>>> Maven
> > >>>>>>>>>> plugin.
> > >>>>>>>>>>
> > >>>>>>>>>> Hope that helps!
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> > >>> <javascript:;>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> I see nothing wrong with this approach.
> > >>>>>>>>>>
> > >>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is
> > >> it
> > >>>>> good
> > >>>>>>> to
> > >>>>>>>>> go?
> > >>>>>>>>>>
> > >>>>>>>>>> —
> > >>>>>>>>>> Denis
> > >>>>>>>>>>
> > >>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > >>>>>>>> dsetrakyan@apache.org <javascript:;>
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket
> > >> on
> > >>>>> it?
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > >>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> There has been lots of talk of proposals about various
> > >>> usage
> > >>>>>>> metrics
> > >>>>>>>>> for
> > >>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect
> > >>> the
> > >>>>>> topic
> > >>>>>>>> and
> > >>>>>>>>>>>> propose something very simple and non-intrusive.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Update Checker
> > >>>>>>>>>>>> The main purpose of the update checker is not to collect
> > >>>>>> metrics,
> > >>>>>>>> but
> > >>>>>>>>> to
> > >>>>>>>>>>>> notify users about a new version of Ignite by accessing
> > >>>>>> maven.org
> > >>>>>>>> and
> > >>>>>>>>>>>> getting the version out of the metadata file:
> > >>>>>>>>>>>> http://repo2.maven.org/maven2/
> > >>> org/apache/ignite/ignite-core/
> > >>>>>>>>>>>> maven-metadata.xml
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This way we do not send any information anywhere and, at
> > >>> the
> > >>>>>> same
> > >>>>>>>>> time,
> > >>>>>>>>>>>> urge our users to download and start using the latest
> > >>>>> version of
> > >>>>>>>>> Ignite.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. Startup Counter
> > >>>>>>>>>>>> This piece is optional, but we can also get an insight in
> > >>> how
> > >>>>>> many
> > >>>>>>>>> times
> > >>>>>>>>>> a
> > >>>>>>>>>>>> certain Ignite release gets started. This is just a cool
> > >>>>> metric
> > >>>>>>> for
> > >>>>>>>>> the
> > >>>>>>>>>>>> community to gauge the project popularity. You can think
> > >> of
> > >>>>> it
> > >>>>>> as
> > >>>>>>>> of a
> > >>>>>>>>>> page
> > >>>>>>>>>>>> visit counter shown on many websites. We can even decide
> > >> to
> > >>>>>>> display
> > >>>>>>>>> this
> > >>>>>>>>>>>> counter on the Ignite website as well.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
> > >>>>> release,
> > >>>>>>> e.g.
> > >>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte.
> > >>>>> Every
> > >>>>>>> time
> > >>>>>>>>> an
> > >>>>>>>>>>>> Ignite node starts, it will download this JAR in the
> > >>>>> background.
> > >>>>>>>> Then
> > >>>>>>>>> we
> > >>>>>>>>>>>> will be able to view the number of the total downloads
> > >> for
> > >>>>> this
> > >>>>>>> JAR
> > >>>>>>>> in
> > >>>>>>>>>>>> Maven Central, which is essentially the number of starts
> > >> of
> > >>>>>> Ignite
> > >>>>>>>>> nodes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> *Note that neither of the above suggestions require
> > >> Ignite
> > >>> to
> > >>>>>> send
> > >>>>>>>> or
> > >>>>>>>>>>>> track any user information whatsoever.*
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Please reply suggesting weather you are OK with this
> > >>>>> approach.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> D.
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Vladimir Ozerov <vo...@gridgain.com>.

I am not sure it is good idea to send requests to 3rd-party addresses from
Ignite node. Let's do not make the same mistakes again.

On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <an...@gridgain.com>
wrote:

> We may directly send request to GA from Ignite node:
> https://developers.google.com/analytics/devguides/collection/protocol/v1/
> <https://developers.google.com/analytics/devguides/collection/protocol/v1/
> >
> Latest version can be received from maven central:
> https://repo1.maven.org/maven2/org/apache/ignite/
> ignite-core/maven-metadata.xml <https://repo1.maven.org/
> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>
>
> > 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org>
> написал(а):
> >
> > Denis,
> >
> > I am not sure I understand. We already do have CGI enabled for
> > download.cgi. Is there something else we need?
> >
> > D.
> >
> > On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com> wrote:
> >
> >> There is an obstacle. There is no way to execute the script using PHP or
> >> similar sever side language and trigger GA as discussed earlier:
> >> https://issues.apache.org/jira/browse/INFRA-15182
> >>
> >> How else can we tackle this?
> >>
> >> Denis
> >>
> >> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> >> wrote:
> >>
> >>> I think it is safe to assume at this point that everyone is in general
> >>> agreement, since there are no active objections.
> >>>
> >>> I have filed a ticket for the 2.3 release. Let's try to make it happen:
> >>> https://issues.apache.org/jira/browse/IGNITE-6305
> >>>
> >>> D.
> >>>
> >>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >> dsetrakyan@apache.org
> >>> <javascript:;>>
> >>> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
> >>> <javascript:;>>
> >>>> wrote:
> >>>>
> >>>>> Yeah, I guess that's doable as well and requires less management
> >> effort
> >>>>> than my suggestion. We could use events [1] to store payload data
> >> (e.g.
> >>>>> IP,
> >>>>> version, etc.)
> >>>>
> >>>>
> >>>> Yes, we could use events or some other similar API provided by GA.
> >>>>
> >>>>
> >>>>> What the download page CGI developed in? PHP?
> >>>>>
> >>>>
> >>>> To be honest, no clue. I guess someone in the community can figure it
> >>> out:
> >>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >>>>
> >>>>
> >>>>> However, I'm not sure whether storing this data in a 3rd party
> >> (Google)
> >>> is
> >>>>> compliant with the ASF policy. I guess it's no biggie, but if there's
> >>>>> doubt
> >>>>> in the PMC, it's better to ask legal@.
> >>>>
> >>>>
> >>>> I am not sure there is anything to ask about. The whole Ignite website
> >> is
> >>>> GA enabled, and all we are doing is accessing a standard web page from
> >>> the
> >>>> Ignite web site. The information gathered from GA is available only to
> >>> the
> >>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
> >>>> question.
> >>>>
> >>>> Even ASF website itself uses GA: https://www.apache.org/
> >>>> foundation/policies/privacy.html
> >>>>
> >>>>
> >>>>> I think Cos said it's OK; maybe Roman can pitch in.
> >>>>>
> >>>>
> >>>> Sure, would be nice to hear from Roman as well.
> >>>>
> >>>>
> >>>>> Cheers.
> >>>>>
> >>>>> [1]
> >>>>> https://developers.google.com/analytics/devguides/collection
> >>>>> /analyticsjs/events
> >>>>>
> >>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>> wrote:
> >>>>>
> >>>>>> Raul,
> >>>>>>
> >>>>>> Could point about Javascript, it will not work because it executes
> >> in
> >>>>> the
> >>>>>> browser. This means we need a server-side script, like CGI we are
> >>> using
> >>>>> on
> >>>>>> our download page.
> >>>>>>
> >>>>>> How about this approach. We create something like ignite-version.cgi
> >>>>> script
> >>>>>> which will invoke a call to GA and then return the latest version.
> >>>>>>
> >>>>>> This should work, right?
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> >> raul.asf@evosent.com
> >>> <javascript:;>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey Dmitriy and all
> >>>>>>>
> >>>>>>> Also, since we have GA enabled for the website, we can track how
> >>> many
> >>>>>> times
> >>>>>>>> this page was accessed, which will be equal to the number of
> >>> starts.
> >>>>>> This
> >>>>>>>> way, the counter information is tracked and monitored by the
> >>> Ignite
> >>>>>> PMC.
> >>>>>>>
> >>>>>>>
> >>>>>>> Unfortunately this won't work because GA is loaded via Javascript
> >> on
> >>>>> the
> >>>>>>> browser, so Google will never receive the page hit.
> >>>>>>>
> >>>>>>> Given the constraints, the most viable solution is an HTTPS
> >> endpoint
> >>>>>>> running on ASF infra that Ignite invokes via a GET or POST
> >> request.
> >>>>> The
> >>>>>>> simplest thing is to write each request in a log file, along with
> >>> the
> >>>>>>> timestamp, the version reported by the client, maybe the IP (not
> >>> sure
> >>>>>> about
> >>>>>>> the ASF rules about this concerning privacy, I guess it's OK if
> >> you
> >>>>> make
> >>>>>> it
> >>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
> >>> creates
> >>>>> on
> >>>>>>> first startup or something.
> >>>>>>>
> >>>>>>> This endpoint would need some basic DDoS protection and
> >> blacklisting
> >>>>> to
> >>>>>>> prevent data spoofing.
> >>>>>>>
> >>>>>>> If we'll be implementing this endpoint anyway, then there's no
> >> point
> >>>>>>> placing another file on Git or elsewhere for reporting the latest
> >>>>>> versions:
> >>>>>>> the endpoint itself can return them.
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>> Cheers.
> >>>>>>>
> >>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Cos, Raul,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback. I completely agree about Maven Central
> >>>>> being a
> >>>>>>> 3rd
> >>>>>>>> party repo (did not think about that initially). All your
> >>>>> suggestions
> >>>>>>> make
> >>>>>>>> sense, but I would like to keep it as simple as possible, and so
> >>> far
> >>>>>>>> everything suggested required GIT dependencies and extra work.
> >>>>>>>>
> >>>>>>>> How about Yakov's suggestion. We simply add a page to the Ignite
> >>>>>> website
> >>>>>>>> which will have only the latest version. Every time a node
> >> starts,
> >>>>> it
> >>>>>>>> receives the latest version from the page and suggests that
> >> users
> >>>>>> upgrade
> >>>>>>>> if needed.
> >>>>>>>>
> >>>>>>>> Also, since we have GA enabled for the website, we can track how
> >>>>> many
> >>>>>>> times
> >>>>>>>> this page was accessed, which will be equal to the number of
> >>> starts.
> >>>>>> This
> >>>>>>>> way, the counter information is tracked and monitored by the
> >>> Ignite
> >>>>>> PMC.
> >>>>>>>>
> >>>>>>>> This approach looks pretty innocent to me and everything is kept
> >>> and
> >>>>>>>> managed within Apache.
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>>
> >>>>>>>> D.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >>>>> cos@apache.org <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I agree with Raul.
> >>>>>>>>> - providing a ping-back address to a 3rd party might be frown
> >>>>> upon by
> >>>>>>>> some.
> >>>>>>>>>  And might have a consequences like stats collection about
> >>> users'
> >>>>>>>>>  infrastructure.
> >>>>>>>>> - checking an ASF git-repo is easy and won't download any
> >> binary
> >>>>>> data:
> >>>>>>>>>  everything is clear text and could be easily monitored by
> >> any
> >>> of
> >>>>>> the
> >>>>>>>>> network
> >>>>>>>>>  diagnostic tools, shall it be required. But it involves a
> >> bit
> >>> of
> >>>>>> the
> >>>>>>>>> release
> >>>>>>>>>  discipline.
> >>>>>>>>> - the binary data download in the runtime is my main concern.
> >>>>> This is
> >>>>>>> the
> >>>>>>>>>  vector of MMA. What if someone gains the control over the
> >>>>>> repository
> >>>>>>>> and
> >>>>>>>>>  replaces the file with some malicious content.
> >>>>>>>>>
> >>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a
> >> call
> >>>>> to
> >>>>>> an
> >>>>>>>>> external script to check upon the atest software version
> >>> available
> >>>>>> for
> >>>>>>>>> download. In the past, the endpoint was running on a 3rd party
> >>>>>> server,
> >>>>>>> I
> >>>>>>>>> believe the best way would be to put this script on ASF infra
> >>> and
> >>>>>> have
> >>>>>>>> the
> >>>>>>>>> "update checker" running in a completely controlled
> >> environment.
> >>>>>>>> Actually,
> >>>>>>>>> I
> >>>>>>>>> recall we had this very discussion during the Incubation; I
> >> can
> >>>>>>> probably
> >>>>>>>>> dig
> >>>>>>>>> out the corresponding thread.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>  Cok
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >>>>>>>>>> Hey guys
> >>>>>>>>>>
> >>>>>>>>>> In my opinion, maven.org is still owned by a third party
> >>>>>> (Sonatype).
> >>>>>>>> We
> >>>>>>>>>> don't know what kind of data analysis or intelligence
> >>> extraction
> >>>>>> they
> >>>>>>>>> run.
> >>>>>>>>>>
> >>>>>>>>>> If Ignite servers all over the world were hitting maven.org
> >>>>>>>> periodically
> >>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
> >> clear
> >>>>>>>> indication
> >>>>>>>>>> of who is running an Ignite server.
> >>>>>>>>>>
> >>>>>>>>>> They could reverse lookup the IP address and find out what
> >>>>>>> corporation
> >>>>>>>> it
> >>>>>>>>>> is.
> >>>>>>>>>>
> >>>>>>>>>> How about having Ignite check the ASF Git directly?
> >>>>>>>>>>
> >>>>>>>>>> We could use the Git ssh API, but that would require a new
> >>>>>>> dependency,
> >>>>>>>>>> which I advise against.
> >>>>>>>>>>
> >>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git
> >>>>> tags:
> >>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >>>>>>>>>>
> >>>>>>>>>> Another option is to place a txt file in the root of the
> >>> master
> >>>>>>> branch
> >>>>>>>>> (e.g
> >>>>>>>>>> LATEST), containing a list of the latest GA versions for
> >> each
> >>>>> major
> >>>>>>>>> version
> >>>>>>>>>> line (1.x, 2.x).
> >>>>>>>>>>
> >>>>>>>>>> I would advocate this last option, but it requires somebody
> >>>>>>> remembering
> >>>>>>>>> to
> >>>>>>>>>> update the file with every release, unless we automate it
> >>> with a
> >>>>>>> Maven
> >>>>>>>>>> plugin.
> >>>>>>>>>>
> >>>>>>>>>> Hope that helps!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> >>> <javascript:;>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I see nothing wrong with this approach.
> >>>>>>>>>>
> >>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is
> >> it
> >>>>> good
> >>>>>>> to
> >>>>>>>>> go?
> >>>>>>>>>>
> >>>>>>>>>> —
> >>>>>>>>>> Denis
> >>>>>>>>>>
> >>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >>>>>>>> dsetrakyan@apache.org <javascript:;>
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket
> >> on
> >>>>> it?
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>
> >>>>>>>>>>>> There has been lots of talk of proposals about various
> >>> usage
> >>>>>>> metrics
> >>>>>>>>> for
> >>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect
> >>> the
> >>>>>> topic
> >>>>>>>> and
> >>>>>>>>>>>> propose something very simple and non-intrusive.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Update Checker
> >>>>>>>>>>>> The main purpose of the update checker is not to collect
> >>>>>> metrics,
> >>>>>>>> but
> >>>>>>>>> to
> >>>>>>>>>>>> notify users about a new version of Ignite by accessing
> >>>>>> maven.org
> >>>>>>>> and
> >>>>>>>>>>>> getting the version out of the metadata file:
> >>>>>>>>>>>> http://repo2.maven.org/maven2/
> >>> org/apache/ignite/ignite-core/
> >>>>>>>>>>>> maven-metadata.xml
> >>>>>>>>>>>>
> >>>>>>>>>>>> This way we do not send any information anywhere and, at
> >>> the
> >>>>>> same
> >>>>>>>>> time,
> >>>>>>>>>>>> urge our users to download and start using the latest
> >>>>> version of
> >>>>>>>>> Ignite.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Startup Counter
> >>>>>>>>>>>> This piece is optional, but we can also get an insight in
> >>> how
> >>>>>> many
> >>>>>>>>> times
> >>>>>>>>>> a
> >>>>>>>>>>>> certain Ignite release gets started. This is just a cool
> >>>>> metric
> >>>>>>> for
> >>>>>>>>> the
> >>>>>>>>>>>> community to gauge the project popularity. You can think
> >> of
> >>>>> it
> >>>>>> as
> >>>>>>>> of a
> >>>>>>>>>> page
> >>>>>>>>>>>> visit counter shown on many websites. We can even decide
> >> to
> >>>>>>> display
> >>>>>>>>> this
> >>>>>>>>>>>> counter on the Ignite website as well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
> >>>>> release,
> >>>>>>> e.g.
> >>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte.
> >>>>> Every
> >>>>>>> time
> >>>>>>>>> an
> >>>>>>>>>>>> Ignite node starts, it will download this JAR in the
> >>>>> background.
> >>>>>>>> Then
> >>>>>>>>> we
> >>>>>>>>>>>> will be able to view the number of the total downloads
> >> for
> >>>>> this
> >>>>>>> JAR
> >>>>>>>> in
> >>>>>>>>>>>> Maven Central, which is essentially the number of starts
> >> of
> >>>>>> Ignite
> >>>>>>>>> nodes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Note that neither of the above suggestions require
> >> Ignite
> >>> to
> >>>>>> send
> >>>>>>>> or
> >>>>>>>>>>>> track any user information whatsoever.*
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please reply suggesting weather you are OK with this
> >>>>> approach.
> >>>>>>>>>>>>
> >>>>>>>>>>>> D.
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Andrey Novikov <an...@gridgain.com>.

We may directly send request to GA from Ignite node: https://developers.google.com/analytics/devguides/collection/protocol/v1/ <https://developers.google.com/analytics/devguides/collection/protocol/v1/>
Latest version can be received from maven central: https://repo1.maven.org/maven2/org/apache/ignite/ignite-core/maven-metadata.xml <https://repo1.maven.org/maven2/org/apache/ignite/ignite-core/maven-metadata.xml>


> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <ds...@apache.org> написал(а):
> 
> Denis,
> 
> I am not sure I understand. We already do have CGI enabled for
> download.cgi. Is there something else we need?
> 
> D.
> 
> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com> wrote:
> 
>> There is an obstacle. There is no way to execute the script using PHP or
>> similar sever side language and trigger GA as discussed earlier:
>> https://issues.apache.org/jira/browse/INFRA-15182
>> 
>> How else can we tackle this?
>> 
>> Denis
>> 
>> On Thursday, September 7, 2017, Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>> 
>>> I think it is safe to assume at this point that everyone is in general
>>> agreement, since there are no active objections.
>>> 
>>> I have filed a ticket for the 2.3 release. Let's try to make it happen:
>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>> 
>>> D.
>>> 
>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org
>>> <javascript:;>>
>>> wrote:
>>> 
>>>> 
>>>> 
>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
>>> <javascript:;>>
>>>> wrote:
>>>> 
>>>>> Yeah, I guess that's doable as well and requires less management
>> effort
>>>>> than my suggestion. We could use events [1] to store payload data
>> (e.g.
>>>>> IP,
>>>>> version, etc.)
>>>> 
>>>> 
>>>> Yes, we could use events or some other similar API provided by GA.
>>>> 
>>>> 
>>>>> What the download page CGI developed in? PHP?
>>>>> 
>>>> 
>>>> To be honest, no clue. I guess someone in the community can figure it
>>> out:
>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>> 
>>>> 
>>>>> However, I'm not sure whether storing this data in a 3rd party
>> (Google)
>>> is
>>>>> compliant with the ASF policy. I guess it's no biggie, but if there's
>>>>> doubt
>>>>> in the PMC, it's better to ask legal@.
>>>> 
>>>> 
>>>> I am not sure there is anything to ask about. The whole Ignite website
>> is
>>>> GA enabled, and all we are doing is accessing a standard web page from
>>> the
>>>> Ignite web site. The information gathered from GA is available only to
>>> the
>>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
>>>> question.
>>>> 
>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>> foundation/policies/privacy.html
>>>> 
>>>> 
>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>> 
>>>> 
>>>> Sure, would be nice to hear from Roman as well.
>>>> 
>>>> 
>>>>> Cheers.
>>>>> 
>>>>> [1]
>>>>> https://developers.google.com/analytics/devguides/collection
>>>>> /analyticsjs/events
>>>>> 
>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>> wrote:
>>>>> 
>>>>>> Raul,
>>>>>> 
>>>>>> Could point about Javascript, it will not work because it executes
>> in
>>>>> the
>>>>>> browser. This means we need a server-side script, like CGI we are
>>> using
>>>>> on
>>>>>> our download page.
>>>>>> 
>>>>>> How about this approach. We create something like ignite-version.cgi
>>>>> script
>>>>>> which will invoke a call to GA and then return the latest version.
>>>>>> 
>>>>>> This should work, right?
>>>>>> 
>>>>>> D.
>>>>>> 
>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>> raul.asf@evosent.com
>>> <javascript:;>>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey Dmitriy and all
>>>>>>> 
>>>>>>> Also, since we have GA enabled for the website, we can track how
>>> many
>>>>>> times
>>>>>>>> this page was accessed, which will be equal to the number of
>>> starts.
>>>>>> This
>>>>>>>> way, the counter information is tracked and monitored by the
>>> Ignite
>>>>>> PMC.
>>>>>>> 
>>>>>>> 
>>>>>>> Unfortunately this won't work because GA is loaded via Javascript
>> on
>>>>> the
>>>>>>> browser, so Google will never receive the page hit.
>>>>>>> 
>>>>>>> Given the constraints, the most viable solution is an HTTPS
>> endpoint
>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>> request.
>>>>> The
>>>>>>> simplest thing is to write each request in a log file, along with
>>> the
>>>>>>> timestamp, the version reported by the client, maybe the IP (not
>>> sure
>>>>>> about
>>>>>>> the ASF rules about this concerning privacy, I guess it's OK if
>> you
>>>>> make
>>>>>> it
>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
>>> creates
>>>>> on
>>>>>>> first startup or something.
>>>>>>> 
>>>>>>> This endpoint would need some basic DDoS protection and
>> blacklisting
>>>>> to
>>>>>>> prevent data spoofing.
>>>>>>> 
>>>>>>> If we'll be implementing this endpoint anyway, then there's no
>> point
>>>>>>> placing another file on Git or elsewhere for reporting the latest
>>>>>> versions:
>>>>>>> the endpoint itself can return them.
>>>>>>> 
>>>>>>> WDYT?
>>>>>>> 
>>>>>>> Cheers.
>>>>>>> 
>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Cos, Raul,
>>>>>>>> 
>>>>>>>> Thanks for the feedback. I completely agree about Maven Central
>>>>> being a
>>>>>>> 3rd
>>>>>>>> party repo (did not think about that initially). All your
>>>>> suggestions
>>>>>>> make
>>>>>>>> sense, but I would like to keep it as simple as possible, and so
>>> far
>>>>>>>> everything suggested required GIT dependencies and extra work.
>>>>>>>> 
>>>>>>>> How about Yakov's suggestion. We simply add a page to the Ignite
>>>>>> website
>>>>>>>> which will have only the latest version. Every time a node
>> starts,
>>>>> it
>>>>>>>> receives the latest version from the page and suggests that
>> users
>>>>>> upgrade
>>>>>>>> if needed.
>>>>>>>> 
>>>>>>>> Also, since we have GA enabled for the website, we can track how
>>>>> many
>>>>>>> times
>>>>>>>> this page was accessed, which will be equal to the number of
>>> starts.
>>>>>> This
>>>>>>>> way, the counter information is tracked and monitored by the
>>> Ignite
>>>>>> PMC.
>>>>>>>> 
>>>>>>>> This approach looks pretty innocent to me and everything is kept
>>> and
>>>>>>>> managed within Apache.
>>>>>>>> 
>>>>>>>> Thoughts?
>>>>>>>> 
>>>>>>>> D.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>> cos@apache.org <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I agree with Raul.
>>>>>>>>> - providing a ping-back address to a 3rd party might be frown
>>>>> upon by
>>>>>>>> some.
>>>>>>>>>  And might have a consequences like stats collection about
>>> users'
>>>>>>>>>  infrastructure.
>>>>>>>>> - checking an ASF git-repo is easy and won't download any
>> binary
>>>>>> data:
>>>>>>>>>  everything is clear text and could be easily monitored by
>> any
>>> of
>>>>>> the
>>>>>>>>> network
>>>>>>>>>  diagnostic tools, shall it be required. But it involves a
>> bit
>>> of
>>>>>> the
>>>>>>>>> release
>>>>>>>>>  discipline.
>>>>>>>>> - the binary data download in the runtime is my main concern.
>>>>> This is
>>>>>>> the
>>>>>>>>>  vector of MMA. What if someone gains the control over the
>>>>>> repository
>>>>>>>> and
>>>>>>>>>  replaces the file with some malicious content.
>>>>>>>>> 
>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a
>> call
>>>>> to
>>>>>> an
>>>>>>>>> external script to check upon the atest software version
>>> available
>>>>>> for
>>>>>>>>> download. In the past, the endpoint was running on a 3rd party
>>>>>> server,
>>>>>>> I
>>>>>>>>> believe the best way would be to put this script on ASF infra
>>> and
>>>>>> have
>>>>>>>> the
>>>>>>>>> "update checker" running in a completely controlled
>> environment.
>>>>>>>> Actually,
>>>>>>>>> I
>>>>>>>>> recall we had this very discussion during the Incubation; I
>> can
>>>>>>> probably
>>>>>>>>> dig
>>>>>>>>> out the corresponding thread.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>>  Cok
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>> Hey guys
>>>>>>>>>> 
>>>>>>>>>> In my opinion, maven.org is still owned by a third party
>>>>>> (Sonatype).
>>>>>>>> We
>>>>>>>>>> don't know what kind of data analysis or intelligence
>>> extraction
>>>>>> they
>>>>>>>>> run.
>>>>>>>>>> 
>>>>>>>>>> If Ignite servers all over the world were hitting maven.org
>>>>>>>> periodically
>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a
>> clear
>>>>>>>> indication
>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>> 
>>>>>>>>>> They could reverse lookup the IP address and find out what
>>>>>>> corporation
>>>>>>>> it
>>>>>>>>>> is.
>>>>>>>>>> 
>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>> 
>>>>>>>>>> We could use the Git ssh API, but that would require a new
>>>>>>> dependency,
>>>>>>>>>> which I advise against.
>>>>>>>>>> 
>>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git
>>>>> tags:
>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>> 
>>>>>>>>>> Another option is to place a txt file in the root of the
>>> master
>>>>>>> branch
>>>>>>>>> (e.g
>>>>>>>>>> LATEST), containing a list of the latest GA versions for
>> each
>>>>> major
>>>>>>>>> version
>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>> 
>>>>>>>>>> I would advocate this last option, but it requires somebody
>>>>>>> remembering
>>>>>>>>> to
>>>>>>>>>> update the file with every release, unless we automate it
>>> with a
>>>>>>> Maven
>>>>>>>>>> plugin.
>>>>>>>>>> 
>>>>>>>>>> Hope that helps!
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>> <javascript:;>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>> 
>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is
>> it
>>>>> good
>>>>>>> to
>>>>>>>>> go?
>>>>>>>>>> 
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>> 
>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket
>> on
>>>>> it?
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>> 
>>>>>>>>>>>> There has been lots of talk of proposals about various
>>> usage
>>>>>>> metrics
>>>>>>>>> for
>>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect
>>> the
>>>>>> topic
>>>>>>>> and
>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>> The main purpose of the update checker is not to collect
>>>>>> metrics,
>>>>>>>> but
>>>>>>>>> to
>>>>>>>>>>>> notify users about a new version of Ignite by accessing
>>>>>> maven.org
>>>>>>>> and
>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>> 
>>>>>>>>>>>> This way we do not send any information anywhere and, at
>>> the
>>>>>> same
>>>>>>>>> time,
>>>>>>>>>>>> urge our users to download and start using the latest
>>>>> version of
>>>>>>>>> Ignite.
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>> This piece is optional, but we can also get an insight in
>>> how
>>>>>> many
>>>>>>>>> times
>>>>>>>>>> a
>>>>>>>>>>>> certain Ignite release gets started. This is just a cool
>>>>> metric
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>> community to gauge the project popularity. You can think
>> of
>>>>> it
>>>>>> as
>>>>>>>> of a
>>>>>>>>>> page
>>>>>>>>>>>> visit counter shown on many websites. We can even decide
>> to
>>>>>>> display
>>>>>>>>> this
>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>> 
>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every
>>>>> release,
>>>>>>> e.g.
>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte.
>>>>> Every
>>>>>>> time
>>>>>>>>> an
>>>>>>>>>>>> Ignite node starts, it will download this JAR in the
>>>>> background.
>>>>>>>> Then
>>>>>>>>> we
>>>>>>>>>>>> will be able to view the number of the total downloads
>> for
>>>>> this
>>>>>>> JAR
>>>>>>>> in
>>>>>>>>>>>> Maven Central, which is essentially the number of starts
>> of
>>>>>> Ignite
>>>>>>>>> nodes.
>>>>>>>>>>>> 
>>>>>>>>>>>> *Note that neither of the above suggestions require
>> Ignite
>>> to
>>>>>> send
>>>>>>>> or
>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>> 
>>>>>>>>>>>> Please reply suggesting weather you are OK with this
>>>>> approach.
>>>>>>>>>>>> 
>>>>>>>>>>>> D.
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

Denis,

I am not sure I understand. We already do have CGI enabled for
download.cgi. Is there something else we need?

D.

On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dm...@gridgain.com> wrote:

> There is an obstacle. There is no way to execute the script using PHP or
> similar sever side language and trigger GA as discussed earlier:
> https://issues.apache.org/jira/browse/INFRA-15182
>
> How else can we tackle this?
>
> Denis
>
> On Thursday, September 7, 2017, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > I think it is safe to assume at this point that everyone is in general
> > agreement, since there are no active objections.
> >
> > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> > https://issues.apache.org/jira/browse/IGNITE-6305
> >
> > D.
> >
> > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > <javascript:;>>
> > wrote:
> >
> > >
> > >
> > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
> > <javascript:;>>
> > > wrote:
> > >
> > >> Yeah, I guess that's doable as well and requires less management
> effort
> > >> than my suggestion. We could use events [1] to store payload data
> (e.g.
> > >> IP,
> > >> version, etc.)
> > >
> > >
> > > Yes, we could use events or some other similar API provided by GA.
> > >
> > >
> > >> What the download page CGI developed in? PHP?
> > >>
> > >
> > > To be honest, no clue. I guess someone in the community can figure it
> > out:
> > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >
> > >
> > >> However, I'm not sure whether storing this data in a 3rd party
> (Google)
> > is
> > >> compliant with the ASF policy. I guess it's no biggie, but if there's
> > >> doubt
> > >> in the PMC, it's better to ask legal@.
> > >
> > >
> > > I am not sure there is anything to ask about. The whole Ignite website
> is
> > > GA enabled, and all we are doing is accessing a standard web page from
> > the
> > > Ignite web site. The information gathered from GA is available only to
> > the
> > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > > question.
> > >
> > > Even ASF website itself uses GA: https://www.apache.org/
> > > foundation/policies/privacy.html
> > >
> > >
> > >> I think Cos said it's OK; maybe Roman can pitch in.
> > >>
> > >
> > >  Sure, would be nice to hear from Roman as well.
> > >
> > >
> > >> Cheers.
> > >>
> > >> [1]
> > >> https://developers.google.com/analytics/devguides/collection
> > >> /analyticsjs/events
> > >>
> > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >> dsetrakyan@apache.org <javascript:;>>
> > >> wrote:
> > >>
> > >> > Raul,
> > >> >
> > >> > Could point about Javascript, it will not work because it executes
> in
> > >> the
> > >> > browser. This means we need a server-side script, like CGI we are
> > using
> > >> on
> > >> > our download page.
> > >> >
> > >> > How about this approach. We create something like ignite-version.cgi
> > >> script
> > >> > which will invoke a call to GA and then return the latest version.
> > >> >
> > >> > This should work, right?
> > >> >
> > >> > D.
> > >> >
> > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> raul.asf@evosent.com
> > <javascript:;>>
> > >> > wrote:
> > >> >
> > >> > > Hey Dmitriy and all
> > >> > >
> > >> > > Also, since we have GA enabled for the website, we can track how
> > many
> > >> > times
> > >> > > > this page was accessed, which will be equal to the number of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by the
> > Ignite
> > >> > PMC.
> > >> > >
> > >> > >
> > >> > > Unfortunately this won't work because GA is loaded via Javascript
> on
> > >> the
> > >> > > browser, so Google will never receive the page hit.
> > >> > >
> > >> > > Given the constraints, the most viable solution is an HTTPS
> endpoint
> > >> > > running on ASF infra that Ignite invokes via a GET or POST
> request.
> > >> The
> > >> > > simplest thing is to write each request in a log file, along with
> > the
> > >> > > timestamp, the version reported by the client, maybe the IP (not
> > sure
> > >> > about
> > >> > > the ASF rules about this concerning privacy, I guess it's OK if
> you
> > >> make
> > >> > it
> > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> > creates
> > >> on
> > >> > > first startup or something.
> > >> > >
> > >> > > This endpoint would need some basic DDoS protection and
> blacklisting
> > >> to
> > >> > > prevent data spoofing.
> > >> > >
> > >> > > If we'll be implementing this endpoint anyway, then there's no
> point
> > >> > > placing another file on Git or elsewhere for reporting the latest
> > >> > versions:
> > >> > > the endpoint itself can return them.
> > >> > >
> > >> > > WDYT?
> > >> > >
> > >> > > Cheers.
> > >> > >
> > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > >> > dsetrakyan@apache.org <javascript:;>>
> > >> > > wrote:
> > >> > >
> > >> > > > Cos, Raul,
> > >> > > >
> > >> > > > Thanks for the feedback. I completely agree about Maven Central
> > >> being a
> > >> > > 3rd
> > >> > > > party repo (did not think about that initially). All your
> > >> suggestions
> > >> > > make
> > >> > > > sense, but I would like to keep it as simple as possible, and so
> > far
> > >> > > > everything suggested required GIT dependencies and extra work.
> > >> > > >
> > >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> > >> > website
> > >> > > > which will have only the latest version. Every time a node
> starts,
> > >> it
> > >> > > > receives the latest version from the page and suggests that
> users
> > >> > upgrade
> > >> > > > if needed.
> > >> > > >
> > >> > > > Also, since we have GA enabled for the website, we can track how
> > >> many
> > >> > > times
> > >> > > > this page was accessed, which will be equal to the number of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by the
> > Ignite
> > >> > PMC.
> > >> > > >
> > >> > > > This approach looks pretty innocent to me and everything is kept
> > and
> > >> > > > managed within Apache.
> > >> > > >
> > >> > > > Thoughts?
> > >> > > >
> > >> > > > D.
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > >> cos@apache.org <javascript:;>>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > I agree with Raul.
> > >> > > > > - providing a ping-back address to a 3rd party might be frown
> > >> upon by
> > >> > > > some.
> > >> > > > >   And might have a consequences like stats collection about
> > users'
> > >> > > > >   infrastructure.
> > >> > > > > - checking an ASF git-repo is easy and won't download any
> binary
> > >> > data:
> > >> > > > >   everything is clear text and could be easily monitored by
> any
> > of
> > >> > the
> > >> > > > > network
> > >> > > > >   diagnostic tools, shall it be required. But it involves a
> bit
> > of
> > >> > the
> > >> > > > > release
> > >> > > > >   discipline.
> > >> > > > > - the binary data download in the runtime is my main concern.
> > >> This is
> > >> > > the
> > >> > > > >   vector of MMA. What if someone gains the control over the
> > >> > repository
> > >> > > > and
> > >> > > > >   replaces the file with some malicious content.
> > >> > > > >
> > >> > > > > As for the particular mechanism: IIRC Ignite used to make a
> call
> > >> to
> > >> > an
> > >> > > > > external script to check upon the atest software version
> > available
> > >> > for
> > >> > > > > download. In the past, the endpoint was running on a 3rd party
> > >> > server,
> > >> > > I
> > >> > > > > believe the best way would be to put this script on ASF infra
> > and
> > >> > have
> > >> > > > the
> > >> > > > > "update checker" running in a completely controlled
> environment.
> > >> > > > Actually,
> > >> > > > > I
> > >> > > > > recall we had this very discussion during the Incubation; I
> can
> > >> > > probably
> > >> > > > > dig
> > >> > > > > out the corresponding thread.
> > >> > > > >
> > >> > > > > Thoughts?
> > >> > > > >   Cok
> > >> > > > >
> > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > >> > > > > > Hey guys
> > >> > > > > >
> > >> > > > > > In my opinion, maven.org is still owned by a third party
> > >> > (Sonatype).
> > >> > > > We
> > >> > > > > > don't know what kind of data analysis or intelligence
> > extraction
> > >> > they
> > >> > > > > run.
> > >> > > > > >
> > >> > > > > > If Ignite servers all over the world were hitting maven.org
> > >> > > > periodically
> > >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a
> clear
> > >> > > > indication
> > >> > > > > > of who is running an Ignite server.
> > >> > > > > >
> > >> > > > > > They could reverse lookup the IP address and find out what
> > >> > > corporation
> > >> > > > it
> > >> > > > > > is.
> > >> > > > > >
> > >> > > > > > How about having Ignite check the ASF Git directly?
> > >> > > > > >
> > >> > > > > > We could use the Git ssh API, but that would require a new
> > >> > > dependency,
> > >> > > > > > which I advise against.
> > >> > > > > >
> > >> > > > > > Alternatively, we could have it scrape this HTML for new Git
> > >> tags:
> > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >> > > > > >
> > >> > > > > > Another option is to place a txt file in the root of the
> > master
> > >> > > branch
> > >> > > > > (e.g
> > >> > > > > > LATEST), containing a list of the latest GA versions for
> each
> > >> major
> > >> > > > > version
> > >> > > > > > line (1.x, 2.x).
> > >> > > > > >
> > >> > > > > > I would advocate this last option, but it requires somebody
> > >> > > remembering
> > >> > > > > to
> > >> > > > > > update the file with every release, unless we automate it
> > with a
> > >> > > Maven
> > >> > > > > > plugin.
> > >> > > > > >
> > >> > > > > > Hope that helps!
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> > <javascript:;>> wrote:
> > >> > > > > >
> > >> > > > > > I see nothing wrong with this approach.
> > >> > > > > >
> > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is
> it
> > >> good
> > >> > > to
> > >> > > > > go?
> > >> > > > > >
> > >> > > > > > —
> > >> > > > > > Denis
> > >> > > > > >
> > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > >> > > > dsetrakyan@apache.org <javascript:;>
> > >> > > > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Is everyone OK with this approach? Should I file a ticket
> on
> > >> it?
> > >> > > > > > >
> > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > >> > > > > dsetrakyan@apache.org <javascript:;>>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > >> Igniters,
> > >> > > > > > >>
> > >> > > > > > >> There has been lots of talk of proposals about various
> > usage
> > >> > > metrics
> > >> > > > > for
> > >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
> > the
> > >> > topic
> > >> > > > and
> > >> > > > > > >> propose something very simple and non-intrusive.
> > >> > > > > > >>
> > >> > > > > > >> 1. Update Checker
> > >> > > > > > >> The main purpose of the update checker is not to collect
> > >> > metrics,
> > >> > > > but
> > >> > > > > to
> > >> > > > > > >> notify users about a new version of Ignite by accessing
> > >> > maven.org
> > >> > > > and
> > >> > > > > > >> getting the version out of the metadata file:
> > >> > > > > > >> http://repo2.maven.org/maven2/
> > org/apache/ignite/ignite-core/
> > >> > > > > > >> maven-metadata.xml
> > >> > > > > > >>
> > >> > > > > > >> This way we do not send any information anywhere and, at
> > the
> > >> > same
> > >> > > > > time,
> > >> > > > > > >> urge our users to download and start using the latest
> > >> version of
> > >> > > > > Ignite.
> > >> > > > > > >>
> > >> > > > > > >> 2. Startup Counter
> > >> > > > > > >> This piece is optional, but we can also get an insight in
> > how
> > >> > many
> > >> > > > > times
> > >> > > > > > a
> > >> > > > > > >> certain Ignite release gets started. This is just a cool
> > >> metric
> > >> > > for
> > >> > > > > the
> > >> > > > > > >> community to gauge the project popularity. You can think
> of
> > >> it
> > >> > as
> > >> > > > of a
> > >> > > > > > page
> > >> > > > > > >> visit counter shown on many websites. We can even decide
> to
> > >> > > display
> > >> > > > > this
> > >> > > > > > >> counter on the Ignite website as well.
> > >> > > > > > >>
> > >> > > > > > >> To do this, we can simply add a JAR in maven for every
> > >> release,
> > >> > > e.g.
> > >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> > >> Every
> > >> > > time
> > >> > > > > an
> > >> > > > > > >> Ignite node starts, it will download this JAR in the
> > >> background.
> > >> > > > Then
> > >> > > > > we
> > >> > > > > > >> will be able to view the number of the total downloads
> for
> > >> this
> > >> > > JAR
> > >> > > > in
> > >> > > > > > >> Maven Central, which is essentially the number of starts
> of
> > >> > Ignite
> > >> > > > > nodes.
> > >> > > > > > >>
> > >> > > > > > >> *Note that neither of the above suggestions require
> Ignite
> > to
> > >> > send
> > >> > > > or
> > >> > > > > > >> track any user information whatsoever.*
> > >> > > > > > >>
> > >> > > > > > >> Please reply suggesting weather you are OK with this
> > >> approach.
> > >> > > > > > >>
> > >> > > > > > >> D.
> > >> > > > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@gridgain.com>.

There is an obstacle. There is no way to execute the script using PHP or
similar sever side language and trigger GA as discussed earlier:
https://issues.apache.org/jira/browse/INFRA-15182

How else can we tackle this?

Denis

On Thursday, September 7, 2017, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> I think it is safe to assume at this point that everyone is in general
> agreement, since there are no active objections.
>
> I have filed a ticket for the 2.3 release. Let's try to make it happen:
> https://issues.apache.org/jira/browse/IGNITE-6305
>
> D.
>
> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> <javascript:;>>
> wrote:
>
> >
> >
> > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
> <javascript:;>>
> > wrote:
> >
> >> Yeah, I guess that's doable as well and requires less management effort
> >> than my suggestion. We could use events [1] to store payload data (e.g.
> >> IP,
> >> version, etc.)
> >
> >
> > Yes, we could use events or some other similar API provided by GA.
> >
> >
> >> What the download page CGI developed in? PHP?
> >>
> >
> > To be honest, no clue. I guess someone in the community can figure it
> out:
> > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >
> >
> >> However, I'm not sure whether storing this data in a 3rd party (Google)
> is
> >> compliant with the ASF policy. I guess it's no biggie, but if there's
> >> doubt
> >> in the PMC, it's better to ask legal@.
> >
> >
> > I am not sure there is anything to ask about. The whole Ignite website is
> > GA enabled, and all we are doing is accessing a standard web page from
> the
> > Ignite web site. The information gathered from GA is available only to
> the
> > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > question.
> >
> > Even ASF website itself uses GA: https://www.apache.org/
> > foundation/policies/privacy.html
> >
> >
> >> I think Cos said it's OK; maybe Roman can pitch in.
> >>
> >
> >  Sure, would be nice to hear from Roman as well.
> >
> >
> >> Cheers.
> >>
> >> [1]
> >> https://developers.google.com/analytics/devguides/collection
> >> /analyticsjs/events
> >>
> >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >> dsetrakyan@apache.org <javascript:;>>
> >> wrote:
> >>
> >> > Raul,
> >> >
> >> > Could point about Javascript, it will not work because it executes in
> >> the
> >> > browser. This means we need a server-side script, like CGI we are
> using
> >> on
> >> > our download page.
> >> >
> >> > How about this approach. We create something like ignite-version.cgi
> >> script
> >> > which will invoke a call to GA and then return the latest version.
> >> >
> >> > This should work, right?
> >> >
> >> > D.
> >> >
> >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <raul.asf@evosent.com
> <javascript:;>>
> >> > wrote:
> >> >
> >> > > Hey Dmitriy and all
> >> > >
> >> > > Also, since we have GA enabled for the website, we can track how
> many
> >> > times
> >> > > > this page was accessed, which will be equal to the number of
> starts.
> >> > This
> >> > > > way, the counter information is tracked and monitored by the
> Ignite
> >> > PMC.
> >> > >
> >> > >
> >> > > Unfortunately this won't work because GA is loaded via Javascript on
> >> the
> >> > > browser, so Google will never receive the page hit.
> >> > >
> >> > > Given the constraints, the most viable solution is an HTTPS endpoint
> >> > > running on ASF infra that Ignite invokes via a GET or POST request.
> >> The
> >> > > simplest thing is to write each request in a log file, along with
> the
> >> > > timestamp, the version reported by the client, maybe the IP (not
> sure
> >> > about
> >> > > the ASF rules about this concerning privacy, I guess it's OK if you
> >> make
> >> > it
> >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> creates
> >> on
> >> > > first startup or something.
> >> > >
> >> > > This endpoint would need some basic DDoS protection and blacklisting
> >> to
> >> > > prevent data spoofing.
> >> > >
> >> > > If we'll be implementing this endpoint anyway, then there's no point
> >> > > placing another file on Git or elsewhere for reporting the latest
> >> > versions:
> >> > > the endpoint itself can return them.
> >> > >
> >> > > WDYT?
> >> > >
> >> > > Cheers.
> >> > >
> >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >> > dsetrakyan@apache.org <javascript:;>>
> >> > > wrote:
> >> > >
> >> > > > Cos, Raul,
> >> > > >
> >> > > > Thanks for the feedback. I completely agree about Maven Central
> >> being a
> >> > > 3rd
> >> > > > party repo (did not think about that initially). All your
> >> suggestions
> >> > > make
> >> > > > sense, but I would like to keep it as simple as possible, and so
> far
> >> > > > everything suggested required GIT dependencies and extra work.
> >> > > >
> >> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> >> > website
> >> > > > which will have only the latest version. Every time a node starts,
> >> it
> >> > > > receives the latest version from the page and suggests that users
> >> > upgrade
> >> > > > if needed.
> >> > > >
> >> > > > Also, since we have GA enabled for the website, we can track how
> >> many
> >> > > times
> >> > > > this page was accessed, which will be equal to the number of
> starts.
> >> > This
> >> > > > way, the counter information is tracked and monitored by the
> Ignite
> >> > PMC.
> >> > > >
> >> > > > This approach looks pretty innocent to me and everything is kept
> and
> >> > > > managed within Apache.
> >> > > >
> >> > > > Thoughts?
> >> > > >
> >> > > > D.
> >> > > >
> >> > > >
> >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> >> cos@apache.org <javascript:;>>
> >> > > > wrote:
> >> > > >
> >> > > > > I agree with Raul.
> >> > > > > - providing a ping-back address to a 3rd party might be frown
> >> upon by
> >> > > > some.
> >> > > > >   And might have a consequences like stats collection about
> users'
> >> > > > >   infrastructure.
> >> > > > > - checking an ASF git-repo is easy and won't download any binary
> >> > data:
> >> > > > >   everything is clear text and could be easily monitored by any
> of
> >> > the
> >> > > > > network
> >> > > > >   diagnostic tools, shall it be required. But it involves a bit
> of
> >> > the
> >> > > > > release
> >> > > > >   discipline.
> >> > > > > - the binary data download in the runtime is my main concern.
> >> This is
> >> > > the
> >> > > > >   vector of MMA. What if someone gains the control over the
> >> > repository
> >> > > > and
> >> > > > >   replaces the file with some malicious content.
> >> > > > >
> >> > > > > As for the particular mechanism: IIRC Ignite used to make a call
> >> to
> >> > an
> >> > > > > external script to check upon the atest software version
> available
> >> > for
> >> > > > > download. In the past, the endpoint was running on a 3rd party
> >> > server,
> >> > > I
> >> > > > > believe the best way would be to put this script on ASF infra
> and
> >> > have
> >> > > > the
> >> > > > > "update checker" running in a completely controlled environment.
> >> > > > Actually,
> >> > > > > I
> >> > > > > recall we had this very discussion during the Incubation; I can
> >> > > probably
> >> > > > > dig
> >> > > > > out the corresponding thread.
> >> > > > >
> >> > > > > Thoughts?
> >> > > > >   Cok
> >> > > > >
> >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> >> > > > > > Hey guys
> >> > > > > >
> >> > > > > > In my opinion, maven.org is still owned by a third party
> >> > (Sonatype).
> >> > > > We
> >> > > > > > don't know what kind of data analysis or intelligence
> extraction
> >> > they
> >> > > > > run.
> >> > > > > >
> >> > > > > > If Ignite servers all over the world were hitting maven.org
> >> > > > periodically
> >> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> >> > > > indication
> >> > > > > > of who is running an Ignite server.
> >> > > > > >
> >> > > > > > They could reverse lookup the IP address and find out what
> >> > > corporation
> >> > > > it
> >> > > > > > is.
> >> > > > > >
> >> > > > > > How about having Ignite check the ASF Git directly?
> >> > > > > >
> >> > > > > > We could use the Git ssh API, but that would require a new
> >> > > dependency,
> >> > > > > > which I advise against.
> >> > > > > >
> >> > > > > > Alternatively, we could have it scrape this HTML for new Git
> >> tags:
> >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >> > > > > >
> >> > > > > > Another option is to place a txt file in the root of the
> master
> >> > > branch
> >> > > > > (e.g
> >> > > > > > LATEST), containing a list of the latest GA versions for each
> >> major
> >> > > > > version
> >> > > > > > line (1.x, 2.x).
> >> > > > > >
> >> > > > > > I would advocate this last option, but it requires somebody
> >> > > remembering
> >> > > > > to
> >> > > > > > update the file with every release, unless we automate it
> with a
> >> > > Maven
> >> > > > > > plugin.
> >> > > > > >
> >> > > > > > Hope that helps!
> >> > > > > >
> >> > > > > >
> >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> <javascript:;>> wrote:
> >> > > > > >
> >> > > > > > I see nothing wrong with this approach.
> >> > > > > >
> >> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it
> >> good
> >> > > to
> >> > > > > go?
> >> > > > > >
> >> > > > > > —
> >> > > > > > Denis
> >> > > > > >
> >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> >> > > > dsetrakyan@apache.org <javascript:;>
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > Is everyone OK with this approach? Should I file a ticket on
> >> it?
> >> > > > > > >
> >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> >> > > > > dsetrakyan@apache.org <javascript:;>>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > >> Igniters,
> >> > > > > > >>
> >> > > > > > >> There has been lots of talk of proposals about various
> usage
> >> > > metrics
> >> > > > > for
> >> > > > > > >> Ignite and nothing came of it. I would like to resurrect
> the
> >> > topic
> >> > > > and
> >> > > > > > >> propose something very simple and non-intrusive.
> >> > > > > > >>
> >> > > > > > >> 1. Update Checker
> >> > > > > > >> The main purpose of the update checker is not to collect
> >> > metrics,
> >> > > > but
> >> > > > > to
> >> > > > > > >> notify users about a new version of Ignite by accessing
> >> > maven.org
> >> > > > and
> >> > > > > > >> getting the version out of the metadata file:
> >> > > > > > >> http://repo2.maven.org/maven2/
> org/apache/ignite/ignite-core/
> >> > > > > > >> maven-metadata.xml
> >> > > > > > >>
> >> > > > > > >> This way we do not send any information anywhere and, at
> the
> >> > same
> >> > > > > time,
> >> > > > > > >> urge our users to download and start using the latest
> >> version of
> >> > > > > Ignite.
> >> > > > > > >>
> >> > > > > > >> 2. Startup Counter
> >> > > > > > >> This piece is optional, but we can also get an insight in
> how
> >> > many
> >> > > > > times
> >> > > > > > a
> >> > > > > > >> certain Ignite release gets started. This is just a cool
> >> metric
> >> > > for
> >> > > > > the
> >> > > > > > >> community to gauge the project popularity. You can think of
> >> it
> >> > as
> >> > > > of a
> >> > > > > > page
> >> > > > > > >> visit counter shown on many websites. We can even decide to
> >> > > display
> >> > > > > this
> >> > > > > > >> counter on the Ignite website as well.
> >> > > > > > >>
> >> > > > > > >> To do this, we can simply add a JAR in maven for every
> >> release,
> >> > > e.g.
> >> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> >> Every
> >> > > time
> >> > > > > an
> >> > > > > > >> Ignite node starts, it will download this JAR in the
> >> background.
> >> > > > Then
> >> > > > > we
> >> > > > > > >> will be able to view the number of the total downloads for
> >> this
> >> > > JAR
> >> > > > in
> >> > > > > > >> Maven Central, which is essentially the number of starts of
> >> > Ignite
> >> > > > > nodes.
> >> > > > > > >>
> >> > > > > > >> *Note that neither of the above suggestions require Ignite
> to
> >> > send
> >> > > > or
> >> > > > > > >> track any user information whatsoever.*
> >> > > > > > >>
> >> > > > > > >> Please reply suggesting weather you are OK with this
> >> approach.
> >> > > > > > >>
> >> > > > > > >> D.
> >> > > > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

I think it is safe to assume at this point that everyone is in general
agreement, since there are no active objections.

I have filed a ticket for the 2.3 release. Let's try to make it happen:
https://issues.apache.org/jira/browse/IGNITE-6305

D.

On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

>
>
> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <ra...@evosent.com>
> wrote:
>
>> Yeah, I guess that's doable as well and requires less management effort
>> than my suggestion. We could use events [1] to store payload data (e.g.
>> IP,
>> version, etc.)
>
>
> Yes, we could use events or some other similar API provided by GA.
>
>
>> What the download page CGI developed in? PHP?
>>
>
> To be honest, no clue. I guess someone in the community can figure it out:
> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>
>
>> However, I'm not sure whether storing this data in a 3rd party (Google) is
>> compliant with the ASF policy. I guess it's no biggie, but if there's
>> doubt
>> in the PMC, it's better to ask legal@.
>
>
> I am not sure there is anything to ask about. The whole Ignite website is
> GA enabled, and all we are doing is accessing a standard web page from the
> Ignite web site. The information gathered from GA is available only to the
> Ignite PMC. Frankly, I think legal@ will be very confused by this
> question.
>
> Even ASF website itself uses GA: https://www.apache.org/
> foundation/policies/privacy.html
>
>
>> I think Cos said it's OK; maybe Roman can pitch in.
>>
>
>  Sure, would be nice to hear from Roman as well.
>
>
>> Cheers.
>>
>> [1]
>> https://developers.google.com/analytics/devguides/collection
>> /analyticsjs/events
>>
>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org>
>> wrote:
>>
>> > Raul,
>> >
>> > Could point about Javascript, it will not work because it executes in
>> the
>> > browser. This means we need a server-side script, like CGI we are using
>> on
>> > our download page.
>> >
>> > How about this approach. We create something like ignite-version.cgi
>> script
>> > which will invoke a call to GA and then return the latest version.
>> >
>> > This should work, right?
>> >
>> > D.
>> >
>> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <ra...@evosent.com>
>> > wrote:
>> >
>> > > Hey Dmitriy and all
>> > >
>> > > Also, since we have GA enabled for the website, we can track how many
>> > times
>> > > > this page was accessed, which will be equal to the number of starts.
>> > This
>> > > > way, the counter information is tracked and monitored by the Ignite
>> > PMC.
>> > >
>> > >
>> > > Unfortunately this won't work because GA is loaded via Javascript on
>> the
>> > > browser, so Google will never receive the page hit.
>> > >
>> > > Given the constraints, the most viable solution is an HTTPS endpoint
>> > > running on ASF infra that Ignite invokes via a GET or POST request.
>> The
>> > > simplest thing is to write each request in a log file, along with the
>> > > timestamp, the version reported by the client, maybe the IP (not sure
>> > about
>> > > the ASF rules about this concerning privacy, I guess it's OK if you
>> make
>> > it
>> > > an opt-in) and a unique node identifier, i.e. a UUID the node creates
>> on
>> > > first startup or something.
>> > >
>> > > This endpoint would need some basic DDoS protection and blacklisting
>> to
>> > > prevent data spoofing.
>> > >
>> > > If we'll be implementing this endpoint anyway, then there's no point
>> > > placing another file on Git or elsewhere for reporting the latest
>> > versions:
>> > > the endpoint itself can return them.
>> > >
>> > > WDYT?
>> > >
>> > > Cheers.
>> > >
>> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>> > dsetrakyan@apache.org>
>> > > wrote:
>> > >
>> > > > Cos, Raul,
>> > > >
>> > > > Thanks for the feedback. I completely agree about Maven Central
>> being a
>> > > 3rd
>> > > > party repo (did not think about that initially). All your
>> suggestions
>> > > make
>> > > > sense, but I would like to keep it as simple as possible, and so far
>> > > > everything suggested required GIT dependencies and extra work.
>> > > >
>> > > > How about Yakov's suggestion. We simply add a page to the Ignite
>> > website
>> > > > which will have only the latest version. Every time a node starts,
>> it
>> > > > receives the latest version from the page and suggests that users
>> > upgrade
>> > > > if needed.
>> > > >
>> > > > Also, since we have GA enabled for the website, we can track how
>> many
>> > > times
>> > > > this page was accessed, which will be equal to the number of starts.
>> > This
>> > > > way, the counter information is tracked and monitored by the Ignite
>> > PMC.
>> > > >
>> > > > This approach looks pretty innocent to me and everything is kept and
>> > > > managed within Apache.
>> > > >
>> > > > Thoughts?
>> > > >
>> > > > D.
>> > > >
>> > > >
>> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>> cos@apache.org>
>> > > > wrote:
>> > > >
>> > > > > I agree with Raul.
>> > > > > - providing a ping-back address to a 3rd party might be frown
>> upon by
>> > > > some.
>> > > > >   And might have a consequences like stats collection about users'
>> > > > >   infrastructure.
>> > > > > - checking an ASF git-repo is easy and won't download any binary
>> > data:
>> > > > >   everything is clear text and could be easily monitored by any of
>> > the
>> > > > > network
>> > > > >   diagnostic tools, shall it be required. But it involves a bit of
>> > the
>> > > > > release
>> > > > >   discipline.
>> > > > > - the binary data download in the runtime is my main concern.
>> This is
>> > > the
>> > > > >   vector of MMA. What if someone gains the control over the
>> > repository
>> > > > and
>> > > > >   replaces the file with some malicious content.
>> > > > >
>> > > > > As for the particular mechanism: IIRC Ignite used to make a call
>> to
>> > an
>> > > > > external script to check upon the atest software version available
>> > for
>> > > > > download. In the past, the endpoint was running on a 3rd party
>> > server,
>> > > I
>> > > > > believe the best way would be to put this script on ASF infra and
>> > have
>> > > > the
>> > > > > "update checker" running in a completely controlled environment.
>> > > > Actually,
>> > > > > I
>> > > > > recall we had this very discussion during the Incubation; I can
>> > > probably
>> > > > > dig
>> > > > > out the corresponding thread.
>> > > > >
>> > > > > Thoughts?
>> > > > >   Cok
>> > > > >
>> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> > > > > > Hey guys
>> > > > > >
>> > > > > > In my opinion, maven.org is still owned by a third party
>> > (Sonatype).
>> > > > We
>> > > > > > don't know what kind of data analysis or intelligence extraction
>> > they
>> > > > > run.
>> > > > > >
>> > > > > > If Ignite servers all over the world were hitting maven.org
>> > > > periodically
>> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a clear
>> > > > indication
>> > > > > > of who is running an Ignite server.
>> > > > > >
>> > > > > > They could reverse lookup the IP address and find out what
>> > > corporation
>> > > > it
>> > > > > > is.
>> > > > > >
>> > > > > > How about having Ignite check the ASF Git directly?
>> > > > > >
>> > > > > > We could use the Git ssh API, but that would require a new
>> > > dependency,
>> > > > > > which I advise against.
>> > > > > >
>> > > > > > Alternatively, we could have it scrape this HTML for new Git
>> tags:
>> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
>> > > > > >
>> > > > > > Another option is to place a txt file in the root of the master
>> > > branch
>> > > > > (e.g
>> > > > > > LATEST), containing a list of the latest GA versions for each
>> major
>> > > > > version
>> > > > > > line (1.x, 2.x).
>> > > > > >
>> > > > > > I would advocate this last option, but it requires somebody
>> > > remembering
>> > > > > to
>> > > > > > update the file with every release, unless we automate it with a
>> > > Maven
>> > > > > > plugin.
>> > > > > >
>> > > > > > Hope that helps!
>> > > > > >
>> > > > > >
>> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
>> > > > > >
>> > > > > > I see nothing wrong with this approach.
>> > > > > >
>> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it
>> good
>> > > to
>> > > > > go?
>> > > > > >
>> > > > > > —
>> > > > > > Denis
>> > > > > >
>> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
>> > > > dsetrakyan@apache.org
>> > > > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Is everyone OK with this approach? Should I file a ticket on
>> it?
>> > > > > > >
>> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> > > > > dsetrakyan@apache.org>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Igniters,
>> > > > > > >>
>> > > > > > >> There has been lots of talk of proposals about various usage
>> > > metrics
>> > > > > for
>> > > > > > >> Ignite and nothing came of it. I would like to resurrect the
>> > topic
>> > > > and
>> > > > > > >> propose something very simple and non-intrusive.
>> > > > > > >>
>> > > > > > >> 1. Update Checker
>> > > > > > >> The main purpose of the update checker is not to collect
>> > metrics,
>> > > > but
>> > > > > to
>> > > > > > >> notify users about a new version of Ignite by accessing
>> > maven.org
>> > > > and
>> > > > > > >> getting the version out of the metadata file:
>> > > > > > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
>> > > > > > >> maven-metadata.xml
>> > > > > > >>
>> > > > > > >> This way we do not send any information anywhere and, at the
>> > same
>> > > > > time,
>> > > > > > >> urge our users to download and start using the latest
>> version of
>> > > > > Ignite.
>> > > > > > >>
>> > > > > > >> 2. Startup Counter
>> > > > > > >> This piece is optional, but we can also get an insight in how
>> > many
>> > > > > times
>> > > > > > a
>> > > > > > >> certain Ignite release gets started. This is just a cool
>> metric
>> > > for
>> > > > > the
>> > > > > > >> community to gauge the project popularity. You can think of
>> it
>> > as
>> > > > of a
>> > > > > > page
>> > > > > > >> visit counter shown on many websites. We can even decide to
>> > > display
>> > > > > this
>> > > > > > >> counter on the Ignite website as well.
>> > > > > > >>
>> > > > > > >> To do this, we can simply add a JAR in maven for every
>> release,
>> > > e.g.
>> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
>> Every
>> > > time
>> > > > > an
>> > > > > > >> Ignite node starts, it will download this JAR in the
>> background.
>> > > > Then
>> > > > > we
>> > > > > > >> will be able to view the number of the total downloads for
>> this
>> > > JAR
>> > > > in
>> > > > > > >> Maven Central, which is essentially the number of starts of
>> > Ignite
>> > > > > nodes.
>> > > > > > >>
>> > > > > > >> *Note that neither of the above suggestions require Ignite to
>> > send
>> > > > or
>> > > > > > >> track any user information whatsoever.*
>> > > > > > >>
>> > > > > > >> Please reply suggesting weather you are OK with this
>> approach.
>> > > > > > >>
>> > > > > > >> D.
>> > > > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <ra...@evosent.com>
wrote:

> Yeah, I guess that's doable as well and requires less management effort
> than my suggestion. We could use events [1] to store payload data (e.g. IP,
> version, etc.)


Yes, we could use events or some other similar API provided by GA.


> What the download page CGI developed in? PHP?
>

To be honest, no clue. I guess someone in the community can figure it out:
https://svn.apache.org/repos/asf/ignite/site/trunk/download.html


> However, I'm not sure whether storing this data in a 3rd party (Google) is
> compliant with the ASF policy. I guess it's no biggie, but if there's doubt
> in the PMC, it's better to ask legal@.


I am not sure there is anything to ask about. The whole Ignite website is
GA enabled, and all we are doing is accessing a standard web page from the
Ignite web site. The information gathered from GA is available only to the
Ignite PMC. Frankly, I think legal@ will be very confused by this question.

Even ASF website itself uses GA:
https://www.apache.org/foundation/policies/privacy.html


> I think Cos said it's OK; maybe Roman can pitch in.
>

 Sure, would be nice to hear from Roman as well.


> Cheers.
>
> [1]
> https://developers.google.com/analytics/devguides/
> collection/analyticsjs/events
>
> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > Raul,
> >
> > Could point about Javascript, it will not work because it executes in the
> > browser. This means we need a server-side script, like CGI we are using
> on
> > our download page.
> >
> > How about this approach. We create something like ignite-version.cgi
> script
> > which will invoke a call to GA and then return the latest version.
> >
> > This should work, right?
> >
> > D.
> >
> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <ra...@evosent.com>
> > wrote:
> >
> > > Hey Dmitriy and all
> > >
> > > Also, since we have GA enabled for the website, we can track how many
> > times
> > > > this page was accessed, which will be equal to the number of starts.
> > This
> > > > way, the counter information is tracked and monitored by the Ignite
> > PMC.
> > >
> > >
> > > Unfortunately this won't work because GA is loaded via Javascript on
> the
> > > browser, so Google will never receive the page hit.
> > >
> > > Given the constraints, the most viable solution is an HTTPS endpoint
> > > running on ASF infra that Ignite invokes via a GET or POST request. The
> > > simplest thing is to write each request in a log file, along with the
> > > timestamp, the version reported by the client, maybe the IP (not sure
> > about
> > > the ASF rules about this concerning privacy, I guess it's OK if you
> make
> > it
> > > an opt-in) and a unique node identifier, i.e. a UUID the node creates
> on
> > > first startup or something.
> > >
> > > This endpoint would need some basic DDoS protection and blacklisting to
> > > prevent data spoofing.
> > >
> > > If we'll be implementing this endpoint anyway, then there's no point
> > > placing another file on Git or elsewhere for reporting the latest
> > versions:
> > > the endpoint itself can return them.
> > >
> > > WDYT?
> > >
> > > Cheers.
> > >
> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > Cos, Raul,
> > > >
> > > > Thanks for the feedback. I completely agree about Maven Central
> being a
> > > 3rd
> > > > party repo (did not think about that initially). All your suggestions
> > > make
> > > > sense, but I would like to keep it as simple as possible, and so far
> > > > everything suggested required GIT dependencies and extra work.
> > > >
> > > > How about Yakov's suggestion. We simply add a page to the Ignite
> > website
> > > > which will have only the latest version. Every time a node starts, it
> > > > receives the latest version from the page and suggests that users
> > upgrade
> > > > if needed.
> > > >
> > > > Also, since we have GA enabled for the website, we can track how many
> > > times
> > > > this page was accessed, which will be equal to the number of starts.
> > This
> > > > way, the counter information is tracked and monitored by the Ignite
> > PMC.
> > > >
> > > > This approach looks pretty innocent to me and everything is kept and
> > > > managed within Apache.
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > > >
> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <cos@apache.org
> >
> > > > wrote:
> > > >
> > > > > I agree with Raul.
> > > > > - providing a ping-back address to a 3rd party might be frown upon
> by
> > > > some.
> > > > >   And might have a consequences like stats collection about users'
> > > > >   infrastructure.
> > > > > - checking an ASF git-repo is easy and won't download any binary
> > data:
> > > > >   everything is clear text and could be easily monitored by any of
> > the
> > > > > network
> > > > >   diagnostic tools, shall it be required. But it involves a bit of
> > the
> > > > > release
> > > > >   discipline.
> > > > > - the binary data download in the runtime is my main concern. This
> is
> > > the
> > > > >   vector of MMA. What if someone gains the control over the
> > repository
> > > > and
> > > > >   replaces the file with some malicious content.
> > > > >
> > > > > As for the particular mechanism: IIRC Ignite used to make a call to
> > an
> > > > > external script to check upon the atest software version available
> > for
> > > > > download. In the past, the endpoint was running on a 3rd party
> > server,
> > > I
> > > > > believe the best way would be to put this script on ASF infra and
> > have
> > > > the
> > > > > "update checker" running in a completely controlled environment.
> > > > Actually,
> > > > > I
> > > > > recall we had this very discussion during the Incubation; I can
> > > probably
> > > > > dig
> > > > > out the corresponding thread.
> > > > >
> > > > > Thoughts?
> > > > >   Cok
> > > > >
> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > > > > Hey guys
> > > > > >
> > > > > > In my opinion, maven.org is still owned by a third party
> > (Sonatype).
> > > > We
> > > > > > don't know what kind of data analysis or intelligence extraction
> > they
> > > > > run.
> > > > > >
> > > > > > If Ignite servers all over the world were hitting maven.org
> > > > periodically
> > > > > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> > > > indication
> > > > > > of who is running an Ignite server.
> > > > > >
> > > > > > They could reverse lookup the IP address and find out what
> > > corporation
> > > > it
> > > > > > is.
> > > > > >
> > > > > > How about having Ignite check the ASF Git directly?
> > > > > >
> > > > > > We could use the Git ssh API, but that would require a new
> > > dependency,
> > > > > > which I advise against.
> > > > > >
> > > > > > Alternatively, we could have it scrape this HTML for new Git
> tags:
> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > > > > >
> > > > > > Another option is to place a txt file in the root of the master
> > > branch
> > > > > (e.g
> > > > > > LATEST), containing a list of the latest GA versions for each
> major
> > > > > version
> > > > > > line (1.x, 2.x).
> > > > > >
> > > > > > I would advocate this last option, but it requires somebody
> > > remembering
> > > > > to
> > > > > > update the file with every release, unless we automate it with a
> > > Maven
> > > > > > plugin.
> > > > > >
> > > > > > Hope that helps!
> > > > > >
> > > > > >
> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
> > > > > >
> > > > > > I see nothing wrong with this approach.
> > > > > >
> > > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it
> good
> > > to
> > > > > go?
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Is everyone OK with this approach? Should I file a ticket on
> it?
> > > > > > >
> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > > > > dsetrakyan@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Igniters,
> > > > > > >>
> > > > > > >> There has been lots of talk of proposals about various usage
> > > metrics
> > > > > for
> > > > > > >> Ignite and nothing came of it. I would like to resurrect the
> > topic
> > > > and
> > > > > > >> propose something very simple and non-intrusive.
> > > > > > >>
> > > > > > >> 1. Update Checker
> > > > > > >> The main purpose of the update checker is not to collect
> > metrics,
> > > > but
> > > > > to
> > > > > > >> notify users about a new version of Ignite by accessing
> > maven.org
> > > > and
> > > > > > >> getting the version out of the metadata file:
> > > > > > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> > > > > > >> maven-metadata.xml
> > > > > > >>
> > > > > > >> This way we do not send any information anywhere and, at the
> > same
> > > > > time,
> > > > > > >> urge our users to download and start using the latest version
> of
> > > > > Ignite.
> > > > > > >>
> > > > > > >> 2. Startup Counter
> > > > > > >> This piece is optional, but we can also get an insight in how
> > many
> > > > > times
> > > > > > a
> > > > > > >> certain Ignite release gets started. This is just a cool
> metric
> > > for
> > > > > the
> > > > > > >> community to gauge the project popularity. You can think of it
> > as
> > > > of a
> > > > > > page
> > > > > > >> visit counter shown on many websites. We can even decide to
> > > display
> > > > > this
> > > > > > >> counter on the Ignite website as well.
> > > > > > >>
> > > > > > >> To do this, we can simply add a JAR in maven for every
> release,
> > > e.g.
> > > > > > >> ignite-start-counter.jar, which will contain only 1 byte.
> Every
> > > time
> > > > > an
> > > > > > >> Ignite node starts, it will download this JAR in the
> background.
> > > > Then
> > > > > we
> > > > > > >> will be able to view the number of the total downloads for
> this
> > > JAR
> > > > in
> > > > > > >> Maven Central, which is essentially the number of starts of
> > Ignite
> > > > > nodes.
> > > > > > >>
> > > > > > >> *Note that neither of the above suggestions require Ignite to
> > send
> > > > or
> > > > > > >> track any user information whatsoever.*
> > > > > > >>
> > > > > > >> Please reply suggesting weather you are OK with this approach.
> > > > > > >>
> > > > > > >> D.
> > > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Raúl Kripalani <ra...@evosent.com>.

Yeah, I guess that's doable as well and requires less management effort
than my suggestion. We could use events [1] to store payload data (e.g. IP,
version, etc.) What the download page CGI developed in? PHP?

However, I'm not sure whether storing this data in a 3rd party (Google) is
compliant with the ASF policy. I guess it's no biggie, but if there's doubt
in the PMC, it's better to ask legal@. I think Cos said it's OK; maybe
Roman can pitch in.

Cheers.

[1]
https://developers.google.com/analytics/devguides/collection/analyticsjs/events

On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Raul,
>
> Could point about Javascript, it will not work because it executes in the
> browser. This means we need a server-side script, like CGI we are using on
> our download page.
>
> How about this approach. We create something like ignite-version.cgi script
> which will invoke a call to GA and then return the latest version.
>
> This should work, right?
>
> D.
>
> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <ra...@evosent.com>
> wrote:
>
> > Hey Dmitriy and all
> >
> > Also, since we have GA enabled for the website, we can track how many
> times
> > > this page was accessed, which will be equal to the number of starts.
> This
> > > way, the counter information is tracked and monitored by the Ignite
> PMC.
> >
> >
> > Unfortunately this won't work because GA is loaded via Javascript on the
> > browser, so Google will never receive the page hit.
> >
> > Given the constraints, the most viable solution is an HTTPS endpoint
> > running on ASF infra that Ignite invokes via a GET or POST request. The
> > simplest thing is to write each request in a log file, along with the
> > timestamp, the version reported by the client, maybe the IP (not sure
> about
> > the ASF rules about this concerning privacy, I guess it's OK if you make
> it
> > an opt-in) and a unique node identifier, i.e. a UUID the node creates on
> > first startup or something.
> >
> > This endpoint would need some basic DDoS protection and blacklisting to
> > prevent data spoofing.
> >
> > If we'll be implementing this endpoint anyway, then there's no point
> > placing another file on Git or elsewhere for reporting the latest
> versions:
> > the endpoint itself can return them.
> >
> > WDYT?
> >
> > Cheers.
> >
> > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Cos, Raul,
> > >
> > > Thanks for the feedback. I completely agree about Maven Central being a
> > 3rd
> > > party repo (did not think about that initially). All your suggestions
> > make
> > > sense, but I would like to keep it as simple as possible, and so far
> > > everything suggested required GIT dependencies and extra work.
> > >
> > > How about Yakov's suggestion. We simply add a page to the Ignite
> website
> > > which will have only the latest version. Every time a node starts, it
> > > receives the latest version from the page and suggests that users
> upgrade
> > > if needed.
> > >
> > > Also, since we have GA enabled for the website, we can track how many
> > times
> > > this page was accessed, which will be equal to the number of starts.
> This
> > > way, the counter information is tracked and monitored by the Ignite
> PMC.
> > >
> > > This approach looks pretty innocent to me and everything is kept and
> > > managed within Apache.
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> > >
> > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <co...@apache.org>
> > > wrote:
> > >
> > > > I agree with Raul.
> > > > - providing a ping-back address to a 3rd party might be frown upon by
> > > some.
> > > >   And might have a consequences like stats collection about users'
> > > >   infrastructure.
> > > > - checking an ASF git-repo is easy and won't download any binary
> data:
> > > >   everything is clear text and could be easily monitored by any of
> the
> > > > network
> > > >   diagnostic tools, shall it be required. But it involves a bit of
> the
> > > > release
> > > >   discipline.
> > > > - the binary data download in the runtime is my main concern. This is
> > the
> > > >   vector of MMA. What if someone gains the control over the
> repository
> > > and
> > > >   replaces the file with some malicious content.
> > > >
> > > > As for the particular mechanism: IIRC Ignite used to make a call to
> an
> > > > external script to check upon the atest software version available
> for
> > > > download. In the past, the endpoint was running on a 3rd party
> server,
> > I
> > > > believe the best way would be to put this script on ASF infra and
> have
> > > the
> > > > "update checker" running in a completely controlled environment.
> > > Actually,
> > > > I
> > > > recall we had this very discussion during the Incubation; I can
> > probably
> > > > dig
> > > > out the corresponding thread.
> > > >
> > > > Thoughts?
> > > >   Cok
> > > >
> > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > > > Hey guys
> > > > >
> > > > > In my opinion, maven.org is still owned by a third party
> (Sonatype).
> > > We
> > > > > don't know what kind of data analysis or intelligence extraction
> they
> > > > run.
> > > > >
> > > > > If Ignite servers all over the world were hitting maven.org
> > > periodically
> > > > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> > > indication
> > > > > of who is running an Ignite server.
> > > > >
> > > > > They could reverse lookup the IP address and find out what
> > corporation
> > > it
> > > > > is.
> > > > >
> > > > > How about having Ignite check the ASF Git directly?
> > > > >
> > > > > We could use the Git ssh API, but that would require a new
> > dependency,
> > > > > which I advise against.
> > > > >
> > > > > Alternatively, we could have it scrape this HTML for new Git tags:
> > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > > > >
> > > > > Another option is to place a txt file in the root of the master
> > branch
> > > > (e.g
> > > > > LATEST), containing a list of the latest GA versions for each major
> > > > version
> > > > > line (1.x, 2.x).
> > > > >
> > > > > I would advocate this last option, but it requires somebody
> > remembering
> > > > to
> > > > > update the file with every release, unless we automate it with a
> > Maven
> > > > > plugin.
> > > > >
> > > > > Hope that helps!
> > > > >
> > > > >
> > > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
> > > > >
> > > > > I see nothing wrong with this approach.
> > > > >
> > > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good
> > to
> > > > go?
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Is everyone OK with this approach? Should I file a ticket on it?
> > > > > >
> > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > >> Igniters,
> > > > > >>
> > > > > >> There has been lots of talk of proposals about various usage
> > metrics
> > > > for
> > > > > >> Ignite and nothing came of it. I would like to resurrect the
> topic
> > > and
> > > > > >> propose something very simple and non-intrusive.
> > > > > >>
> > > > > >> 1. Update Checker
> > > > > >> The main purpose of the update checker is not to collect
> metrics,
> > > but
> > > > to
> > > > > >> notify users about a new version of Ignite by accessing
> maven.org
> > > and
> > > > > >> getting the version out of the metadata file:
> > > > > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> > > > > >> maven-metadata.xml
> > > > > >>
> > > > > >> This way we do not send any information anywhere and, at the
> same
> > > > time,
> > > > > >> urge our users to download and start using the latest version of
> > > > Ignite.
> > > > > >>
> > > > > >> 2. Startup Counter
> > > > > >> This piece is optional, but we can also get an insight in how
> many
> > > > times
> > > > > a
> > > > > >> certain Ignite release gets started. This is just a cool metric
> > for
> > > > the
> > > > > >> community to gauge the project popularity. You can think of it
> as
> > > of a
> > > > > page
> > > > > >> visit counter shown on many websites. We can even decide to
> > display
> > > > this
> > > > > >> counter on the Ignite website as well.
> > > > > >>
> > > > > >> To do this, we can simply add a JAR in maven for every release,
> > e.g.
> > > > > >> ignite-start-counter.jar, which will contain only 1 byte. Every
> > time
> > > > an
> > > > > >> Ignite node starts, it will download this JAR in the background.
> > > Then
> > > > we
> > > > > >> will be able to view the number of the total downloads for this
> > JAR
> > > in
> > > > > >> Maven Central, which is essentially the number of starts of
> Ignite
> > > > nodes.
> > > > > >>
> > > > > >> *Note that neither of the above suggestions require Ignite to
> send
> > > or
> > > > > >> track any user information whatsoever.*
> > > > > >>
> > > > > >> Please reply suggesting weather you are OK with this approach.
> > > > > >>
> > > > > >> D.
> > > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

Raul,

Could point about Javascript, it will not work because it executes in the
browser. This means we need a server-side script, like CGI we are using on
our download page.

How about this approach. We create something like ignite-version.cgi script
which will invoke a call to GA and then return the latest version.

This should work, right?

D.

On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <ra...@evosent.com>
wrote:

> Hey Dmitriy and all
>
> Also, since we have GA enabled for the website, we can track how many times
> > this page was accessed, which will be equal to the number of starts. This
> > way, the counter information is tracked and monitored by the Ignite PMC.
>
>
> Unfortunately this won't work because GA is loaded via Javascript on the
> browser, so Google will never receive the page hit.
>
> Given the constraints, the most viable solution is an HTTPS endpoint
> running on ASF infra that Ignite invokes via a GET or POST request. The
> simplest thing is to write each request in a log file, along with the
> timestamp, the version reported by the client, maybe the IP (not sure about
> the ASF rules about this concerning privacy, I guess it's OK if you make it
> an opt-in) and a unique node identifier, i.e. a UUID the node creates on
> first startup or something.
>
> This endpoint would need some basic DDoS protection and blacklisting to
> prevent data spoofing.
>
> If we'll be implementing this endpoint anyway, then there's no point
> placing another file on Git or elsewhere for reporting the latest versions:
> the endpoint itself can return them.
>
> WDYT?
>
> Cheers.
>
> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Cos, Raul,
> >
> > Thanks for the feedback. I completely agree about Maven Central being a
> 3rd
> > party repo (did not think about that initially). All your suggestions
> make
> > sense, but I would like to keep it as simple as possible, and so far
> > everything suggested required GIT dependencies and extra work.
> >
> > How about Yakov's suggestion. We simply add a page to the Ignite website
> > which will have only the latest version. Every time a node starts, it
> > receives the latest version from the page and suggests that users upgrade
> > if needed.
> >
> > Also, since we have GA enabled for the website, we can track how many
> times
> > this page was accessed, which will be equal to the number of starts. This
> > way, the counter information is tracked and monitored by the Ignite PMC.
> >
> > This approach looks pretty innocent to me and everything is kept and
> > managed within Apache.
> >
> > Thoughts?
> >
> > D.
> >
> >
> > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> >
> > > I agree with Raul.
> > > - providing a ping-back address to a 3rd party might be frown upon by
> > some.
> > >   And might have a consequences like stats collection about users'
> > >   infrastructure.
> > > - checking an ASF git-repo is easy and won't download any binary data:
> > >   everything is clear text and could be easily monitored by any of the
> > > network
> > >   diagnostic tools, shall it be required. But it involves a bit of the
> > > release
> > >   discipline.
> > > - the binary data download in the runtime is my main concern. This is
> the
> > >   vector of MMA. What if someone gains the control over the repository
> > and
> > >   replaces the file with some malicious content.
> > >
> > > As for the particular mechanism: IIRC Ignite used to make a call to an
> > > external script to check upon the atest software version available for
> > > download. In the past, the endpoint was running on a 3rd party server,
> I
> > > believe the best way would be to put this script on ASF infra and have
> > the
> > > "update checker" running in a completely controlled environment.
> > Actually,
> > > I
> > > recall we had this very discussion during the Incubation; I can
> probably
> > > dig
> > > out the corresponding thread.
> > >
> > > Thoughts?
> > >   Cok
> > >
> > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > > Hey guys
> > > >
> > > > In my opinion, maven.org is still owned by a third party (Sonatype).
> > We
> > > > don't know what kind of data analysis or intelligence extraction they
> > > run.
> > > >
> > > > If Ignite servers all over the world were hitting maven.org
> > periodically
> > > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> > indication
> > > > of who is running an Ignite server.
> > > >
> > > > They could reverse lookup the IP address and find out what
> corporation
> > it
> > > > is.
> > > >
> > > > How about having Ignite check the ASF Git directly?
> > > >
> > > > We could use the Git ssh API, but that would require a new
> dependency,
> > > > which I advise against.
> > > >
> > > > Alternatively, we could have it scrape this HTML for new Git tags:
> > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > > >
> > > > Another option is to place a txt file in the root of the master
> branch
> > > (e.g
> > > > LATEST), containing a list of the latest GA versions for each major
> > > version
> > > > line (1.x, 2.x).
> > > >
> > > > I would advocate this last option, but it requires somebody
> remembering
> > > to
> > > > update the file with every release, unless we automate it with a
> Maven
> > > > plugin.
> > > >
> > > > Hope that helps!
> > > >
> > > >
> > > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
> > > >
> > > > I see nothing wrong with this approach.
> > > >
> > > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good
> to
> > > go?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > Is everyone OK with this approach? Should I file a ticket on it?
> > > > >
> > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Igniters,
> > > > >>
> > > > >> There has been lots of talk of proposals about various usage
> metrics
> > > for
> > > > >> Ignite and nothing came of it. I would like to resurrect the topic
> > and
> > > > >> propose something very simple and non-intrusive.
> > > > >>
> > > > >> 1. Update Checker
> > > > >> The main purpose of the update checker is not to collect metrics,
> > but
> > > to
> > > > >> notify users about a new version of Ignite by accessing maven.org
> > and
> > > > >> getting the version out of the metadata file:
> > > > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> > > > >> maven-metadata.xml
> > > > >>
> > > > >> This way we do not send any information anywhere and, at the same
> > > time,
> > > > >> urge our users to download and start using the latest version of
> > > Ignite.
> > > > >>
> > > > >> 2. Startup Counter
> > > > >> This piece is optional, but we can also get an insight in how many
> > > times
> > > > a
> > > > >> certain Ignite release gets started. This is just a cool metric
> for
> > > the
> > > > >> community to gauge the project popularity. You can think of it as
> > of a
> > > > page
> > > > >> visit counter shown on many websites. We can even decide to
> display
> > > this
> > > > >> counter on the Ignite website as well.
> > > > >>
> > > > >> To do this, we can simply add a JAR in maven for every release,
> e.g.
> > > > >> ignite-start-counter.jar, which will contain only 1 byte. Every
> time
> > > an
> > > > >> Ignite node starts, it will download this JAR in the background.
> > Then
> > > we
> > > > >> will be able to view the number of the total downloads for this
> JAR
> > in
> > > > >> Maven Central, which is essentially the number of starts of Ignite
> > > nodes.
> > > > >>
> > > > >> *Note that neither of the above suggestions require Ignite to send
> > or
> > > > >> track any user information whatsoever.*
> > > > >>
> > > > >> Please reply suggesting weather you are OK with this approach.
> > > > >>
> > > > >> D.
> > > > >>
> > >
> > >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Raúl Kripalani <ra...@evosent.com>.

Hey Dmitriy and all

Also, since we have GA enabled for the website, we can track how many times
> this page was accessed, which will be equal to the number of starts. This
> way, the counter information is tracked and monitored by the Ignite PMC.


Unfortunately this won't work because GA is loaded via Javascript on the
browser, so Google will never receive the page hit.

Given the constraints, the most viable solution is an HTTPS endpoint
running on ASF infra that Ignite invokes via a GET or POST request. The
simplest thing is to write each request in a log file, along with the
timestamp, the version reported by the client, maybe the IP (not sure about
the ASF rules about this concerning privacy, I guess it's OK if you make it
an opt-in) and a unique node identifier, i.e. a UUID the node creates on
first startup or something.

This endpoint would need some basic DDoS protection and blacklisting to
prevent data spoofing.

If we'll be implementing this endpoint anyway, then there's no point
placing another file on Git or elsewhere for reporting the latest versions:
the endpoint itself can return them.

WDYT?

Cheers.

On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Cos, Raul,
>
> Thanks for the feedback. I completely agree about Maven Central being a 3rd
> party repo (did not think about that initially). All your suggestions make
> sense, but I would like to keep it as simple as possible, and so far
> everything suggested required GIT dependencies and extra work.
>
> How about Yakov's suggestion. We simply add a page to the Ignite website
> which will have only the latest version. Every time a node starts, it
> receives the latest version from the page and suggests that users upgrade
> if needed.
>
> Also, since we have GA enabled for the website, we can track how many times
> this page was accessed, which will be equal to the number of starts. This
> way, the counter information is tracked and monitored by the Ignite PMC.
>
> This approach looks pretty innocent to me and everything is kept and
> managed within Apache.
>
> Thoughts?
>
> D.
>
>
> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
>
> > I agree with Raul.
> > - providing a ping-back address to a 3rd party might be frown upon by
> some.
> >   And might have a consequences like stats collection about users'
> >   infrastructure.
> > - checking an ASF git-repo is easy and won't download any binary data:
> >   everything is clear text and could be easily monitored by any of the
> > network
> >   diagnostic tools, shall it be required. But it involves a bit of the
> > release
> >   discipline.
> > - the binary data download in the runtime is my main concern. This is the
> >   vector of MMA. What if someone gains the control over the repository
> and
> >   replaces the file with some malicious content.
> >
> > As for the particular mechanism: IIRC Ignite used to make a call to an
> > external script to check upon the atest software version available for
> > download. In the past, the endpoint was running on a 3rd party server, I
> > believe the best way would be to put this script on ASF infra and have
> the
> > "update checker" running in a completely controlled environment.
> Actually,
> > I
> > recall we had this very discussion during the Incubation; I can probably
> > dig
> > out the corresponding thread.
> >
> > Thoughts?
> >   Cok
> >
> > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > Hey guys
> > >
> > > In my opinion, maven.org is still owned by a third party (Sonatype).
> We
> > > don't know what kind of data analysis or intelligence extraction they
> > run.
> > >
> > > If Ignite servers all over the world were hitting maven.org
> periodically
> > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> indication
> > > of who is running an Ignite server.
> > >
> > > They could reverse lookup the IP address and find out what corporation
> it
> > > is.
> > >
> > > How about having Ignite check the ASF Git directly?
> > >
> > > We could use the Git ssh API, but that would require a new dependency,
> > > which I advise against.
> > >
> > > Alternatively, we could have it scrape this HTML for new Git tags:
> > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >
> > > Another option is to place a txt file in the root of the master branch
> > (e.g
> > > LATEST), containing a list of the latest GA versions for each major
> > version
> > > line (1.x, 2.x).
> > >
> > > I would advocate this last option, but it requires somebody remembering
> > to
> > > update the file with every release, unless we automate it with a Maven
> > > plugin.
> > >
> > > Hope that helps!
> > >
> > >
> > > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
> > >
> > > I see nothing wrong with this approach.
> > >
> > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to
> > go?
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >
> > > wrote:
> > > >
> > > > Is everyone OK with this approach? Should I file a ticket on it?
> > > >
> > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > > wrote:
> > > >
> > > >> Igniters,
> > > >>
> > > >> There has been lots of talk of proposals about various usage metrics
> > for
> > > >> Ignite and nothing came of it. I would like to resurrect the topic
> and
> > > >> propose something very simple and non-intrusive.
> > > >>
> > > >> 1. Update Checker
> > > >> The main purpose of the update checker is not to collect metrics,
> but
> > to
> > > >> notify users about a new version of Ignite by accessing maven.org
> and
> > > >> getting the version out of the metadata file:
> > > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> > > >> maven-metadata.xml
> > > >>
> > > >> This way we do not send any information anywhere and, at the same
> > time,
> > > >> urge our users to download and start using the latest version of
> > Ignite.
> > > >>
> > > >> 2. Startup Counter
> > > >> This piece is optional, but we can also get an insight in how many
> > times
> > > a
> > > >> certain Ignite release gets started. This is just a cool metric for
> > the
> > > >> community to gauge the project popularity. You can think of it as
> of a
> > > page
> > > >> visit counter shown on many websites. We can even decide to display
> > this
> > > >> counter on the Ignite website as well.
> > > >>
> > > >> To do this, we can simply add a JAR in maven for every release, e.g.
> > > >> ignite-start-counter.jar, which will contain only 1 byte. Every time
> > an
> > > >> Ignite node starts, it will download this JAR in the background.
> Then
> > we
> > > >> will be able to view the number of the total downloads for this JAR
> in
> > > >> Maven Central, which is essentially the number of starts of Ignite
> > nodes.
> > > >>
> > > >> *Note that neither of the above suggestions require Ignite to send
> or
> > > >> track any user information whatsoever.*
> > > >>
> > > >> Please reply suggesting weather you are OK with this approach.
> > > >>
> > > >> D.
> > > >>
> >
> >
>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

Cos, Raul,

Thanks for the feedback. I completely agree about Maven Central being a 3rd
party repo (did not think about that initially). All your suggestions make
sense, but I would like to keep it as simple as possible, and so far
everything suggested required GIT dependencies and extra work.

How about Yakov's suggestion. We simply add a page to the Ignite website
which will have only the latest version. Every time a node starts, it
receives the latest version from the page and suggests that users upgrade
if needed.

Also, since we have GA enabled for the website, we can track how many times
this page was accessed, which will be equal to the number of starts. This
way, the counter information is tracked and monitored by the Ignite PMC.

This approach looks pretty innocent to me and everything is kept and
managed within Apache.

Thoughts?

D.


On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <co...@apache.org> wrote:

> I agree with Raul.
> - providing a ping-back address to a 3rd party might be frown upon by some.
>   And might have a consequences like stats collection about users'
>   infrastructure.
> - checking an ASF git-repo is easy and won't download any binary data:
>   everything is clear text and could be easily monitored by any of the
> network
>   diagnostic tools, shall it be required. But it involves a bit of the
> release
>   discipline.
> - the binary data download in the runtime is my main concern. This is the
>   vector of MMA. What if someone gains the control over the repository and
>   replaces the file with some malicious content.
>
> As for the particular mechanism: IIRC Ignite used to make a call to an
> external script to check upon the atest software version available for
> download. In the past, the endpoint was running on a 3rd party server, I
> believe the best way would be to put this script on ASF infra and have the
> "update checker" running in a completely controlled environment. Actually,
> I
> recall we had this very discussion during the Incubation; I can probably
> dig
> out the corresponding thread.
>
> Thoughts?
>   Cok
>
> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > Hey guys
> >
> > In my opinion, maven.org is still owned by a third party (Sonatype). We
> > don't know what kind of data analysis or intelligence extraction they
> run.
> >
> > If Ignite servers all over the world were hitting maven.org periodically
> > asking for an Ignite Maven artifact, it gives Sonatype a clear indication
> > of who is running an Ignite server.
> >
> > They could reverse lookup the IP address and find out what corporation it
> > is.
> >
> > How about having Ignite check the ASF Git directly?
> >
> > We could use the Git ssh API, but that would require a new dependency,
> > which I advise against.
> >
> > Alternatively, we could have it scrape this HTML for new Git tags:
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >
> > Another option is to place a txt file in the root of the master branch
> (e.g
> > LATEST), containing a list of the latest GA versions for each major
> version
> > line (1.x, 2.x).
> >
> > I would advocate this last option, but it requires somebody remembering
> to
> > update the file with every release, unless we automate it with a Maven
> > plugin.
> >
> > Hope that helps!
> >
> >
> > On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
> >
> > I see nothing wrong with this approach.
> >
> > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to
> go?
> >
> > —
> > Denis
> >
> > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> > wrote:
> > >
> > > Is everyone OK with this approach? Should I file a ticket on it?
> > >
> > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > > wrote:
> > >
> > >> Igniters,
> > >>
> > >> There has been lots of talk of proposals about various usage metrics
> for
> > >> Ignite and nothing came of it. I would like to resurrect the topic and
> > >> propose something very simple and non-intrusive.
> > >>
> > >> 1. Update Checker
> > >> The main purpose of the update checker is not to collect metrics, but
> to
> > >> notify users about a new version of Ignite by accessing maven.org and
> > >> getting the version out of the metadata file:
> > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> > >> maven-metadata.xml
> > >>
> > >> This way we do not send any information anywhere and, at the same
> time,
> > >> urge our users to download and start using the latest version of
> Ignite.
> > >>
> > >> 2. Startup Counter
> > >> This piece is optional, but we can also get an insight in how many
> times
> > a
> > >> certain Ignite release gets started. This is just a cool metric for
> the
> > >> community to gauge the project popularity. You can think of it as of a
> > page
> > >> visit counter shown on many websites. We can even decide to display
> this
> > >> counter on the Ignite website as well.
> > >>
> > >> To do this, we can simply add a JAR in maven for every release, e.g.
> > >> ignite-start-counter.jar, which will contain only 1 byte. Every time
> an
> > >> Ignite node starts, it will download this JAR in the background. Then
> we
> > >> will be able to view the number of the total downloads for this JAR in
> > >> Maven Central, which is essentially the number of starts of Ignite
> nodes.
> > >>
> > >> *Note that neither of the above suggestions require Ignite to send or
> > >> track any user information whatsoever.*
> > >>
> > >> Please reply suggesting weather you are OK with this approach.
> > >>
> > >> D.
> > >>
>
>

Re: [DISCUSS] Ignite Update Checker

Posted by Konstantin Boudnik <co...@apache.org>.

I agree with Raul.
- providing a ping-back address to a 3rd party might be frown upon by some.
  And might have a consequences like stats collection about users'
  infrastructure.
- checking an ASF git-repo is easy and won't download any binary data:
  everything is clear text and could be easily monitored by any of the network
  diagnostic tools, shall it be required. But it involves a bit of the release
  discipline.
- the binary data download in the runtime is my main concern. This is the
  vector of MMA. What if someone gains the control over the repository and
  replaces the file with some malicious content.

As for the particular mechanism: IIRC Ignite used to make a call to an
external script to check upon the atest software version available for
download. In the past, the endpoint was running on a 3rd party server, I
believe the best way would be to put this script on ASF infra and have the
"update checker" running in a completely controlled environment. Actually, I
recall we had this very discussion during the Incubation; I can probably dig
out the corresponding thread.

Thoughts?
  Cok

On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> Hey guys
> 
> In my opinion, maven.org is still owned by a third party (Sonatype). We
> don't know what kind of data analysis or intelligence extraction they run.
> 
> If Ignite servers all over the world were hitting maven.org periodically
> asking for an Ignite Maven artifact, it gives Sonatype a clear indication
> of who is running an Ignite server.
> 
> They could reverse lookup the IP address and find out what corporation it
> is.
> 
> How about having Ignite check the ASF Git directly?
> 
> We could use the Git ssh API, but that would require a new dependency,
> which I advise against.
> 
> Alternatively, we could have it scrape this HTML for new Git tags:
> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> 
> Another option is to place a txt file in the root of the master branch (e.g
> LATEST), containing a list of the latest GA versions for each major version
> line (1.x, 2.x).
> 
> I would advocate this last option, but it requires somebody remembering to
> update the file with every release, unless we automate it with a Maven
> plugin.
> 
> Hope that helps!
> 
> 
> On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:
> 
> I see nothing wrong with this approach.
> 
> Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to go?
> 
> —
> Denis
> 
> > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > Is everyone OK with this approach? Should I file a ticket on it?
> >
> > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <ds...@apache.org>
> > wrote:
> >
> >> Igniters,
> >>
> >> There has been lots of talk of proposals about various usage metrics for
> >> Ignite and nothing came of it. I would like to resurrect the topic and
> >> propose something very simple and non-intrusive.
> >>
> >> 1. Update Checker
> >> The main purpose of the update checker is not to collect metrics, but to
> >> notify users about a new version of Ignite by accessing maven.org and
> >> getting the version out of the metadata file:
> >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> >> maven-metadata.xml
> >>
> >> This way we do not send any information anywhere and, at the same time,
> >> urge our users to download and start using the latest version of Ignite.
> >>
> >> 2. Startup Counter
> >> This piece is optional, but we can also get an insight in how many times
> a
> >> certain Ignite release gets started. This is just a cool metric for the
> >> community to gauge the project popularity. You can think of it as of a
> page
> >> visit counter shown on many websites. We can even decide to display this
> >> counter on the Ignite website as well.
> >>
> >> To do this, we can simply add a JAR in maven for every release, e.g.
> >> ignite-start-counter.jar, which will contain only 1 byte. Every time an
> >> Ignite node starts, it will download this JAR in the background. Then we
> >> will be able to view the number of the total downloads for this JAR in
> >> Maven Central, which is essentially the number of starts of Ignite nodes.
> >>
> >> *Note that neither of the above suggestions require Ignite to send or
> >> track any user information whatsoever.*
> >>
> >> Please reply suggesting weather you are OK with this approach.
> >>
> >> D.
> >>

Re: [DISCUSS] Ignite Update Checker

Posted by Yakov Zhdanov <yz...@apache.org>.

Raul, any ideas on startup counter?

BTW, we have google analytics applied to Ignite site. Can it be used to
count node starts?

--Yakov

Re: [DISCUSS] Ignite Update Checker

Posted by Raul Kripalani <ra...@apache.org>.

Hey guys

In my opinion, maven.org is still owned by a third party (Sonatype). We
don't know what kind of data analysis or intelligence extraction they run.

If Ignite servers all over the world were hitting maven.org periodically
asking for an Ignite Maven artifact, it gives Sonatype a clear indication
of who is running an Ignite server.

They could reverse lookup the IP address and find out what corporation it
is.

How about having Ignite check the ASF Git directly?

We could use the Git ssh API, but that would require a new dependency,
which I advise against.

Alternatively, we could have it scrape this HTML for new Git tags:
https://git-wip-us.apache.org/repos/asf?p=ignite.git

Another option is to place a txt file in the root of the master branch (e.g
LATEST), containing a list of the latest GA versions for each major version
line (1.x, 2.x).

I would advocate this last option, but it requires somebody remembering to
update the file with every release, unless we automate it with a Maven
plugin.

Hope that helps!

On 24 Aug 2017 19:37, "Denis Magda" <dm...@apache.org> wrote:

I see nothing wrong with this approach.

Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to go?

—
Denis

> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
>
> Is everyone OK with this approach? Should I file a ticket on it?
>
> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> Igniters,
>>
>> There has been lots of talk of proposals about various usage metrics for
>> Ignite and nothing came of it. I would like to resurrect the topic and
>> propose something very simple and non-intrusive.
>>
>> 1. Update Checker
>> The main purpose of the update checker is not to collect metrics, but to
>> notify users about a new version of Ignite by accessing maven.org and
>> getting the version out of the metadata file:
>> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
>> maven-metadata.xml
>>
>> This way we do not send any information anywhere and, at the same time,
>> urge our users to download and start using the latest version of Ignite.
>>
>> 2. Startup Counter
>> This piece is optional, but we can also get an insight in how many times
a
>> certain Ignite release gets started. This is just a cool metric for the
>> community to gauge the project popularity. You can think of it as of a
page
>> visit counter shown on many websites. We can even decide to display this
>> counter on the Ignite website as well.
>>
>> To do this, we can simply add a JAR in maven for every release, e.g.
>> ignite-start-counter.jar, which will contain only 1 byte. Every time an
>> Ignite node starts, it will download this JAR in the background. Then we
>> will be able to view the number of the total downloads for this JAR in
>> Maven Central, which is essentially the number of starts of Ignite nodes.
>>
>> *Note that neither of the above suggestions require Ignite to send or
>> track any user information whatsoever.*
>>
>> Please reply suggesting weather you are OK with this approach.
>>
>> D.
>>

Re: [DISCUSS] Ignite Update Checker

Posted by Denis Magda <dm...@apache.org>.

I see nothing wrong with this approach.

Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to go?

—
Denis

> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
> 
> Is everyone OK with this approach? Should I file a ticket on it?
> 
> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> 
>> Igniters,
>> 
>> There has been lots of talk of proposals about various usage metrics for
>> Ignite and nothing came of it. I would like to resurrect the topic and
>> propose something very simple and non-intrusive.
>> 
>> 1. Update Checker
>> The main purpose of the update checker is not to collect metrics, but to
>> notify users about a new version of Ignite by accessing maven.org and
>> getting the version out of the metadata file:
>> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
>> maven-metadata.xml
>> 
>> This way we do not send any information anywhere and, at the same time,
>> urge our users to download and start using the latest version of Ignite.
>> 
>> 2. Startup Counter
>> This piece is optional, but we can also get an insight in how many times a
>> certain Ignite release gets started. This is just a cool metric for the
>> community to gauge the project popularity. You can think of it as of a page
>> visit counter shown on many websites. We can even decide to display this
>> counter on the Ignite website as well.
>> 
>> To do this, we can simply add a JAR in maven for every release, e.g.
>> ignite-start-counter.jar, which will contain only 1 byte. Every time an
>> Ignite node starts, it will download this JAR in the background. Then we
>> will be able to view the number of the total downloads for this JAR in
>> Maven Central, which is essentially the number of starts of Ignite nodes.
>> 
>> *Note that neither of the above suggestions require Ignite to send or
>> track any user information whatsoever.*
>> 
>> Please reply suggesting weather you are OK with this approach.
>> 
>> D.
>>

Re: [DISCUSS] Ignite Update Checker

Posted by Dmitriy Setrakyan <ds...@apache.org>.

Is everyone OK with this approach? Should I file a ticket on it?

On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Igniters,
>
> There has been lots of talk of proposals about various usage metrics for
> Ignite and nothing came of it. I would like to resurrect the topic and
> propose something very simple and non-intrusive.
>
> 1. Update Checker
> The main purpose of the update checker is not to collect metrics, but to
> notify users about a new version of Ignite by accessing maven.org and
> getting the version out of the metadata file:
> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> maven-metadata.xml
>
> This way we do not send any information anywhere and, at the same time,
> urge our users to download and start using the latest version of Ignite.
>
> 2. Startup Counter
> This piece is optional, but we can also get an insight in how many times a
> certain Ignite release gets started. This is just a cool metric for the
> community to gauge the project popularity. You can think of it as of a page
> visit counter shown on many websites. We can even decide to display this
> counter on the Ignite website as well.
>
> To do this, we can simply add a JAR in maven for every release, e.g.
> ignite-start-counter.jar, which will contain only 1 byte. Every time an
> Ignite node starts, it will download this JAR in the background. Then we
> will be able to view the number of the total downloads for this JAR in
> Maven Central, which is essentially the number of starts of Ignite nodes.
>
> *Note that neither of the above suggestions require Ignite to send or
> track any user information whatsoever.*
>
> Please reply suggesting weather you are OK with this approach.
>
> D.
>