You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by Ning Wang <wa...@gmail.com> on 2019/03/08 21:58:37 UTC
[DISCUSS] Hosting Heron binary release packages
Hi,
I have been trying to release Heron 0.20.1 (being distracted time by time
though) and the most recent question I am having is where to put the binary
packages.
The binary packages are (when we were doing releases on github):
- tar.gz packages for osx, centos and ubuntu, each one includes all modules
like core, lib, tools, etc.
- .sh packages for the three platforms. which is an installer for the
modules in the tar.gz packages.
- docker image (dockerhub, not github most of the times)
Currently each package is more than 400MB.
I was trying to understand the Apache rules and my impression was that
these package should be on dist.apache.org like the src packages (I might
be wrong about the rules though) and it looks like Apache Storm has a
binary package in their release.
However it seems Apache infra has a byte limit of 500MB for each release. I
guess it means that the binary packages are not "required" to be on Apache
infra?
The binary packages are convenient for users. So I think they should be
included in release. The question is where should we put them?
So far it looks like the options are:
- ask for an exception and publish them to dist.apache.org. It seems like
Apache infra guys don't suggest this solution.
- publish only src package to dist.apache.org and publish the binary
packages on github (or is there any other suggestion?). This is convenient
for us and there is no problem so far (we have binary packages for all the
previous releases and github hasn't complained). The question about this
option is more like if this is acceptable (or ever better suggested) by
Apache?
What do you think about the two options above and any other options we
should consider?
Thanks.
--ning
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Ning Wang <wa...@gmail.com>.
Thx~ It makes sense.
On Thu, Mar 14, 2019 at 10:41 AM Dave Fisher <da...@comcast.net> wrote:
> Hi -
>
> On Mar 14, 2019, at 10:31 AM, Ning Wang <wa...@gmail.com> wrote:
>
> Thanks Dave! Let me start a vote~
>
> No complain to you guys. :) It seems like Apache infra doesn't have a
> clear guideline about this case. Otherwise it could be a lot more
> straightforward. Hopefully this issue will be improved soon.
>
>
> The ASF has had continual evolution over 20 + years. What Infrastructure
> provides now is way more obvious than in the past. You mentors are here to
> guide you through things.
>
> The Docker capability is new and that is certainly not well understood.
> The best thing to do with Infra if your mentors cannot help is to ask
> asfinfra on Slack (it was hip chat last year) - https://theasf.slack.com
>
>
> I don't think our jars contain LICENSE files, this is required? All
> dependencies are built from source code so hopefully they are ok.
>
>
> The first time is the hardest with binaries and it will take time. The key
> is listing each dependency that gets included and reviewing its license. If
> any are “category X” then they need to be replaced. The Release Policy is
> the responsibility of Legal Affairs.
>
> Here is the reference about which licenses can be included. [1] This is
> frequently updates. If you have any doubts then we ask through a LEGAL jira
> or legal-discuss@apache.org ( a public list)
>
> Regards,
> Dave
>
> [1] http://www.apache.org/legal/resolved.html
>
>
>
>
>
> On Thu, Mar 14, 2019 at 10:24 AM Dave Fisher <da...@comcast.net>
> wrote:
>
>> Hi -
>>
>> Agreed. Let’s just make sure we review how this convenience is shown as
>> some on the IPMC are concerned about how the Github release tab appears.
>>
>> For binary packages we really need to know and carefully review the
>> licensing for all the included dependencies. It is more than just the Jars
>> as a whole but what the builds have included. I apologize for not having
>> enough time to push this part of the review.
>>
>> On the Dev list we can discuss where users might find these prospective
>> binary packages which will have no guarantees about licensing.
>>
>> Let’s do an Official Apache Source Release.
>>
>> Regards,
>> Dave
>>
>> On Mar 14, 2019, at 10:17 AM, Ning Wang <wa...@gmail.com> wrote:
>>
>> I am thinking that we can leave the "convenience community binary
>> packages" in github for this release (because of the file size) and then
>> try to reduce the package size for future releases so that they are more
>> friendly to Apache infrastructure. It has been blocked by this for a while
>> and I am hoping to start the vote sooner than later.
>>
>> Is this ok? @Dave Fisher <da...@comcast.net> @P. Taylor Goetz
>> <pt...@gmail.com> ^^
>>
>>
>>
>> On Tue, Mar 12, 2019 at 5:37 PM Ning Wang <wa...@gmail.com> wrote:
>>
>>> For the content in the packages:
>>>
>>> Currently we have binary packages for three platforms: osx, centos and
>>> ubuntu. The file sizes are fairly close among them. I am going to use
>>> ubuntu as example:
>>>
>>> - there is a tar.gz file and an sh file. I am thinking we may keep the
>>> sh file only since it is easier to use and contains the same data as the
>>> tar.gz file. This can save about 50% in size (from 470 * 6 -> 470 * 3).
>>>
>>> The file size is 470m.
>>> -rw-r--r--@ 1 nwang staff 470508875 Feb 27 10:57
>>> heron-v0.20.1-incubating-rc1-ubuntu14.04.tar.gz
>>>
>>> The package is trim-able I believe, but there isn't many obvious ones:
>>> drwxr-xr-x@ 14 nwang staff 448 Dec 31 1969 bin
>>> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 conf
>>> drwxr-xr-x@ 5 nwang staff 160 Mar 6 15:10 dist
>>> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 examples
>>> drwxr-xr-x@ 11 nwang staff 352 Dec 31 1969 include
>>> drwxr-xr-x@ 12 nwang staff 384 Mar 6 15:04 lib
>>> -r-xr-xr-x@ 1 nwang staff 285 Dec 31 1969 release.yaml
>>>
>>> It is used to install a local environment that contains all necessary
>>> components:
>>> - bin: binaries for tools like api server, tracker, ui, etc
>>> - conf: configurations for different environments
>>> - dist: heron core and its tar.gz file (194m, might be trim-able).
>>> - examples: example topology jars
>>> - include: c++ header files
>>> - lib: c++ and java lib files
>>>
>>> More detailed information is attached at the end of this email.
>>>
>>> /dist/core/core and /lib are the biggest dirs and they might have a few
>>> duplicated big files we can try to get rid of.
>>>
>>> heron-downloader.jar 16m * 2
>>> heron-metricscachemgr.jar 8.8m * 2
>>> heron-binpacking-packing.jar 5.3m * 3
>>> heron-roundrobin-packing.jar 5.3m * 2
>>> scheduler jars 60m * 2
>>> heron-localfs-statemgr.jar 5.3m * 2
>>> heron-zookeeper-statemgr.jar 6.6m * 2
>>>
>>> In case we can get rid of the core.tar.gz and the duplicated files, we
>>> might be able to reduce the size to 480m - 190m - 100m = 190m. Then these
>>> binary packages will be 190 * 3 = 570m, after some works.
>>>
>>>
>>>
>>> heron-v0.20.1-incubating-rc1-ubuntu14.04
>>> ├── [ 448] bin
>>> │ ├── [2.1M] heron
>>> │ ├── [2.1M] heron-admin
>>> │ ├── [1.5K] heron-apiserver
>>> │ ├── [1.5K] heron-apiserver.sh
>>> │ ├── [1.2K] heron-downloader
>>> │ ├── [ 931] heron-downloader-config
>>> │ ├── [ 931] heron-downloader-config.sh
>>> │ ├── [1.2K] heron-downloader.sh
>>> │ ├── [2.2M] heron-explorer
>>> │ ├── [ 42M] heron-nomad
>>> │ ├── [2.5M] heron-tracker
>>> │ └── [3.5M] heron-ui
>>> ├── [ 512] conf
>>> │ ├── [ 384] aurora
>>> │ │ ├── [1.1K] client.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [3.3K] heron.aurora
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [2.7K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [2.3K] statemgr.yaml
>>> │ │ └── [1.1K] uploader.yaml
>>> │ ├── [ 352] examples
>>> │ │ ├── [1.3K] aurora_scheduler.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [1.3K] local_scheduler.yaml
>>> │ │ ├── [1.5K] local_stateful.yaml
>>> │ │ ├── [ 951] localfs_statemgr.yaml
>>> │ │ ├── [1001] localfs_uploader.yaml
>>> │ │ ├── [7.3K] metrics_sinks.yaml
>>> │ │ └── [ 800] roundrobin_packing.yaml
>>> │ ├── [2.3K] heron_tracker.yaml
>>> │ ├── [ 352] kubernetes
>>> │ │ ├── [ 979] client.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.3K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [1.5K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.7K] statemgr.yaml
>>> │ │ └── [1.1K] uploader.yaml
>>> │ ├── [ 384] local
>>> │ │ ├── [ 984] client.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [2.2K] healthmgr.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [1.3K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.3K] statemgr.yaml
>>> │ │ └── [1.2K] uploader.yaml
>>> │ ├── [ 352] localzk
>>> │ │ ├── [ 915] client.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [1.3K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [2.5K] statemgr.yaml
>>> │ │ └── [1.2K] uploader.yaml
>>> │ ├── [ 320] marathon
>>> │ │ ├── [1.1K] client.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [1.5K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.3K] statemgr.yaml
>>> │ │ └── [1.1K] uploader.yaml
>>> │ ├── [ 320] mesos
>>> │ │ ├── [ 800] client.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [2.0K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.3K] statemgr.yaml
>>> │ │ └── [1.2K] uploader.yaml
>>> │ ├── [ 416] nomad
>>> │ │ ├── [1.1K] client.yaml
>>> │ │ ├── [1.0K] cluster.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [1.8K] heron_nomad.sh
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [2.7K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.7K] statemgr.yaml
>>> │ │ └── [1.1K] uploader.yaml
>>> │ ├── [ 384] sandbox
>>> │ │ ├── [ 984] client.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [2.2K] healthmgr.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [1.3K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.3K] statemgr.yaml
>>> │ │ └── [1.2K] uploader.yaml
>>> │ ├── [ 352] slurm
>>> │ │ ├── [ 915] client.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [1.4K] scheduler.yaml
>>> │ │ ├── [1.2K] slurm.sh
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.2K] statemgr.yaml
>>> │ │ └── [1.2K] uploader.yaml
>>> │ ├── [ 512] standalone
>>> │ │ ├── [ 984] client.yaml
>>> │ │ ├── [1.0K] cluster.yaml
>>> │ │ ├── [1.1K] downloader.yaml
>>> │ │ ├── [ 12K] heron_internals.yaml
>>> │ │ ├── [1.8K] heron_nomad.sh
>>> │ │ ├── [ 845] inventory.yaml
>>> │ │ ├── [6.4K] metrics_sinks.yaml
>>> │ │ ├── [1.0K] packing.yaml
>>> │ │ ├── [ 96] resources
>>> │ │ │ └── [1015] master.hcl
>>> │ │ ├── [2.5K] scheduler.yaml
>>> │ │ ├── [1.6K] stateful.yaml
>>> │ │ ├── [1.7K] statemgr.yaml
>>> │ │ ├── [ 256] templates
>>> │ │ │ ├── [1.5K] apiserver.template.hcl
>>> │ │ │ ├── [1.6K] heron_tools.template.hcl
>>> │ │ │ ├── [2.4K] scheduler.template.yaml
>>> │ │ │ ├── [1.1K] slave.template.hcl
>>> │ │ │ ├── [1.7K] statemgr.template.yaml
>>> │ │ │ └── [1.0K] uploader.template.yaml
>>> │ │ └── [1.0K] uploader.yaml
>>> │ ├── [ 160] test
>>> │ │ ├── [ 11K] test_heron_internals.yaml
>>> │ │ ├── [8.1K] test_metrics_sinks.yaml
>>> │ │ └── [ 816] test_override.yaml
>>> │ └── [ 384] yarn
>>> │ ├── [ 915] client.yaml
>>> │ ├── [1.1K] downloader.yaml
>>> │ ├── [2.2K] healthmgr.yaml
>>> │ ├── [ 12K] heron_internals.yaml
>>> │ ├── [6.4K] metrics_sinks.yaml
>>> │ ├── [1.0K] packing.yaml
>>> │ ├── [1.6K] scheduler.yaml
>>> │ ├── [1.6K] stateful.yaml
>>> │ ├── [1.2K] statemgr.yaml
>>> │ └── [1.1K] uploader.yaml
>>> ├── [ 160] dist
>>> │ ├── [ 160] heron-core
>>> │ │ ├── [ 160] heron-core
>>> │ │ │ ├── [ 384] bin
>>> │ │ │ │ ├── [6.3M] heron-cpp-instance
>>> │ │ │ │ ├── [1.2K] heron-downloader
>>> │ │ │ │ ├── [ 931] heron-downloader-config
>>> │ │ │ │ ├── [ 931] heron-downloader-config.sh
>>> │ │ │ │ ├── [1.2K] heron-downloader.sh
>>> │ │ │ │ ├── [1.7M] heron-executor
>>> │ │ │ │ ├── [1.8M] heron-python-instance
>>> │ │ │ │ ├── [2.5M] heron-shell
>>> │ │ │ │ ├── [7.4M] heron-stmgr
>>> │ │ │ │ └── [7.7M] heron-tmaster
>>> │ │ │ └── [ 416] lib
>>> │ │ │ ├── [ 96] ckptmgr
>>> │ │ │ │ └── [5.4M] heron-ckptmgr.jar
>>> │ │ │ ├── [ 96] downloaders
>>> │ │ │ │ └── [ 16M] heron-downloader.jar
>>> │ │ │ ├── [ 96] healthmgr
>>> │ │ │ │ └── [ 32M] heron-healthmgr.jar
>>> │ │ │ ├── [ 96] instance
>>> │ │ │ │ └── [3.3M] heron-instance.jar
>>> │ │ │ ├── [ 96] metricscachemgr
>>> │ │ │ │ └── [8.8M] heron-metricscachemgr.jar
>>> │ │ │ ├── [ 96] metricsmgr
>>> │ │ │ │ └── [6.9M] heron-metricsmgr.jar
>>> │ │ │ ├── [ 128] packing
>>> │ │ │ │ ├── [5.3M] heron-binpacking-packing.jar
>>> │ │ │ │ └── [5.3M] heron-roundrobin-packing.jar
>>> │ │ │ ├── [ 288] scheduler
>>> │ │ │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
>>> │ │ │ │ ├── [7.5M] heron-local-scheduler.jar
>>> │ │ │ │ ├── [7.5M] heron-marathon-scheduler.jar
>>> │ │ │ │ ├── [9.4M] heron-mesos-scheduler.jar
>>> │ │ │ │ ├── [ 13M] heron-nomad-scheduler.jar
>>> │ │ │ │ ├── [7.3M] heron-scheduler.jar
>>> │ │ │ │ └── [7.5M] heron-slurm-scheduler.jar
>>> │ │ │ ├── [ 160] statefulstorage
>>> │ │ │ │ ├── [ 18M] heron-dlog-statefulstorage.jar
>>> │ │ │ │ ├── [5.0M] heron-hdfs-statefulstorage.jar
>>> │ │ │ │ └── [5.0M] heron-localfs-statefulstorage.jar
>>> │ │ │ └── [ 128] statemgr
>>> │ │ │ ├── [5.3M] heron-localfs-statemgr.jar
>>> │ │ │ └── [6.6M] heron-zookeeper-statemgr.jar
>>> │ │ └── [ 285] release.yaml
>>> │ └── [186M] heron-core.tar.gz
>>> ├── [ 512] examples
>>> │ ├── [2.0K] fibonacci.yaml
>>> │ ├── [3.3M] heron-api-examples.jar
>>> │ ├── [4.4M] heron-eco-examples.jar
>>> │ ├── [1.6K] heron-stateful-windowing.yaml
>>> │ ├── [1.2K] heron-stateful-word-count.yaml
>>> │ ├── [ 12M] heron-streamlet-examples.jar
>>> │ ├── [ 31M] heron-streamlet-scala-examples.jar
>>> │ ├── [2.1K] heron_fibonacci.yaml
>>> │ ├── [1.9K] heron_windowing.yaml
>>> │ ├── [2.0K] heron_wordcount.yaml
>>> │ ├── [ 860] sample.properties
>>> │ ├── [1.8K] simple_windowing.yaml
>>> │ ├── [2.0K] simple_wordcount.yaml
>>> │ └── [4.4M] storm-eco-examples.jar
>>> ├── [ 352] include
>>> │ ├── [ 288] bolt
>>> │ │ ├── [2.0K] base-basic-bolt.h
>>> │ │ ├── [1.3K] base-rich-bolt.h
>>> │ │ ├── [2.2K] basic-bolt-collector.h
>>> │ │ ├── [2.8K] ibasic-bolt.h
>>> │ │ ├── [3.0K] ibolt-output-collector.h
>>> │ │ ├── [4.3K] ibolt.h
>>> │ │ └── [1.2K] irich-bolt.h
>>> │ ├── [ 96] config
>>> │ │ └── [ 12K] config.h
>>> │ ├── [ 160] exceptions
>>> │ │ ├── [1.2K] already-alive-exception.h
>>> │ │ ├── [1.3K] invalid-topology-exception.h
>>> │ │ └── [1.3K] serialization-exception.h
>>> │ ├── [ 320] metric
>>> │ │ ├── [1.4K] assignable-metric.h
>>> │ │ ├── [1.5K] count-metric.h
>>> │ │ ├── [1.2K] imetric.h
>>> │ │ ├── [1.6K] imetrics-registrar.h
>>> │ │ ├── [1.3K] imulti-metric.h
>>> │ │ ├── [1.6K] mean-metric.h
>>> │ │ ├── [1.9K] multi-count-metric.h
>>> │ │ └── [1.9K] multi-mean-metric.h
>>> │ ├── [ 192] serializer
>>> │ │ ├── [1.6K] cereal-serializer.h
>>> │ │ ├── [1.9K] ipluggable-serializer.h
>>> │ │ ├── [1.5K] string-serializer.h
>>> │ │ └── [6.2K] tuple-serializer-utils.h
>>> │ ├── [ 192] spout
>>> │ │ ├── [1.4K] base-rich-spout.h
>>> │ │ ├── [1.3K] irich-spout.h
>>> │ │ ├── [2.4K] ispout-output-collector.h
>>> │ │ └── [5.6K] ispout.h
>>> │ ├── [ 192] topology
>>> │ │ ├── [1.4K] base-rich-spout.h
>>> │ │ ├── [1.3K] irich-spout.h
>>> │ │ ├── [2.4K] ispout-output-collector.h
>>> │ │ └── [5.6K] ispout.h
>>> │ ├── [ 128] tuple
>>> │ │ ├── [2.9K] fields.h
>>> │ │ └── [2.8K] tuple.h
>>> │ └── [ 96] utils
>>> │ └── [2.4K] utils.h
>>> ├── [ 384] lib
>>> │ ├── [ 224] api
>>> │ │ ├── [ 43K] api-scala.jar
>>> │ │ ├── [ 46M] heron-apiserver.jar
>>> │ │ ├── [110K] libapi-java.jar
>>> │ │ ├── [ 60K] libcxx-api.a
>>> │ │ └── [ 61K] libcxx-api.pic.a
>>> │ ├── [ 96] downloaders
>>> │ │ └── [ 16M] heron-downloader.jar
>>> │ ├── [ 96] metricscachemgr
>>> │ │ └── [8.8M] heron-metricscachemgr.jar
>>> │ ├── [ 128] packing
>>> │ │ ├── [5.3M] heron-binpacking-packing.jar
>>> │ │ └── [5.3M] heron-roundrobin-packing.jar
>>> │ ├── [ 416] scheduler
>>> │ │ ├── [7.5M] heron-aurora-scheduler.jar
>>> │ │ ├── [5.3M] heron-binpacking-packing.jar
>>> │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
>>> │ │ ├── [7.5M] heron-local-scheduler.jar
>>> │ │ ├── [7.5M] heron-marathon-scheduler.jar
>>> │ │ ├── [9.4M] heron-mesos-scheduler.jar
>>> │ │ ├── [ 13M] heron-nomad-scheduler.jar
>>> │ │ ├── [5.3M] heron-roundrobin-packing.jar
>>> │ │ ├── [7.3M] heron-scheduler.jar
>>> │ │ ├── [7.5M] heron-slurm-scheduler.jar
>>> │ │ └── [9.0M] heron-yarn-scheduler.jar
>>> │ ├── [ 96] simulator
>>> │ │ └── [ 45K] libsimulator-java.jar
>>> │ ├── [ 128] statemgr
>>> │ │ ├── [5.3M] heron-localfs-statemgr.jar
>>> │ │ └── [6.6M] heron-zookeeper-statemgr.jar
>>> │ ├── [ 160] third_party
>>> │ │ ├── [1.3M] libprotobuf_java.jar
>>> │ │ ├── [ 29K] slf4j-api-1.7.7.jar
>>> │ │ └── [7.7K] slf4j-jdk14-1.7.7.jar
>>> │ └── [ 320] uploader
>>> │ ├── [ 16M] heron-dlog-uploader.jar
>>> │ ├── [4.8M] heron-gcs-uploader.jar
>>> │ ├── [2.5M] heron-hdfs-uploader.jar
>>> │ ├── [3.9M] heron-http-uploader.jar
>>> │ ├── [2.5M] heron-localfs-uploader.jar
>>> │ ├── [2.5M] heron-null-uploader.jar
>>> │ ├── [7.2M] heron-s3-uploader.jar
>>> │ └── [2.5M] heron-scp-uploader.jar
>>> └── [ 285] release.yaml
>>>
>>> 53 directories, 253 files
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Mar 10, 2019 at 11:54 PM Ning Wang <wa...@gmail.com> wrote:
>>>
>>>> Thanks Dave!
>>>>
>>>> Yeah. I will get a list of content in packages.
>>>>
>>>> For the docker image, I think it should be ok. Let me try to publish it
>>>> to Apache docker hub and see if there is anything missing.
>>>>
>>>>
>>>>
>>>> On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <da...@comcast.net>
>>>> wrote:
>>>>
>>>>> Hi -
>>>>>
>>>>> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wa...@gmail.com> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I have been trying to release Heron 0.20.1 (being distracted time by
>>>>> time
>>>>> > though) and the most recent question I am having is where to put the
>>>>> binary
>>>>> > packages.
>>>>> >
>>>>> > The binary packages are (when we were doing releases on github):
>>>>> > - tar.gz packages for osx, centos and ubuntu, each one includes all
>>>>> modules
>>>>> > like core, lib, tools, etc.
>>>>> > - .sh packages for the three platforms. which is an installer for the
>>>>> > modules in the tar.gz packages.
>>>>> > - docker image (dockerhub, not github most of the times)
>>>>>
>>>>> Let’s discuss the components in each binary package and how big that
>>>>> they really are and need to be.
>>>>>
>>>>> >
>>>>> > Currently each package is more than 400MB.
>>>>>
>>>>> When packages of this size are released from dev to the release area
>>>>> it requires replication to the Apache Mirror system. When the size exceeds
>>>>> an aggregate of 1GB then Infra needs to manually handle things to avoid
>>>>> impacting the mirrors. (There are 250 projects using the mirrors.)
>>>>>
>>>>>
>>>>>
>>>>> >
>>>>> > I was trying to understand the Apache rules and my impression was
>>>>> that
>>>>> > these package should be on dist.apache.org like the src packages (I
>>>>> might
>>>>> > be wrong about the rules though) and it looks like Apache Storm has a
>>>>> > binary package in their release.
>>>>>
>>>>> Make the case for Heron without comparison to other projects.
>>>>>
>>>>> There is a place to make Apache Docker releases on docker hub. Let’s
>>>>> figure out this if it is a valid distribution that Heron could make.
>>>>>
>>>>> >
>>>>> > However it seems Apache infra has a byte limit of 500MB for each
>>>>> release. I
>>>>> > guess it means that the binary packages are not "required" to be on
>>>>> Apache
>>>>> > infra?
>>>>>
>>>>> Yes and no. Let’s discuss the packages first.
>>>>>
>>>>> >
>>>>> > The binary packages are convenient for users. So I think they should
>>>>> be
>>>>> > included in release. The question is where should we put them?
>>>>> >
>>>>> > So far it looks like the options are:
>>>>> > - ask for an exception and publish them to dist.apache.org. It
>>>>> seems like
>>>>> > Apache infra guys don't suggest this solution.
>>>>> > - publish only src package to dist.apache.org and publish the binary
>>>>> > packages on github (or is there any other suggestion?). This is
>>>>> convenient
>>>>> > for us and there is no problem so far (we have binary packages for
>>>>> all the
>>>>> > previous releases and github hasn't complained). The question about
>>>>> this
>>>>> > option is more like if this is acceptable (or ever better suggested)
>>>>> by
>>>>> > Apache?
>>>>> >
>>>>> > What do you think about the two options above and any other options
>>>>> we
>>>>> > should consider?
>>>>>
>>>>> Let’s discuss the packages.
>>>>>
>>>>> Next we will also need to discuss the website.
>>>>>
>>>>> >
>>>>> > Thanks.
>>>>> > --ning
>>>>>
>>>>>
>>
>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Dave Fisher <da...@comcast.net>.
Hi -
> On Mar 14, 2019, at 10:31 AM, Ning Wang <wa...@gmail.com> wrote:
>
> Thanks Dave! Let me start a vote~
>
> No complain to you guys. :) It seems like Apache infra doesn't have a clear guideline about this case. Otherwise it could be a lot more straightforward. Hopefully this issue will be improved soon.
The ASF has had continual evolution over 20 + years. What Infrastructure provides now is way more obvious than in the past. You mentors are here to guide you through things.
The Docker capability is new and that is certainly not well understood. The best thing to do with Infra if your mentors cannot help is to ask asfinfra on Slack (it was hip chat last year) - https://theasf.slack.com <https://theasf.slack.com/>
>
> I don't think our jars contain LICENSE files, this is required? All dependencies are built from source code so hopefully they are ok.
The first time is the hardest with binaries and it will take time. The key is listing each dependency that gets included and reviewing its license. If any are “category X” then they need to be replaced. The Release Policy is the responsibility of Legal Affairs.
Here is the reference about which licenses can be included. [1] This is frequently updates. If you have any doubts then we ask through a LEGAL jira or legal-discuss@apache.org <ma...@apache.org> ( a public list)
Regards,
Dave
[1] http://www.apache.org/legal/resolved.html
>
>
>
> On Thu, Mar 14, 2019 at 10:24 AM Dave Fisher <dave2wave@comcast.net <ma...@comcast.net>> wrote:
> Hi -
>
> Agreed. Let’s just make sure we review how this convenience is shown as some on the IPMC are concerned about how the Github release tab appears.
>
> For binary packages we really need to know and carefully review the licensing for all the included dependencies. It is more than just the Jars as a whole but what the builds have included. I apologize for not having enough time to push this part of the review.
>
> On the Dev list we can discuss where users might find these prospective binary packages which will have no guarantees about licensing.
>
> Let’s do an Official Apache Source Release.
>
> Regards,
> Dave
>
>> On Mar 14, 2019, at 10:17 AM, Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
>>
>> I am thinking that we can leave the "convenience community binary packages" in github for this release (because of the file size) and then try to reduce the package size for future releases so that they are more friendly to Apache infrastructure. It has been blocked by this for a while and I am hoping to start the vote sooner than later.
>>
>> Is this ok? @Dave Fisher <ma...@comcast.net> @P. Taylor Goetz <ma...@gmail.com> ^^
>>
>>
>>
>> On Tue, Mar 12, 2019 at 5:37 PM Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
>> For the content in the packages:
>>
>> Currently we have binary packages for three platforms: osx, centos and ubuntu. The file sizes are fairly close among them. I am going to use ubuntu as example:
>>
>> - there is a tar.gz file and an sh file. I am thinking we may keep the sh file only since it is easier to use and contains the same data as the tar.gz file. This can save about 50% in size (from 470 * 6 -> 470 * 3).
>>
>> The file size is 470m.
>> -rw-r--r--@ 1 nwang staff 470508875 Feb 27 10:57 heron-v0.20.1-incubating-rc1-ubuntu14.04.tar.gz
>>
>> The package is trim-able I believe, but there isn't many obvious ones:
>> drwxr-xr-x@ 14 nwang staff 448 Dec 31 1969 bin
>> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 conf
>> drwxr-xr-x@ 5 nwang staff 160 Mar 6 15:10 dist
>> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 examples
>> drwxr-xr-x@ 11 nwang staff 352 Dec 31 1969 include
>> drwxr-xr-x@ 12 nwang staff 384 Mar 6 15:04 lib
>> -r-xr-xr-x@ 1 nwang staff 285 Dec 31 1969 release.yaml
>>
>> It is used to install a local environment that contains all necessary components:
>> - bin: binaries for tools like api server, tracker, ui, etc
>> - conf: configurations for different environments
>> - dist: heron core and its tar.gz file (194m, might be trim-able).
>> - examples: example topology jars
>> - include: c++ header files
>> - lib: c++ and java lib files
>>
>> More detailed information is attached at the end of this email.
>>
>> /dist/core/core and /lib are the biggest dirs and they might have a few duplicated big files we can try to get rid of.
>>
>> heron-downloader.jar 16m * 2
>> heron-metricscachemgr.jar 8.8m * 2
>> heron-binpacking-packing.jar 5.3m * 3
>> heron-roundrobin-packing.jar 5.3m * 2
>> scheduler jars 60m * 2
>> heron-localfs-statemgr.jar 5.3m * 2
>> heron-zookeeper-statemgr.jar 6.6m * 2
>>
>> In case we can get rid of the core.tar.gz and the duplicated files, we might be able to reduce the size to 480m - 190m - 100m = 190m. Then these binary packages will be 190 * 3 = 570m, after some works.
>>
>>
>>
>> heron-v0.20.1-incubating-rc1-ubuntu14.04
>> ├── [ 448] bin
>> │ ├── [2.1M] heron
>> │ ├── [2.1M] heron-admin
>> │ ├── [1.5K] heron-apiserver
>> │ ├── [1.5K] heron-apiserver.sh
>> │ ├── [1.2K] heron-downloader
>> │ ├── [ 931] heron-downloader-config
>> │ ├── [ 931] heron-downloader-config.sh
>> │ ├── [1.2K] heron-downloader.sh
>> │ ├── [2.2M] heron-explorer
>> │ ├── [ 42M] heron-nomad
>> │ ├── [2.5M] heron-tracker
>> │ └── [3.5M] heron-ui
>> ├── [ 512] conf
>> │ ├── [ 384] aurora
>> │ │ ├── [1.1K] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [3.3K] heron.aurora
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [2.7K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [2.3K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 352] examples
>> │ │ ├── [1.3K] aurora_scheduler.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [1.3K] local_scheduler.yaml
>> │ │ ├── [1.5K] local_stateful.yaml
>> │ │ ├── [ 951] localfs_statemgr.yaml
>> │ │ ├── [1001] localfs_uploader.yaml
>> │ │ ├── [7.3K] metrics_sinks.yaml
>> │ │ └── [ 800] roundrobin_packing.yaml
>> │ ├── [2.3K] heron_tracker.yaml
>> │ ├── [ 352] kubernetes
>> │ │ ├── [ 979] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.3K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.5K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.7K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 384] local
>> │ │ ├── [ 984] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [2.2K] healthmgr.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.3K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 352] localzk
>> │ │ ├── [ 915] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.3K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [2.5K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 320] marathon
>> │ │ ├── [1.1K] client.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.5K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 320] mesos
>> │ │ ├── [ 800] client.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [2.0K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 416] nomad
>> │ │ ├── [1.1K] client.yaml
>> │ │ ├── [1.0K] cluster.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [1.8K] heron_nomad.sh
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [2.7K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.7K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 384] sandbox
>> │ │ ├── [ 984] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [2.2K] healthmgr.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.3K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 352] slurm
>> │ │ ├── [ 915] client.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.4K] scheduler.yaml
>> │ │ ├── [1.2K] slurm.sh
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.2K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 512] standalone
>> │ │ ├── [ 984] client.yaml
>> │ │ ├── [1.0K] cluster.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [1.8K] heron_nomad.sh
>> │ │ ├── [ 845] inventory.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [ 96] resources
>> │ │ │ └── [1015] master.hcl
>> │ │ ├── [2.5K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.7K] statemgr.yaml
>> │ │ ├── [ 256] templates
>> │ │ │ ├── [1.5K] apiserver.template.hcl
>> │ │ │ ├── [1.6K] heron_tools.template.hcl
>> │ │ │ ├── [2.4K] scheduler.template.yaml
>> │ │ │ ├── [1.1K] slave.template.hcl
>> │ │ │ ├── [1.7K] statemgr.template.yaml
>> │ │ │ └── [1.0K] uploader.template.yaml
>> │ │ └── [1.0K] uploader.yaml
>> │ ├── [ 160] test
>> │ │ ├── [ 11K] test_heron_internals.yaml
>> │ │ ├── [8.1K] test_metrics_sinks.yaml
>> │ │ └── [ 816] test_override.yaml
>> │ └── [ 384] yarn
>> │ ├── [ 915] client.yaml
>> │ ├── [1.1K] downloader.yaml
>> │ ├── [2.2K] healthmgr.yaml
>> │ ├── [ 12K] heron_internals.yaml
>> │ ├── [6.4K] metrics_sinks.yaml
>> │ ├── [1.0K] packing.yaml
>> │ ├── [1.6K] scheduler.yaml
>> │ ├── [1.6K] stateful.yaml
>> │ ├── [1.2K] statemgr.yaml
>> │ └── [1.1K] uploader.yaml
>> ├── [ 160] dist
>> │ ├── [ 160] heron-core
>> │ │ ├── [ 160] heron-core
>> │ │ │ ├── [ 384] bin
>> │ │ │ │ ├── [6.3M] heron-cpp-instance
>> │ │ │ │ ├── [1.2K] heron-downloader
>> │ │ │ │ ├── [ 931] heron-downloader-config
>> │ │ │ │ ├── [ 931] heron-downloader-config.sh
>> │ │ │ │ ├── [1.2K] heron-downloader.sh
>> │ │ │ │ ├── [1.7M] heron-executor
>> │ │ │ │ ├── [1.8M] heron-python-instance
>> │ │ │ │ ├── [2.5M] heron-shell
>> │ │ │ │ ├── [7.4M] heron-stmgr
>> │ │ │ │ └── [7.7M] heron-tmaster
>> │ │ │ └── [ 416] lib
>> │ │ │ ├── [ 96] ckptmgr
>> │ │ │ │ └── [5.4M] heron-ckptmgr.jar
>> │ │ │ ├── [ 96] downloaders
>> │ │ │ │ └── [ 16M] heron-downloader.jar
>> │ │ │ ├── [ 96] healthmgr
>> │ │ │ │ └── [ 32M] heron-healthmgr.jar
>> │ │ │ ├── [ 96] instance
>> │ │ │ │ └── [3.3M] heron-instance.jar
>> │ │ │ ├── [ 96] metricscachemgr
>> │ │ │ │ └── [8.8M] heron-metricscachemgr.jar
>> │ │ │ ├── [ 96] metricsmgr
>> │ │ │ │ └── [6.9M] heron-metricsmgr.jar
>> │ │ │ ├── [ 128] packing
>> │ │ │ │ ├── [5.3M] heron-binpacking-packing.jar
>> │ │ │ │ └── [5.3M] heron-roundrobin-packing.jar
>> │ │ │ ├── [ 288] scheduler
>> │ │ │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
>> │ │ │ │ ├── [7.5M] heron-local-scheduler.jar
>> │ │ │ │ ├── [7.5M] heron-marathon-scheduler.jar
>> │ │ │ │ ├── [9.4M] heron-mesos-scheduler.jar
>> │ │ │ │ ├── [ 13M] heron-nomad-scheduler.jar
>> │ │ │ │ ├── [7.3M] heron-scheduler.jar
>> │ │ │ │ └── [7.5M] heron-slurm-scheduler.jar
>> │ │ │ ├── [ 160] statefulstorage
>> │ │ │ │ ├── [ 18M] heron-dlog-statefulstorage.jar
>> │ │ │ │ ├── [5.0M] heron-hdfs-statefulstorage.jar
>> │ │ │ │ └── [5.0M] heron-localfs-statefulstorage.jar
>> │ │ │ └── [ 128] statemgr
>> │ │ │ ├── [5.3M] heron-localfs-statemgr.jar
>> │ │ │ └── [6.6M] heron-zookeeper-statemgr.jar
>> │ │ └── [ 285] release.yaml
>> │ └── [186M] heron-core.tar.gz
>> ├── [ 512] examples
>> │ ├── [2.0K] fibonacci.yaml
>> │ ├── [3.3M] heron-api-examples.jar
>> │ ├── [4.4M] heron-eco-examples.jar
>> │ ├── [1.6K] heron-stateful-windowing.yaml
>> │ ├── [1.2K] heron-stateful-word-count.yaml
>> │ ├── [ 12M] heron-streamlet-examples.jar
>> │ ├── [ 31M] heron-streamlet-scala-examples.jar
>> │ ├── [2.1K] heron_fibonacci.yaml
>> │ ├── [1.9K] heron_windowing.yaml
>> │ ├── [2.0K] heron_wordcount.yaml
>> │ ├── [ 860] sample.properties
>> │ ├── [1.8K] simple_windowing.yaml
>> │ ├── [2.0K] simple_wordcount.yaml
>> │ └── [4.4M] storm-eco-examples.jar
>> ├── [ 352] include
>> │ ├── [ 288] bolt
>> │ │ ├── [2.0K] base-basic-bolt.h
>> │ │ ├── [1.3K] base-rich-bolt.h
>> │ │ ├── [2.2K] basic-bolt-collector.h
>> │ │ ├── [2.8K] ibasic-bolt.h
>> │ │ ├── [3.0K] ibolt-output-collector.h
>> │ │ ├── [4.3K] ibolt.h
>> │ │ └── [1.2K] irich-bolt.h
>> │ ├── [ 96] config
>> │ │ └── [ 12K] config.h
>> │ ├── [ 160] exceptions
>> │ │ ├── [1.2K] already-alive-exception.h
>> │ │ ├── [1.3K] invalid-topology-exception.h
>> │ │ └── [1.3K] serialization-exception.h
>> │ ├── [ 320] metric
>> │ │ ├── [1.4K] assignable-metric.h
>> │ │ ├── [1.5K] count-metric.h
>> │ │ ├── [1.2K] imetric.h
>> │ │ ├── [1.6K] imetrics-registrar.h
>> │ │ ├── [1.3K] imulti-metric.h
>> │ │ ├── [1.6K] mean-metric.h
>> │ │ ├── [1.9K] multi-count-metric.h
>> │ │ └── [1.9K] multi-mean-metric.h
>> │ ├── [ 192] serializer
>> │ │ ├── [1.6K] cereal-serializer.h
>> │ │ ├── [1.9K] ipluggable-serializer.h
>> │ │ ├── [1.5K] string-serializer.h
>> │ │ └── [6.2K] tuple-serializer-utils.h
>> │ ├── [ 192] spout
>> │ │ ├── [1.4K] base-rich-spout.h
>> │ │ ├── [1.3K] irich-spout.h
>> │ │ ├── [2.4K] ispout-output-collector.h
>> │ │ └── [5.6K] ispout.h
>> │ ├── [ 192] topology
>> │ │ ├── [1.4K] base-rich-spout.h
>> │ │ ├── [1.3K] irich-spout.h
>> │ │ ├── [2.4K] ispout-output-collector.h
>> │ │ └── [5.6K] ispout.h
>> │ ├── [ 128] tuple
>> │ │ ├── [2.9K] fields.h
>> │ │ └── [2.8K] tuple.h
>> │ └── [ 96] utils
>> │ └── [2.4K] utils.h
>> ├── [ 384] lib
>> │ ├── [ 224] api
>> │ │ ├── [ 43K] api-scala.jar
>> │ │ ├── [ 46M] heron-apiserver.jar
>> │ │ ├── [110K] libapi-java.jar
>> │ │ ├── [ 60K] libcxx-api.a
>> │ │ └── [ 61K] libcxx-api.pic.a
>> │ ├── [ 96] downloaders
>> │ │ └── [ 16M] heron-downloader.jar
>> │ ├── [ 96] metricscachemgr
>> │ │ └── [8.8M] heron-metricscachemgr.jar
>> │ ├── [ 128] packing
>> │ │ ├── [5.3M] heron-binpacking-packing.jar
>> │ │ └── [5.3M] heron-roundrobin-packing.jar
>> │ ├── [ 416] scheduler
>> │ │ ├── [7.5M] heron-aurora-scheduler.jar
>> │ │ ├── [5.3M] heron-binpacking-packing.jar
>> │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
>> │ │ ├── [7.5M] heron-local-scheduler.jar
>> │ │ ├── [7.5M] heron-marathon-scheduler.jar
>> │ │ ├── [9.4M] heron-mesos-scheduler.jar
>> │ │ ├── [ 13M] heron-nomad-scheduler.jar
>> │ │ ├── [5.3M] heron-roundrobin-packing.jar
>> │ │ ├── [7.3M] heron-scheduler.jar
>> │ │ ├── [7.5M] heron-slurm-scheduler.jar
>> │ │ └── [9.0M] heron-yarn-scheduler.jar
>> │ ├── [ 96] simulator
>> │ │ └── [ 45K] libsimulator-java.jar
>> │ ├── [ 128] statemgr
>> │ │ ├── [5.3M] heron-localfs-statemgr.jar
>> │ │ └── [6.6M] heron-zookeeper-statemgr.jar
>> │ ├── [ 160] third_party
>> │ │ ├── [1.3M] libprotobuf_java.jar
>> │ │ ├── [ 29K] slf4j-api-1.7.7.jar
>> │ │ └── [7.7K] slf4j-jdk14-1.7.7.jar
>> │ └── [ 320] uploader
>> │ ├── [ 16M] heron-dlog-uploader.jar
>> │ ├── [4.8M] heron-gcs-uploader.jar
>> │ ├── [2.5M] heron-hdfs-uploader.jar
>> │ ├── [3.9M] heron-http-uploader.jar
>> │ ├── [2.5M] heron-localfs-uploader.jar
>> │ ├── [2.5M] heron-null-uploader.jar
>> │ ├── [7.2M] heron-s3-uploader.jar
>> │ └── [2.5M] heron-scp-uploader.jar
>> └── [ 285] release.yaml
>>
>> 53 directories, 253 files
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Mar 10, 2019 at 11:54 PM Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
>> Thanks Dave!
>>
>> Yeah. I will get a list of content in packages.
>>
>> For the docker image, I think it should be ok. Let me try to publish it to Apache docker hub and see if there is anything missing.
>>
>>
>>
>> On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <dave2wave@comcast.net <ma...@comcast.net>> wrote:
>> Hi -
>>
>> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Hi,
>> >
>> > I have been trying to release Heron 0.20.1 (being distracted time by time
>> > though) and the most recent question I am having is where to put the binary
>> > packages.
>> >
>> > The binary packages are (when we were doing releases on github):
>> > - tar.gz packages for osx, centos and ubuntu, each one includes all modules
>> > like core, lib, tools, etc.
>> > - .sh packages for the three platforms. which is an installer for the
>> > modules in the tar.gz packages.
>> > - docker image (dockerhub, not github most of the times)
>>
>> Let’s discuss the components in each binary package and how big that they really are and need to be.
>>
>> >
>> > Currently each package is more than 400MB.
>>
>> When packages of this size are released from dev to the release area it requires replication to the Apache Mirror system. When the size exceeds an aggregate of 1GB then Infra needs to manually handle things to avoid impacting the mirrors. (There are 250 projects using the mirrors.)
>>
>>
>>
>> >
>> > I was trying to understand the Apache rules and my impression was that
>> > these package should be on dist.apache.org <http://dist.apache.org/> like the src packages (I might
>> > be wrong about the rules though) and it looks like Apache Storm has a
>> > binary package in their release.
>>
>> Make the case for Heron without comparison to other projects.
>>
>> There is a place to make Apache Docker releases on docker hub. Let’s figure out this if it is a valid distribution that Heron could make.
>>
>> >
>> > However it seems Apache infra has a byte limit of 500MB for each release. I
>> > guess it means that the binary packages are not "required" to be on Apache
>> > infra?
>>
>> Yes and no. Let’s discuss the packages first.
>>
>> >
>> > The binary packages are convenient for users. So I think they should be
>> > included in release. The question is where should we put them?
>> >
>> > So far it looks like the options are:
>> > - ask for an exception and publish them to dist.apache.org <http://dist.apache.org/>. It seems like
>> > Apache infra guys don't suggest this solution.
>> > - publish only src package to dist.apache.org <http://dist.apache.org/> and publish the binary
>> > packages on github (or is there any other suggestion?). This is convenient
>> > for us and there is no problem so far (we have binary packages for all the
>> > previous releases and github hasn't complained). The question about this
>> > option is more like if this is acceptable (or ever better suggested) by
>> > Apache?
>> >
>> > What do you think about the two options above and any other options we
>> > should consider?
>>
>> Let’s discuss the packages.
>>
>> Next we will also need to discuss the website.
>>
>> >
>> > Thanks.
>> > --ning
>>
>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Ning Wang <wa...@gmail.com>.
Thanks Dave! Let me start a vote~
No complain to you guys. :) It seems like Apache infra doesn't have a
clear guideline about this case. Otherwise it could be a lot more
straightforward. Hopefully this issue will be improved soon.
I don't think our jars contain LICENSE files, this is required? All
dependencies are built from source code so hopefully they are ok.
On Thu, Mar 14, 2019 at 10:24 AM Dave Fisher <da...@comcast.net> wrote:
> Hi -
>
> Agreed. Let’s just make sure we review how this convenience is shown as
> some on the IPMC are concerned about how the Github release tab appears.
>
> For binary packages we really need to know and carefully review the
> licensing for all the included dependencies. It is more than just the Jars
> as a whole but what the builds have included. I apologize for not having
> enough time to push this part of the review.
>
> On the Dev list we can discuss where users might find these prospective
> binary packages which will have no guarantees about licensing.
>
> Let’s do an Official Apache Source Release.
>
> Regards,
> Dave
>
> On Mar 14, 2019, at 10:17 AM, Ning Wang <wa...@gmail.com> wrote:
>
> I am thinking that we can leave the "convenience community binary
> packages" in github for this release (because of the file size) and then
> try to reduce the package size for future releases so that they are more
> friendly to Apache infrastructure. It has been blocked by this for a while
> and I am hoping to start the vote sooner than later.
>
> Is this ok? @Dave Fisher <da...@comcast.net> @P. Taylor Goetz
> <pt...@gmail.com> ^^
>
>
>
> On Tue, Mar 12, 2019 at 5:37 PM Ning Wang <wa...@gmail.com> wrote:
>
>> For the content in the packages:
>>
>> Currently we have binary packages for three platforms: osx, centos and
>> ubuntu. The file sizes are fairly close among them. I am going to use
>> ubuntu as example:
>>
>> - there is a tar.gz file and an sh file. I am thinking we may keep the sh
>> file only since it is easier to use and contains the same data as the
>> tar.gz file. This can save about 50% in size (from 470 * 6 -> 470 * 3).
>>
>> The file size is 470m.
>> -rw-r--r--@ 1 nwang staff 470508875 Feb 27 10:57
>> heron-v0.20.1-incubating-rc1-ubuntu14.04.tar.gz
>>
>> The package is trim-able I believe, but there isn't many obvious ones:
>> drwxr-xr-x@ 14 nwang staff 448 Dec 31 1969 bin
>> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 conf
>> drwxr-xr-x@ 5 nwang staff 160 Mar 6 15:10 dist
>> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 examples
>> drwxr-xr-x@ 11 nwang staff 352 Dec 31 1969 include
>> drwxr-xr-x@ 12 nwang staff 384 Mar 6 15:04 lib
>> -r-xr-xr-x@ 1 nwang staff 285 Dec 31 1969 release.yaml
>>
>> It is used to install a local environment that contains all necessary
>> components:
>> - bin: binaries for tools like api server, tracker, ui, etc
>> - conf: configurations for different environments
>> - dist: heron core and its tar.gz file (194m, might be trim-able).
>> - examples: example topology jars
>> - include: c++ header files
>> - lib: c++ and java lib files
>>
>> More detailed information is attached at the end of this email.
>>
>> /dist/core/core and /lib are the biggest dirs and they might have a few
>> duplicated big files we can try to get rid of.
>>
>> heron-downloader.jar 16m * 2
>> heron-metricscachemgr.jar 8.8m * 2
>> heron-binpacking-packing.jar 5.3m * 3
>> heron-roundrobin-packing.jar 5.3m * 2
>> scheduler jars 60m * 2
>> heron-localfs-statemgr.jar 5.3m * 2
>> heron-zookeeper-statemgr.jar 6.6m * 2
>>
>> In case we can get rid of the core.tar.gz and the duplicated files, we
>> might be able to reduce the size to 480m - 190m - 100m = 190m. Then these
>> binary packages will be 190 * 3 = 570m, after some works.
>>
>>
>>
>> heron-v0.20.1-incubating-rc1-ubuntu14.04
>> ├── [ 448] bin
>> │ ├── [2.1M] heron
>> │ ├── [2.1M] heron-admin
>> │ ├── [1.5K] heron-apiserver
>> │ ├── [1.5K] heron-apiserver.sh
>> │ ├── [1.2K] heron-downloader
>> │ ├── [ 931] heron-downloader-config
>> │ ├── [ 931] heron-downloader-config.sh
>> │ ├── [1.2K] heron-downloader.sh
>> │ ├── [2.2M] heron-explorer
>> │ ├── [ 42M] heron-nomad
>> │ ├── [2.5M] heron-tracker
>> │ └── [3.5M] heron-ui
>> ├── [ 512] conf
>> │ ├── [ 384] aurora
>> │ │ ├── [1.1K] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [3.3K] heron.aurora
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [2.7K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [2.3K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 352] examples
>> │ │ ├── [1.3K] aurora_scheduler.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [1.3K] local_scheduler.yaml
>> │ │ ├── [1.5K] local_stateful.yaml
>> │ │ ├── [ 951] localfs_statemgr.yaml
>> │ │ ├── [1001] localfs_uploader.yaml
>> │ │ ├── [7.3K] metrics_sinks.yaml
>> │ │ └── [ 800] roundrobin_packing.yaml
>> │ ├── [2.3K] heron_tracker.yaml
>> │ ├── [ 352] kubernetes
>> │ │ ├── [ 979] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.3K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.5K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.7K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 384] local
>> │ │ ├── [ 984] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [2.2K] healthmgr.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.3K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 352] localzk
>> │ │ ├── [ 915] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.3K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [2.5K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 320] marathon
>> │ │ ├── [1.1K] client.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.5K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 320] mesos
>> │ │ ├── [ 800] client.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [2.0K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 416] nomad
>> │ │ ├── [1.1K] client.yaml
>> │ │ ├── [1.0K] cluster.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [1.8K] heron_nomad.sh
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [2.7K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.7K] statemgr.yaml
>> │ │ └── [1.1K] uploader.yaml
>> │ ├── [ 384] sandbox
>> │ │ ├── [ 984] client.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [2.2K] healthmgr.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.3K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.3K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 352] slurm
>> │ │ ├── [ 915] client.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [1.4K] scheduler.yaml
>> │ │ ├── [1.2K] slurm.sh
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.2K] statemgr.yaml
>> │ │ └── [1.2K] uploader.yaml
>> │ ├── [ 512] standalone
>> │ │ ├── [ 984] client.yaml
>> │ │ ├── [1.0K] cluster.yaml
>> │ │ ├── [1.1K] downloader.yaml
>> │ │ ├── [ 12K] heron_internals.yaml
>> │ │ ├── [1.8K] heron_nomad.sh
>> │ │ ├── [ 845] inventory.yaml
>> │ │ ├── [6.4K] metrics_sinks.yaml
>> │ │ ├── [1.0K] packing.yaml
>> │ │ ├── [ 96] resources
>> │ │ │ └── [1015] master.hcl
>> │ │ ├── [2.5K] scheduler.yaml
>> │ │ ├── [1.6K] stateful.yaml
>> │ │ ├── [1.7K] statemgr.yaml
>> │ │ ├── [ 256] templates
>> │ │ │ ├── [1.5K] apiserver.template.hcl
>> │ │ │ ├── [1.6K] heron_tools.template.hcl
>> │ │ │ ├── [2.4K] scheduler.template.yaml
>> │ │ │ ├── [1.1K] slave.template.hcl
>> │ │ │ ├── [1.7K] statemgr.template.yaml
>> │ │ │ └── [1.0K] uploader.template.yaml
>> │ │ └── [1.0K] uploader.yaml
>> │ ├── [ 160] test
>> │ │ ├── [ 11K] test_heron_internals.yaml
>> │ │ ├── [8.1K] test_metrics_sinks.yaml
>> │ │ └── [ 816] test_override.yaml
>> │ └── [ 384] yarn
>> │ ├── [ 915] client.yaml
>> │ ├── [1.1K] downloader.yaml
>> │ ├── [2.2K] healthmgr.yaml
>> │ ├── [ 12K] heron_internals.yaml
>> │ ├── [6.4K] metrics_sinks.yaml
>> │ ├── [1.0K] packing.yaml
>> │ ├── [1.6K] scheduler.yaml
>> │ ├── [1.6K] stateful.yaml
>> │ ├── [1.2K] statemgr.yaml
>> │ └── [1.1K] uploader.yaml
>> ├── [ 160] dist
>> │ ├── [ 160] heron-core
>> │ │ ├── [ 160] heron-core
>> │ │ │ ├── [ 384] bin
>> │ │ │ │ ├── [6.3M] heron-cpp-instance
>> │ │ │ │ ├── [1.2K] heron-downloader
>> │ │ │ │ ├── [ 931] heron-downloader-config
>> │ │ │ │ ├── [ 931] heron-downloader-config.sh
>> │ │ │ │ ├── [1.2K] heron-downloader.sh
>> │ │ │ │ ├── [1.7M] heron-executor
>> │ │ │ │ ├── [1.8M] heron-python-instance
>> │ │ │ │ ├── [2.5M] heron-shell
>> │ │ │ │ ├── [7.4M] heron-stmgr
>> │ │ │ │ └── [7.7M] heron-tmaster
>> │ │ │ └── [ 416] lib
>> │ │ │ ├── [ 96] ckptmgr
>> │ │ │ │ └── [5.4M] heron-ckptmgr.jar
>> │ │ │ ├── [ 96] downloaders
>> │ │ │ │ └── [ 16M] heron-downloader.jar
>> │ │ │ ├── [ 96] healthmgr
>> │ │ │ │ └── [ 32M] heron-healthmgr.jar
>> │ │ │ ├── [ 96] instance
>> │ │ │ │ └── [3.3M] heron-instance.jar
>> │ │ │ ├── [ 96] metricscachemgr
>> │ │ │ │ └── [8.8M] heron-metricscachemgr.jar
>> │ │ │ ├── [ 96] metricsmgr
>> │ │ │ │ └── [6.9M] heron-metricsmgr.jar
>> │ │ │ ├── [ 128] packing
>> │ │ │ │ ├── [5.3M] heron-binpacking-packing.jar
>> │ │ │ │ └── [5.3M] heron-roundrobin-packing.jar
>> │ │ │ ├── [ 288] scheduler
>> │ │ │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
>> │ │ │ │ ├── [7.5M] heron-local-scheduler.jar
>> │ │ │ │ ├── [7.5M] heron-marathon-scheduler.jar
>> │ │ │ │ ├── [9.4M] heron-mesos-scheduler.jar
>> │ │ │ │ ├── [ 13M] heron-nomad-scheduler.jar
>> │ │ │ │ ├── [7.3M] heron-scheduler.jar
>> │ │ │ │ └── [7.5M] heron-slurm-scheduler.jar
>> │ │ │ ├── [ 160] statefulstorage
>> │ │ │ │ ├── [ 18M] heron-dlog-statefulstorage.jar
>> │ │ │ │ ├── [5.0M] heron-hdfs-statefulstorage.jar
>> │ │ │ │ └── [5.0M] heron-localfs-statefulstorage.jar
>> │ │ │ └── [ 128] statemgr
>> │ │ │ ├── [5.3M] heron-localfs-statemgr.jar
>> │ │ │ └── [6.6M] heron-zookeeper-statemgr.jar
>> │ │ └── [ 285] release.yaml
>> │ └── [186M] heron-core.tar.gz
>> ├── [ 512] examples
>> │ ├── [2.0K] fibonacci.yaml
>> │ ├── [3.3M] heron-api-examples.jar
>> │ ├── [4.4M] heron-eco-examples.jar
>> │ ├── [1.6K] heron-stateful-windowing.yaml
>> │ ├── [1.2K] heron-stateful-word-count.yaml
>> │ ├── [ 12M] heron-streamlet-examples.jar
>> │ ├── [ 31M] heron-streamlet-scala-examples.jar
>> │ ├── [2.1K] heron_fibonacci.yaml
>> │ ├── [1.9K] heron_windowing.yaml
>> │ ├── [2.0K] heron_wordcount.yaml
>> │ ├── [ 860] sample.properties
>> │ ├── [1.8K] simple_windowing.yaml
>> │ ├── [2.0K] simple_wordcount.yaml
>> │ └── [4.4M] storm-eco-examples.jar
>> ├── [ 352] include
>> │ ├── [ 288] bolt
>> │ │ ├── [2.0K] base-basic-bolt.h
>> │ │ ├── [1.3K] base-rich-bolt.h
>> │ │ ├── [2.2K] basic-bolt-collector.h
>> │ │ ├── [2.8K] ibasic-bolt.h
>> │ │ ├── [3.0K] ibolt-output-collector.h
>> │ │ ├── [4.3K] ibolt.h
>> │ │ └── [1.2K] irich-bolt.h
>> │ ├── [ 96] config
>> │ │ └── [ 12K] config.h
>> │ ├── [ 160] exceptions
>> │ │ ├── [1.2K] already-alive-exception.h
>> │ │ ├── [1.3K] invalid-topology-exception.h
>> │ │ └── [1.3K] serialization-exception.h
>> │ ├── [ 320] metric
>> │ │ ├── [1.4K] assignable-metric.h
>> │ │ ├── [1.5K] count-metric.h
>> │ │ ├── [1.2K] imetric.h
>> │ │ ├── [1.6K] imetrics-registrar.h
>> │ │ ├── [1.3K] imulti-metric.h
>> │ │ ├── [1.6K] mean-metric.h
>> │ │ ├── [1.9K] multi-count-metric.h
>> │ │ └── [1.9K] multi-mean-metric.h
>> │ ├── [ 192] serializer
>> │ │ ├── [1.6K] cereal-serializer.h
>> │ │ ├── [1.9K] ipluggable-serializer.h
>> │ │ ├── [1.5K] string-serializer.h
>> │ │ └── [6.2K] tuple-serializer-utils.h
>> │ ├── [ 192] spout
>> │ │ ├── [1.4K] base-rich-spout.h
>> │ │ ├── [1.3K] irich-spout.h
>> │ │ ├── [2.4K] ispout-output-collector.h
>> │ │ └── [5.6K] ispout.h
>> │ ├── [ 192] topology
>> │ │ ├── [1.4K] base-rich-spout.h
>> │ │ ├── [1.3K] irich-spout.h
>> │ │ ├── [2.4K] ispout-output-collector.h
>> │ │ └── [5.6K] ispout.h
>> │ ├── [ 128] tuple
>> │ │ ├── [2.9K] fields.h
>> │ │ └── [2.8K] tuple.h
>> │ └── [ 96] utils
>> │ └── [2.4K] utils.h
>> ├── [ 384] lib
>> │ ├── [ 224] api
>> │ │ ├── [ 43K] api-scala.jar
>> │ │ ├── [ 46M] heron-apiserver.jar
>> │ │ ├── [110K] libapi-java.jar
>> │ │ ├── [ 60K] libcxx-api.a
>> │ │ └── [ 61K] libcxx-api.pic.a
>> │ ├── [ 96] downloaders
>> │ │ └── [ 16M] heron-downloader.jar
>> │ ├── [ 96] metricscachemgr
>> │ │ └── [8.8M] heron-metricscachemgr.jar
>> │ ├── [ 128] packing
>> │ │ ├── [5.3M] heron-binpacking-packing.jar
>> │ │ └── [5.3M] heron-roundrobin-packing.jar
>> │ ├── [ 416] scheduler
>> │ │ ├── [7.5M] heron-aurora-scheduler.jar
>> │ │ ├── [5.3M] heron-binpacking-packing.jar
>> │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
>> │ │ ├── [7.5M] heron-local-scheduler.jar
>> │ │ ├── [7.5M] heron-marathon-scheduler.jar
>> │ │ ├── [9.4M] heron-mesos-scheduler.jar
>> │ │ ├── [ 13M] heron-nomad-scheduler.jar
>> │ │ ├── [5.3M] heron-roundrobin-packing.jar
>> │ │ ├── [7.3M] heron-scheduler.jar
>> │ │ ├── [7.5M] heron-slurm-scheduler.jar
>> │ │ └── [9.0M] heron-yarn-scheduler.jar
>> │ ├── [ 96] simulator
>> │ │ └── [ 45K] libsimulator-java.jar
>> │ ├── [ 128] statemgr
>> │ │ ├── [5.3M] heron-localfs-statemgr.jar
>> │ │ └── [6.6M] heron-zookeeper-statemgr.jar
>> │ ├── [ 160] third_party
>> │ │ ├── [1.3M] libprotobuf_java.jar
>> │ │ ├── [ 29K] slf4j-api-1.7.7.jar
>> │ │ └── [7.7K] slf4j-jdk14-1.7.7.jar
>> │ └── [ 320] uploader
>> │ ├── [ 16M] heron-dlog-uploader.jar
>> │ ├── [4.8M] heron-gcs-uploader.jar
>> │ ├── [2.5M] heron-hdfs-uploader.jar
>> │ ├── [3.9M] heron-http-uploader.jar
>> │ ├── [2.5M] heron-localfs-uploader.jar
>> │ ├── [2.5M] heron-null-uploader.jar
>> │ ├── [7.2M] heron-s3-uploader.jar
>> │ └── [2.5M] heron-scp-uploader.jar
>> └── [ 285] release.yaml
>>
>> 53 directories, 253 files
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Mar 10, 2019 at 11:54 PM Ning Wang <wa...@gmail.com> wrote:
>>
>>> Thanks Dave!
>>>
>>> Yeah. I will get a list of content in packages.
>>>
>>> For the docker image, I think it should be ok. Let me try to publish it
>>> to Apache docker hub and see if there is anything missing.
>>>
>>>
>>>
>>> On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <da...@comcast.net>
>>> wrote:
>>>
>>>> Hi -
>>>>
>>>> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wa...@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have been trying to release Heron 0.20.1 (being distracted time by
>>>> time
>>>> > though) and the most recent question I am having is where to put the
>>>> binary
>>>> > packages.
>>>> >
>>>> > The binary packages are (when we were doing releases on github):
>>>> > - tar.gz packages for osx, centos and ubuntu, each one includes all
>>>> modules
>>>> > like core, lib, tools, etc.
>>>> > - .sh packages for the three platforms. which is an installer for the
>>>> > modules in the tar.gz packages.
>>>> > - docker image (dockerhub, not github most of the times)
>>>>
>>>> Let’s discuss the components in each binary package and how big that
>>>> they really are and need to be.
>>>>
>>>> >
>>>> > Currently each package is more than 400MB.
>>>>
>>>> When packages of this size are released from dev to the release area it
>>>> requires replication to the Apache Mirror system. When the size exceeds an
>>>> aggregate of 1GB then Infra needs to manually handle things to avoid
>>>> impacting the mirrors. (There are 250 projects using the mirrors.)
>>>>
>>>>
>>>>
>>>> >
>>>> > I was trying to understand the Apache rules and my impression was that
>>>> > these package should be on dist.apache.org like the src packages (I
>>>> might
>>>> > be wrong about the rules though) and it looks like Apache Storm has a
>>>> > binary package in their release.
>>>>
>>>> Make the case for Heron without comparison to other projects.
>>>>
>>>> There is a place to make Apache Docker releases on docker hub. Let’s
>>>> figure out this if it is a valid distribution that Heron could make.
>>>>
>>>> >
>>>> > However it seems Apache infra has a byte limit of 500MB for each
>>>> release. I
>>>> > guess it means that the binary packages are not "required" to be on
>>>> Apache
>>>> > infra?
>>>>
>>>> Yes and no. Let’s discuss the packages first.
>>>>
>>>> >
>>>> > The binary packages are convenient for users. So I think they should
>>>> be
>>>> > included in release. The question is where should we put them?
>>>> >
>>>> > So far it looks like the options are:
>>>> > - ask for an exception and publish them to dist.apache.org. It seems
>>>> like
>>>> > Apache infra guys don't suggest this solution.
>>>> > - publish only src package to dist.apache.org and publish the binary
>>>> > packages on github (or is there any other suggestion?). This is
>>>> convenient
>>>> > for us and there is no problem so far (we have binary packages for
>>>> all the
>>>> > previous releases and github hasn't complained). The question about
>>>> this
>>>> > option is more like if this is acceptable (or ever better suggested)
>>>> by
>>>> > Apache?
>>>> >
>>>> > What do you think about the two options above and any other options we
>>>> > should consider?
>>>>
>>>> Let’s discuss the packages.
>>>>
>>>> Next we will also need to discuss the website.
>>>>
>>>> >
>>>> > Thanks.
>>>> > --ning
>>>>
>>>>
>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Dave Fisher <da...@comcast.net>.
Hi -
Agreed. Let’s just make sure we review how this convenience is shown as some on the IPMC are concerned about how the Github release tab appears.
For binary packages we really need to know and carefully review the licensing for all the included dependencies. It is more than just the Jars as a whole but what the builds have included. I apologize for not having enough time to push this part of the review.
On the Dev list we can discuss where users might find these prospective binary packages which will have no guarantees about licensing.
Let’s do an Official Apache Source Release.
Regards,
Dave
> On Mar 14, 2019, at 10:17 AM, Ning Wang <wa...@gmail.com> wrote:
>
> I am thinking that we can leave the "convenience community binary packages" in github for this release (because of the file size) and then try to reduce the package size for future releases so that they are more friendly to Apache infrastructure. It has been blocked by this for a while and I am hoping to start the vote sooner than later.
>
> Is this ok? @Dave Fisher <ma...@comcast.net> @P. Taylor Goetz <ma...@gmail.com> ^^
>
>
>
> On Tue, Mar 12, 2019 at 5:37 PM Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
> For the content in the packages:
>
> Currently we have binary packages for three platforms: osx, centos and ubuntu. The file sizes are fairly close among them. I am going to use ubuntu as example:
>
> - there is a tar.gz file and an sh file. I am thinking we may keep the sh file only since it is easier to use and contains the same data as the tar.gz file. This can save about 50% in size (from 470 * 6 -> 470 * 3).
>
> The file size is 470m.
> -rw-r--r--@ 1 nwang staff 470508875 Feb 27 10:57 heron-v0.20.1-incubating-rc1-ubuntu14.04.tar.gz
>
> The package is trim-able I believe, but there isn't many obvious ones:
> drwxr-xr-x@ 14 nwang staff 448 Dec 31 1969 bin
> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 conf
> drwxr-xr-x@ 5 nwang staff 160 Mar 6 15:10 dist
> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 examples
> drwxr-xr-x@ 11 nwang staff 352 Dec 31 1969 include
> drwxr-xr-x@ 12 nwang staff 384 Mar 6 15:04 lib
> -r-xr-xr-x@ 1 nwang staff 285 Dec 31 1969 release.yaml
>
> It is used to install a local environment that contains all necessary components:
> - bin: binaries for tools like api server, tracker, ui, etc
> - conf: configurations for different environments
> - dist: heron core and its tar.gz file (194m, might be trim-able).
> - examples: example topology jars
> - include: c++ header files
> - lib: c++ and java lib files
>
> More detailed information is attached at the end of this email.
>
> /dist/core/core and /lib are the biggest dirs and they might have a few duplicated big files we can try to get rid of.
>
> heron-downloader.jar 16m * 2
> heron-metricscachemgr.jar 8.8m * 2
> heron-binpacking-packing.jar 5.3m * 3
> heron-roundrobin-packing.jar 5.3m * 2
> scheduler jars 60m * 2
> heron-localfs-statemgr.jar 5.3m * 2
> heron-zookeeper-statemgr.jar 6.6m * 2
>
> In case we can get rid of the core.tar.gz and the duplicated files, we might be able to reduce the size to 480m - 190m - 100m = 190m. Then these binary packages will be 190 * 3 = 570m, after some works.
>
>
>
> heron-v0.20.1-incubating-rc1-ubuntu14.04
> ├── [ 448] bin
> │ ├── [2.1M] heron
> │ ├── [2.1M] heron-admin
> │ ├── [1.5K] heron-apiserver
> │ ├── [1.5K] heron-apiserver.sh
> │ ├── [1.2K] heron-downloader
> │ ├── [ 931] heron-downloader-config
> │ ├── [ 931] heron-downloader-config.sh
> │ ├── [1.2K] heron-downloader.sh
> │ ├── [2.2M] heron-explorer
> │ ├── [ 42M] heron-nomad
> │ ├── [2.5M] heron-tracker
> │ └── [3.5M] heron-ui
> ├── [ 512] conf
> │ ├── [ 384] aurora
> │ │ ├── [1.1K] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [3.3K] heron.aurora
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [2.7K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [2.3K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 352] examples
> │ │ ├── [1.3K] aurora_scheduler.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [1.3K] local_scheduler.yaml
> │ │ ├── [1.5K] local_stateful.yaml
> │ │ ├── [ 951] localfs_statemgr.yaml
> │ │ ├── [1001] localfs_uploader.yaml
> │ │ ├── [7.3K] metrics_sinks.yaml
> │ │ └── [ 800] roundrobin_packing.yaml
> │ ├── [2.3K] heron_tracker.yaml
> │ ├── [ 352] kubernetes
> │ │ ├── [ 979] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.3K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.5K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.7K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 384] local
> │ │ ├── [ 984] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [2.2K] healthmgr.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.3K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 352] localzk
> │ │ ├── [ 915] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.3K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [2.5K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 320] marathon
> │ │ ├── [1.1K] client.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.5K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 320] mesos
> │ │ ├── [ 800] client.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [2.0K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 416] nomad
> │ │ ├── [1.1K] client.yaml
> │ │ ├── [1.0K] cluster.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [1.8K] heron_nomad.sh
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [2.7K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.7K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 384] sandbox
> │ │ ├── [ 984] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [2.2K] healthmgr.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.3K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 352] slurm
> │ │ ├── [ 915] client.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.4K] scheduler.yaml
> │ │ ├── [1.2K] slurm.sh
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.2K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 512] standalone
> │ │ ├── [ 984] client.yaml
> │ │ ├── [1.0K] cluster.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [1.8K] heron_nomad.sh
> │ │ ├── [ 845] inventory.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [ 96] resources
> │ │ │ └── [1015] master.hcl
> │ │ ├── [2.5K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.7K] statemgr.yaml
> │ │ ├── [ 256] templates
> │ │ │ ├── [1.5K] apiserver.template.hcl
> │ │ │ ├── [1.6K] heron_tools.template.hcl
> │ │ │ ├── [2.4K] scheduler.template.yaml
> │ │ │ ├── [1.1K] slave.template.hcl
> │ │ │ ├── [1.7K] statemgr.template.yaml
> │ │ │ └── [1.0K] uploader.template.yaml
> │ │ └── [1.0K] uploader.yaml
> │ ├── [ 160] test
> │ │ ├── [ 11K] test_heron_internals.yaml
> │ │ ├── [8.1K] test_metrics_sinks.yaml
> │ │ └── [ 816] test_override.yaml
> │ └── [ 384] yarn
> │ ├── [ 915] client.yaml
> │ ├── [1.1K] downloader.yaml
> │ ├── [2.2K] healthmgr.yaml
> │ ├── [ 12K] heron_internals.yaml
> │ ├── [6.4K] metrics_sinks.yaml
> │ ├── [1.0K] packing.yaml
> │ ├── [1.6K] scheduler.yaml
> │ ├── [1.6K] stateful.yaml
> │ ├── [1.2K] statemgr.yaml
> │ └── [1.1K] uploader.yaml
> ├── [ 160] dist
> │ ├── [ 160] heron-core
> │ │ ├── [ 160] heron-core
> │ │ │ ├── [ 384] bin
> │ │ │ │ ├── [6.3M] heron-cpp-instance
> │ │ │ │ ├── [1.2K] heron-downloader
> │ │ │ │ ├── [ 931] heron-downloader-config
> │ │ │ │ ├── [ 931] heron-downloader-config.sh
> │ │ │ │ ├── [1.2K] heron-downloader.sh
> │ │ │ │ ├── [1.7M] heron-executor
> │ │ │ │ ├── [1.8M] heron-python-instance
> │ │ │ │ ├── [2.5M] heron-shell
> │ │ │ │ ├── [7.4M] heron-stmgr
> │ │ │ │ └── [7.7M] heron-tmaster
> │ │ │ └── [ 416] lib
> │ │ │ ├── [ 96] ckptmgr
> │ │ │ │ └── [5.4M] heron-ckptmgr.jar
> │ │ │ ├── [ 96] downloaders
> │ │ │ │ └── [ 16M] heron-downloader.jar
> │ │ │ ├── [ 96] healthmgr
> │ │ │ │ └── [ 32M] heron-healthmgr.jar
> │ │ │ ├── [ 96] instance
> │ │ │ │ └── [3.3M] heron-instance.jar
> │ │ │ ├── [ 96] metricscachemgr
> │ │ │ │ └── [8.8M] heron-metricscachemgr.jar
> │ │ │ ├── [ 96] metricsmgr
> │ │ │ │ └── [6.9M] heron-metricsmgr.jar
> │ │ │ ├── [ 128] packing
> │ │ │ │ ├── [5.3M] heron-binpacking-packing.jar
> │ │ │ │ └── [5.3M] heron-roundrobin-packing.jar
> │ │ │ ├── [ 288] scheduler
> │ │ │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
> │ │ │ │ ├── [7.5M] heron-local-scheduler.jar
> │ │ │ │ ├── [7.5M] heron-marathon-scheduler.jar
> │ │ │ │ ├── [9.4M] heron-mesos-scheduler.jar
> │ │ │ │ ├── [ 13M] heron-nomad-scheduler.jar
> │ │ │ │ ├── [7.3M] heron-scheduler.jar
> │ │ │ │ └── [7.5M] heron-slurm-scheduler.jar
> │ │ │ ├── [ 160] statefulstorage
> │ │ │ │ ├── [ 18M] heron-dlog-statefulstorage.jar
> │ │ │ │ ├── [5.0M] heron-hdfs-statefulstorage.jar
> │ │ │ │ └── [5.0M] heron-localfs-statefulstorage.jar
> │ │ │ └── [ 128] statemgr
> │ │ │ ├── [5.3M] heron-localfs-statemgr.jar
> │ │ │ └── [6.6M] heron-zookeeper-statemgr.jar
> │ │ └── [ 285] release.yaml
> │ └── [186M] heron-core.tar.gz
> ├── [ 512] examples
> │ ├── [2.0K] fibonacci.yaml
> │ ├── [3.3M] heron-api-examples.jar
> │ ├── [4.4M] heron-eco-examples.jar
> │ ├── [1.6K] heron-stateful-windowing.yaml
> │ ├── [1.2K] heron-stateful-word-count.yaml
> │ ├── [ 12M] heron-streamlet-examples.jar
> │ ├── [ 31M] heron-streamlet-scala-examples.jar
> │ ├── [2.1K] heron_fibonacci.yaml
> │ ├── [1.9K] heron_windowing.yaml
> │ ├── [2.0K] heron_wordcount.yaml
> │ ├── [ 860] sample.properties
> │ ├── [1.8K] simple_windowing.yaml
> │ ├── [2.0K] simple_wordcount.yaml
> │ └── [4.4M] storm-eco-examples.jar
> ├── [ 352] include
> │ ├── [ 288] bolt
> │ │ ├── [2.0K] base-basic-bolt.h
> │ │ ├── [1.3K] base-rich-bolt.h
> │ │ ├── [2.2K] basic-bolt-collector.h
> │ │ ├── [2.8K] ibasic-bolt.h
> │ │ ├── [3.0K] ibolt-output-collector.h
> │ │ ├── [4.3K] ibolt.h
> │ │ └── [1.2K] irich-bolt.h
> │ ├── [ 96] config
> │ │ └── [ 12K] config.h
> │ ├── [ 160] exceptions
> │ │ ├── [1.2K] already-alive-exception.h
> │ │ ├── [1.3K] invalid-topology-exception.h
> │ │ └── [1.3K] serialization-exception.h
> │ ├── [ 320] metric
> │ │ ├── [1.4K] assignable-metric.h
> │ │ ├── [1.5K] count-metric.h
> │ │ ├── [1.2K] imetric.h
> │ │ ├── [1.6K] imetrics-registrar.h
> │ │ ├── [1.3K] imulti-metric.h
> │ │ ├── [1.6K] mean-metric.h
> │ │ ├── [1.9K] multi-count-metric.h
> │ │ └── [1.9K] multi-mean-metric.h
> │ ├── [ 192] serializer
> │ │ ├── [1.6K] cereal-serializer.h
> │ │ ├── [1.9K] ipluggable-serializer.h
> │ │ ├── [1.5K] string-serializer.h
> │ │ └── [6.2K] tuple-serializer-utils.h
> │ ├── [ 192] spout
> │ │ ├── [1.4K] base-rich-spout.h
> │ │ ├── [1.3K] irich-spout.h
> │ │ ├── [2.4K] ispout-output-collector.h
> │ │ └── [5.6K] ispout.h
> │ ├── [ 192] topology
> │ │ ├── [1.4K] base-rich-spout.h
> │ │ ├── [1.3K] irich-spout.h
> │ │ ├── [2.4K] ispout-output-collector.h
> │ │ └── [5.6K] ispout.h
> │ ├── [ 128] tuple
> │ │ ├── [2.9K] fields.h
> │ │ └── [2.8K] tuple.h
> │ └── [ 96] utils
> │ └── [2.4K] utils.h
> ├── [ 384] lib
> │ ├── [ 224] api
> │ │ ├── [ 43K] api-scala.jar
> │ │ ├── [ 46M] heron-apiserver.jar
> │ │ ├── [110K] libapi-java.jar
> │ │ ├── [ 60K] libcxx-api.a
> │ │ └── [ 61K] libcxx-api.pic.a
> │ ├── [ 96] downloaders
> │ │ └── [ 16M] heron-downloader.jar
> │ ├── [ 96] metricscachemgr
> │ │ └── [8.8M] heron-metricscachemgr.jar
> │ ├── [ 128] packing
> │ │ ├── [5.3M] heron-binpacking-packing.jar
> │ │ └── [5.3M] heron-roundrobin-packing.jar
> │ ├── [ 416] scheduler
> │ │ ├── [7.5M] heron-aurora-scheduler.jar
> │ │ ├── [5.3M] heron-binpacking-packing.jar
> │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
> │ │ ├── [7.5M] heron-local-scheduler.jar
> │ │ ├── [7.5M] heron-marathon-scheduler.jar
> │ │ ├── [9.4M] heron-mesos-scheduler.jar
> │ │ ├── [ 13M] heron-nomad-scheduler.jar
> │ │ ├── [5.3M] heron-roundrobin-packing.jar
> │ │ ├── [7.3M] heron-scheduler.jar
> │ │ ├── [7.5M] heron-slurm-scheduler.jar
> │ │ └── [9.0M] heron-yarn-scheduler.jar
> │ ├── [ 96] simulator
> │ │ └── [ 45K] libsimulator-java.jar
> │ ├── [ 128] statemgr
> │ │ ├── [5.3M] heron-localfs-statemgr.jar
> │ │ └── [6.6M] heron-zookeeper-statemgr.jar
> │ ├── [ 160] third_party
> │ │ ├── [1.3M] libprotobuf_java.jar
> │ │ ├── [ 29K] slf4j-api-1.7.7.jar
> │ │ └── [7.7K] slf4j-jdk14-1.7.7.jar
> │ └── [ 320] uploader
> │ ├── [ 16M] heron-dlog-uploader.jar
> │ ├── [4.8M] heron-gcs-uploader.jar
> │ ├── [2.5M] heron-hdfs-uploader.jar
> │ ├── [3.9M] heron-http-uploader.jar
> │ ├── [2.5M] heron-localfs-uploader.jar
> │ ├── [2.5M] heron-null-uploader.jar
> │ ├── [7.2M] heron-s3-uploader.jar
> │ └── [2.5M] heron-scp-uploader.jar
> └── [ 285] release.yaml
>
> 53 directories, 253 files
>
>
>
>
>
>
>
>
> On Sun, Mar 10, 2019 at 11:54 PM Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
> Thanks Dave!
>
> Yeah. I will get a list of content in packages.
>
> For the docker image, I think it should be ok. Let me try to publish it to Apache docker hub and see if there is anything missing.
>
>
>
> On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <dave2wave@comcast.net <ma...@comcast.net>> wrote:
> Hi -
>
> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wangninggm@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi,
> >
> > I have been trying to release Heron 0.20.1 (being distracted time by time
> > though) and the most recent question I am having is where to put the binary
> > packages.
> >
> > The binary packages are (when we were doing releases on github):
> > - tar.gz packages for osx, centos and ubuntu, each one includes all modules
> > like core, lib, tools, etc.
> > - .sh packages for the three platforms. which is an installer for the
> > modules in the tar.gz packages.
> > - docker image (dockerhub, not github most of the times)
>
> Let’s discuss the components in each binary package and how big that they really are and need to be.
>
> >
> > Currently each package is more than 400MB.
>
> When packages of this size are released from dev to the release area it requires replication to the Apache Mirror system. When the size exceeds an aggregate of 1GB then Infra needs to manually handle things to avoid impacting the mirrors. (There are 250 projects using the mirrors.)
>
>
>
> >
> > I was trying to understand the Apache rules and my impression was that
> > these package should be on dist.apache.org <http://dist.apache.org/> like the src packages (I might
> > be wrong about the rules though) and it looks like Apache Storm has a
> > binary package in their release.
>
> Make the case for Heron without comparison to other projects.
>
> There is a place to make Apache Docker releases on docker hub. Let’s figure out this if it is a valid distribution that Heron could make.
>
> >
> > However it seems Apache infra has a byte limit of 500MB for each release. I
> > guess it means that the binary packages are not "required" to be on Apache
> > infra?
>
> Yes and no. Let’s discuss the packages first.
>
> >
> > The binary packages are convenient for users. So I think they should be
> > included in release. The question is where should we put them?
> >
> > So far it looks like the options are:
> > - ask for an exception and publish them to dist.apache.org <http://dist.apache.org/>. It seems like
> > Apache infra guys don't suggest this solution.
> > - publish only src package to dist.apache.org <http://dist.apache.org/> and publish the binary
> > packages on github (or is there any other suggestion?). This is convenient
> > for us and there is no problem so far (we have binary packages for all the
> > previous releases and github hasn't complained). The question about this
> > option is more like if this is acceptable (or ever better suggested) by
> > Apache?
> >
> > What do you think about the two options above and any other options we
> > should consider?
>
> Let’s discuss the packages.
>
> Next we will also need to discuss the website.
>
> >
> > Thanks.
> > --ning
>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Ning Wang <wa...@gmail.com>.
I am thinking that we can leave the "convenience community binary packages"
in github for this release (because of the file size) and then try to
reduce the package size for future releases so that they are more friendly
to Apache infrastructure. It has been blocked by this for a while and I am
hoping to start the vote sooner than later.
Is this ok? @Dave Fisher <da...@comcast.net> @P. Taylor Goetz
<pt...@gmail.com> ^^
On Tue, Mar 12, 2019 at 5:37 PM Ning Wang <wa...@gmail.com> wrote:
> For the content in the packages:
>
> Currently we have binary packages for three platforms: osx, centos and
> ubuntu. The file sizes are fairly close among them. I am going to use
> ubuntu as example:
>
> - there is a tar.gz file and an sh file. I am thinking we may keep the sh
> file only since it is easier to use and contains the same data as the
> tar.gz file. This can save about 50% in size (from 470 * 6 -> 470 * 3).
>
> The file size is 470m.
> -rw-r--r--@ 1 nwang staff 470508875 Feb 27 10:57
> heron-v0.20.1-incubating-rc1-ubuntu14.04.tar.gz
>
> The package is trim-able I believe, but there isn't many obvious ones:
> drwxr-xr-x@ 14 nwang staff 448 Dec 31 1969 bin
> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 conf
> drwxr-xr-x@ 5 nwang staff 160 Mar 6 15:10 dist
> drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 examples
> drwxr-xr-x@ 11 nwang staff 352 Dec 31 1969 include
> drwxr-xr-x@ 12 nwang staff 384 Mar 6 15:04 lib
> -r-xr-xr-x@ 1 nwang staff 285 Dec 31 1969 release.yaml
>
> It is used to install a local environment that contains all necessary
> components:
> - bin: binaries for tools like api server, tracker, ui, etc
> - conf: configurations for different environments
> - dist: heron core and its tar.gz file (194m, might be trim-able).
> - examples: example topology jars
> - include: c++ header files
> - lib: c++ and java lib files
>
> More detailed information is attached at the end of this email.
>
> /dist/core/core and /lib are the biggest dirs and they might have a few
> duplicated big files we can try to get rid of.
>
> heron-downloader.jar 16m * 2
> heron-metricscachemgr.jar 8.8m * 2
> heron-binpacking-packing.jar 5.3m * 3
> heron-roundrobin-packing.jar 5.3m * 2
> scheduler jars 60m * 2
> heron-localfs-statemgr.jar 5.3m * 2
> heron-zookeeper-statemgr.jar 6.6m * 2
>
> In case we can get rid of the core.tar.gz and the duplicated files, we
> might be able to reduce the size to 480m - 190m - 100m = 190m. Then these
> binary packages will be 190 * 3 = 570m, after some works.
>
>
>
> heron-v0.20.1-incubating-rc1-ubuntu14.04
> ├── [ 448] bin
> │ ├── [2.1M] heron
> │ ├── [2.1M] heron-admin
> │ ├── [1.5K] heron-apiserver
> │ ├── [1.5K] heron-apiserver.sh
> │ ├── [1.2K] heron-downloader
> │ ├── [ 931] heron-downloader-config
> │ ├── [ 931] heron-downloader-config.sh
> │ ├── [1.2K] heron-downloader.sh
> │ ├── [2.2M] heron-explorer
> │ ├── [ 42M] heron-nomad
> │ ├── [2.5M] heron-tracker
> │ └── [3.5M] heron-ui
> ├── [ 512] conf
> │ ├── [ 384] aurora
> │ │ ├── [1.1K] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [3.3K] heron.aurora
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [2.7K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [2.3K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 352] examples
> │ │ ├── [1.3K] aurora_scheduler.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [1.3K] local_scheduler.yaml
> │ │ ├── [1.5K] local_stateful.yaml
> │ │ ├── [ 951] localfs_statemgr.yaml
> │ │ ├── [1001] localfs_uploader.yaml
> │ │ ├── [7.3K] metrics_sinks.yaml
> │ │ └── [ 800] roundrobin_packing.yaml
> │ ├── [2.3K] heron_tracker.yaml
> │ ├── [ 352] kubernetes
> │ │ ├── [ 979] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.3K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.5K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.7K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 384] local
> │ │ ├── [ 984] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [2.2K] healthmgr.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.3K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 352] localzk
> │ │ ├── [ 915] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.3K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [2.5K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 320] marathon
> │ │ ├── [1.1K] client.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.5K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 320] mesos
> │ │ ├── [ 800] client.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [2.0K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 416] nomad
> │ │ ├── [1.1K] client.yaml
> │ │ ├── [1.0K] cluster.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [1.8K] heron_nomad.sh
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [2.7K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.7K] statemgr.yaml
> │ │ └── [1.1K] uploader.yaml
> │ ├── [ 384] sandbox
> │ │ ├── [ 984] client.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [2.2K] healthmgr.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.3K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.3K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 352] slurm
> │ │ ├── [ 915] client.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [1.4K] scheduler.yaml
> │ │ ├── [1.2K] slurm.sh
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.2K] statemgr.yaml
> │ │ └── [1.2K] uploader.yaml
> │ ├── [ 512] standalone
> │ │ ├── [ 984] client.yaml
> │ │ ├── [1.0K] cluster.yaml
> │ │ ├── [1.1K] downloader.yaml
> │ │ ├── [ 12K] heron_internals.yaml
> │ │ ├── [1.8K] heron_nomad.sh
> │ │ ├── [ 845] inventory.yaml
> │ │ ├── [6.4K] metrics_sinks.yaml
> │ │ ├── [1.0K] packing.yaml
> │ │ ├── [ 96] resources
> │ │ │ └── [1015] master.hcl
> │ │ ├── [2.5K] scheduler.yaml
> │ │ ├── [1.6K] stateful.yaml
> │ │ ├── [1.7K] statemgr.yaml
> │ │ ├── [ 256] templates
> │ │ │ ├── [1.5K] apiserver.template.hcl
> │ │ │ ├── [1.6K] heron_tools.template.hcl
> │ │ │ ├── [2.4K] scheduler.template.yaml
> │ │ │ ├── [1.1K] slave.template.hcl
> │ │ │ ├── [1.7K] statemgr.template.yaml
> │ │ │ └── [1.0K] uploader.template.yaml
> │ │ └── [1.0K] uploader.yaml
> │ ├── [ 160] test
> │ │ ├── [ 11K] test_heron_internals.yaml
> │ │ ├── [8.1K] test_metrics_sinks.yaml
> │ │ └── [ 816] test_override.yaml
> │ └── [ 384] yarn
> │ ├── [ 915] client.yaml
> │ ├── [1.1K] downloader.yaml
> │ ├── [2.2K] healthmgr.yaml
> │ ├── [ 12K] heron_internals.yaml
> │ ├── [6.4K] metrics_sinks.yaml
> │ ├── [1.0K] packing.yaml
> │ ├── [1.6K] scheduler.yaml
> │ ├── [1.6K] stateful.yaml
> │ ├── [1.2K] statemgr.yaml
> │ └── [1.1K] uploader.yaml
> ├── [ 160] dist
> │ ├── [ 160] heron-core
> │ │ ├── [ 160] heron-core
> │ │ │ ├── [ 384] bin
> │ │ │ │ ├── [6.3M] heron-cpp-instance
> │ │ │ │ ├── [1.2K] heron-downloader
> │ │ │ │ ├── [ 931] heron-downloader-config
> │ │ │ │ ├── [ 931] heron-downloader-config.sh
> │ │ │ │ ├── [1.2K] heron-downloader.sh
> │ │ │ │ ├── [1.7M] heron-executor
> │ │ │ │ ├── [1.8M] heron-python-instance
> │ │ │ │ ├── [2.5M] heron-shell
> │ │ │ │ ├── [7.4M] heron-stmgr
> │ │ │ │ └── [7.7M] heron-tmaster
> │ │ │ └── [ 416] lib
> │ │ │ ├── [ 96] ckptmgr
> │ │ │ │ └── [5.4M] heron-ckptmgr.jar
> │ │ │ ├── [ 96] downloaders
> │ │ │ │ └── [ 16M] heron-downloader.jar
> │ │ │ ├── [ 96] healthmgr
> │ │ │ │ └── [ 32M] heron-healthmgr.jar
> │ │ │ ├── [ 96] instance
> │ │ │ │ └── [3.3M] heron-instance.jar
> │ │ │ ├── [ 96] metricscachemgr
> │ │ │ │ └── [8.8M] heron-metricscachemgr.jar
> │ │ │ ├── [ 96] metricsmgr
> │ │ │ │ └── [6.9M] heron-metricsmgr.jar
> │ │ │ ├── [ 128] packing
> │ │ │ │ ├── [5.3M] heron-binpacking-packing.jar
> │ │ │ │ └── [5.3M] heron-roundrobin-packing.jar
> │ │ │ ├── [ 288] scheduler
> │ │ │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
> │ │ │ │ ├── [7.5M] heron-local-scheduler.jar
> │ │ │ │ ├── [7.5M] heron-marathon-scheduler.jar
> │ │ │ │ ├── [9.4M] heron-mesos-scheduler.jar
> │ │ │ │ ├── [ 13M] heron-nomad-scheduler.jar
> │ │ │ │ ├── [7.3M] heron-scheduler.jar
> │ │ │ │ └── [7.5M] heron-slurm-scheduler.jar
> │ │ │ ├── [ 160] statefulstorage
> │ │ │ │ ├── [ 18M] heron-dlog-statefulstorage.jar
> │ │ │ │ ├── [5.0M] heron-hdfs-statefulstorage.jar
> │ │ │ │ └── [5.0M] heron-localfs-statefulstorage.jar
> │ │ │ └── [ 128] statemgr
> │ │ │ ├── [5.3M] heron-localfs-statemgr.jar
> │ │ │ └── [6.6M] heron-zookeeper-statemgr.jar
> │ │ └── [ 285] release.yaml
> │ └── [186M] heron-core.tar.gz
> ├── [ 512] examples
> │ ├── [2.0K] fibonacci.yaml
> │ ├── [3.3M] heron-api-examples.jar
> │ ├── [4.4M] heron-eco-examples.jar
> │ ├── [1.6K] heron-stateful-windowing.yaml
> │ ├── [1.2K] heron-stateful-word-count.yaml
> │ ├── [ 12M] heron-streamlet-examples.jar
> │ ├── [ 31M] heron-streamlet-scala-examples.jar
> │ ├── [2.1K] heron_fibonacci.yaml
> │ ├── [1.9K] heron_windowing.yaml
> │ ├── [2.0K] heron_wordcount.yaml
> │ ├── [ 860] sample.properties
> │ ├── [1.8K] simple_windowing.yaml
> │ ├── [2.0K] simple_wordcount.yaml
> │ └── [4.4M] storm-eco-examples.jar
> ├── [ 352] include
> │ ├── [ 288] bolt
> │ │ ├── [2.0K] base-basic-bolt.h
> │ │ ├── [1.3K] base-rich-bolt.h
> │ │ ├── [2.2K] basic-bolt-collector.h
> │ │ ├── [2.8K] ibasic-bolt.h
> │ │ ├── [3.0K] ibolt-output-collector.h
> │ │ ├── [4.3K] ibolt.h
> │ │ └── [1.2K] irich-bolt.h
> │ ├── [ 96] config
> │ │ └── [ 12K] config.h
> │ ├── [ 160] exceptions
> │ │ ├── [1.2K] already-alive-exception.h
> │ │ ├── [1.3K] invalid-topology-exception.h
> │ │ └── [1.3K] serialization-exception.h
> │ ├── [ 320] metric
> │ │ ├── [1.4K] assignable-metric.h
> │ │ ├── [1.5K] count-metric.h
> │ │ ├── [1.2K] imetric.h
> │ │ ├── [1.6K] imetrics-registrar.h
> │ │ ├── [1.3K] imulti-metric.h
> │ │ ├── [1.6K] mean-metric.h
> │ │ ├── [1.9K] multi-count-metric.h
> │ │ └── [1.9K] multi-mean-metric.h
> │ ├── [ 192] serializer
> │ │ ├── [1.6K] cereal-serializer.h
> │ │ ├── [1.9K] ipluggable-serializer.h
> │ │ ├── [1.5K] string-serializer.h
> │ │ └── [6.2K] tuple-serializer-utils.h
> │ ├── [ 192] spout
> │ │ ├── [1.4K] base-rich-spout.h
> │ │ ├── [1.3K] irich-spout.h
> │ │ ├── [2.4K] ispout-output-collector.h
> │ │ └── [5.6K] ispout.h
> │ ├── [ 192] topology
> │ │ ├── [1.4K] base-rich-spout.h
> │ │ ├── [1.3K] irich-spout.h
> │ │ ├── [2.4K] ispout-output-collector.h
> │ │ └── [5.6K] ispout.h
> │ ├── [ 128] tuple
> │ │ ├── [2.9K] fields.h
> │ │ └── [2.8K] tuple.h
> │ └── [ 96] utils
> │ └── [2.4K] utils.h
> ├── [ 384] lib
> │ ├── [ 224] api
> │ │ ├── [ 43K] api-scala.jar
> │ │ ├── [ 46M] heron-apiserver.jar
> │ │ ├── [110K] libapi-java.jar
> │ │ ├── [ 60K] libcxx-api.a
> │ │ └── [ 61K] libcxx-api.pic.a
> │ ├── [ 96] downloaders
> │ │ └── [ 16M] heron-downloader.jar
> │ ├── [ 96] metricscachemgr
> │ │ └── [8.8M] heron-metricscachemgr.jar
> │ ├── [ 128] packing
> │ │ ├── [5.3M] heron-binpacking-packing.jar
> │ │ └── [5.3M] heron-roundrobin-packing.jar
> │ ├── [ 416] scheduler
> │ │ ├── [7.5M] heron-aurora-scheduler.jar
> │ │ ├── [5.3M] heron-binpacking-packing.jar
> │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
> │ │ ├── [7.5M] heron-local-scheduler.jar
> │ │ ├── [7.5M] heron-marathon-scheduler.jar
> │ │ ├── [9.4M] heron-mesos-scheduler.jar
> │ │ ├── [ 13M] heron-nomad-scheduler.jar
> │ │ ├── [5.3M] heron-roundrobin-packing.jar
> │ │ ├── [7.3M] heron-scheduler.jar
> │ │ ├── [7.5M] heron-slurm-scheduler.jar
> │ │ └── [9.0M] heron-yarn-scheduler.jar
> │ ├── [ 96] simulator
> │ │ └── [ 45K] libsimulator-java.jar
> │ ├── [ 128] statemgr
> │ │ ├── [5.3M] heron-localfs-statemgr.jar
> │ │ └── [6.6M] heron-zookeeper-statemgr.jar
> │ ├── [ 160] third_party
> │ │ ├── [1.3M] libprotobuf_java.jar
> │ │ ├── [ 29K] slf4j-api-1.7.7.jar
> │ │ └── [7.7K] slf4j-jdk14-1.7.7.jar
> │ └── [ 320] uploader
> │ ├── [ 16M] heron-dlog-uploader.jar
> │ ├── [4.8M] heron-gcs-uploader.jar
> │ ├── [2.5M] heron-hdfs-uploader.jar
> │ ├── [3.9M] heron-http-uploader.jar
> │ ├── [2.5M] heron-localfs-uploader.jar
> │ ├── [2.5M] heron-null-uploader.jar
> │ ├── [7.2M] heron-s3-uploader.jar
> │ └── [2.5M] heron-scp-uploader.jar
> └── [ 285] release.yaml
>
> 53 directories, 253 files
>
>
>
>
>
>
>
>
> On Sun, Mar 10, 2019 at 11:54 PM Ning Wang <wa...@gmail.com> wrote:
>
>> Thanks Dave!
>>
>> Yeah. I will get a list of content in packages.
>>
>> For the docker image, I think it should be ok. Let me try to publish it
>> to Apache docker hub and see if there is anything missing.
>>
>>
>>
>> On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <da...@comcast.net> wrote:
>>
>>> Hi -
>>>
>>> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wa...@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have been trying to release Heron 0.20.1 (being distracted time by
>>> time
>>> > though) and the most recent question I am having is where to put the
>>> binary
>>> > packages.
>>> >
>>> > The binary packages are (when we were doing releases on github):
>>> > - tar.gz packages for osx, centos and ubuntu, each one includes all
>>> modules
>>> > like core, lib, tools, etc.
>>> > - .sh packages for the three platforms. which is an installer for the
>>> > modules in the tar.gz packages.
>>> > - docker image (dockerhub, not github most of the times)
>>>
>>> Let’s discuss the components in each binary package and how big that
>>> they really are and need to be.
>>>
>>> >
>>> > Currently each package is more than 400MB.
>>>
>>> When packages of this size are released from dev to the release area it
>>> requires replication to the Apache Mirror system. When the size exceeds an
>>> aggregate of 1GB then Infra needs to manually handle things to avoid
>>> impacting the mirrors. (There are 250 projects using the mirrors.)
>>>
>>>
>>>
>>> >
>>> > I was trying to understand the Apache rules and my impression was that
>>> > these package should be on dist.apache.org like the src packages (I
>>> might
>>> > be wrong about the rules though) and it looks like Apache Storm has a
>>> > binary package in their release.
>>>
>>> Make the case for Heron without comparison to other projects.
>>>
>>> There is a place to make Apache Docker releases on docker hub. Let’s
>>> figure out this if it is a valid distribution that Heron could make.
>>>
>>> >
>>> > However it seems Apache infra has a byte limit of 500MB for each
>>> release. I
>>> > guess it means that the binary packages are not "required" to be on
>>> Apache
>>> > infra?
>>>
>>> Yes and no. Let’s discuss the packages first.
>>>
>>> >
>>> > The binary packages are convenient for users. So I think they should be
>>> > included in release. The question is where should we put them?
>>> >
>>> > So far it looks like the options are:
>>> > - ask for an exception and publish them to dist.apache.org. It seems
>>> like
>>> > Apache infra guys don't suggest this solution.
>>> > - publish only src package to dist.apache.org and publish the binary
>>> > packages on github (or is there any other suggestion?). This is
>>> convenient
>>> > for us and there is no problem so far (we have binary packages for all
>>> the
>>> > previous releases and github hasn't complained). The question about
>>> this
>>> > option is more like if this is acceptable (or ever better suggested) by
>>> > Apache?
>>> >
>>> > What do you think about the two options above and any other options we
>>> > should consider?
>>>
>>> Let’s discuss the packages.
>>>
>>> Next we will also need to discuss the website.
>>>
>>> >
>>> > Thanks.
>>> > --ning
>>>
>>>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Ning Wang <wa...@gmail.com>.
For the content in the packages:
Currently we have binary packages for three platforms: osx, centos and
ubuntu. The file sizes are fairly close among them. I am going to use
ubuntu as example:
- there is a tar.gz file and an sh file. I am thinking we may keep the sh
file only since it is easier to use and contains the same data as the
tar.gz file. This can save about 50% in size (from 470 * 6 -> 470 * 3).
The file size is 470m.
-rw-r--r--@ 1 nwang staff 470508875 Feb 27 10:57
heron-v0.20.1-incubating-rc1-ubuntu14.04.tar.gz
The package is trim-able I believe, but there isn't many obvious ones:
drwxr-xr-x@ 14 nwang staff 448 Dec 31 1969 bin
drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 conf
drwxr-xr-x@ 5 nwang staff 160 Mar 6 15:10 dist
drwxr-xr-x@ 16 nwang staff 512 Dec 31 1969 examples
drwxr-xr-x@ 11 nwang staff 352 Dec 31 1969 include
drwxr-xr-x@ 12 nwang staff 384 Mar 6 15:04 lib
-r-xr-xr-x@ 1 nwang staff 285 Dec 31 1969 release.yaml
It is used to install a local environment that contains all necessary
components:
- bin: binaries for tools like api server, tracker, ui, etc
- conf: configurations for different environments
- dist: heron core and its tar.gz file (194m, might be trim-able).
- examples: example topology jars
- include: c++ header files
- lib: c++ and java lib files
More detailed information is attached at the end of this email.
/dist/core/core and /lib are the biggest dirs and they might have a few
duplicated big files we can try to get rid of.
heron-downloader.jar 16m * 2
heron-metricscachemgr.jar 8.8m * 2
heron-binpacking-packing.jar 5.3m * 3
heron-roundrobin-packing.jar 5.3m * 2
scheduler jars 60m * 2
heron-localfs-statemgr.jar 5.3m * 2
heron-zookeeper-statemgr.jar 6.6m * 2
In case we can get rid of the core.tar.gz and the duplicated files, we
might be able to reduce the size to 480m - 190m - 100m = 190m. Then these
binary packages will be 190 * 3 = 570m, after some works.
heron-v0.20.1-incubating-rc1-ubuntu14.04
├── [ 448] bin
│ ├── [2.1M] heron
│ ├── [2.1M] heron-admin
│ ├── [1.5K] heron-apiserver
│ ├── [1.5K] heron-apiserver.sh
│ ├── [1.2K] heron-downloader
│ ├── [ 931] heron-downloader-config
│ ├── [ 931] heron-downloader-config.sh
│ ├── [1.2K] heron-downloader.sh
│ ├── [2.2M] heron-explorer
│ ├── [ 42M] heron-nomad
│ ├── [2.5M] heron-tracker
│ └── [3.5M] heron-ui
├── [ 512] conf
│ ├── [ 384] aurora
│ │ ├── [1.1K] client.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [3.3K] heron.aurora
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [2.7K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [2.3K] statemgr.yaml
│ │ └── [1.1K] uploader.yaml
│ ├── [ 352] examples
│ │ ├── [1.3K] aurora_scheduler.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [1.3K] local_scheduler.yaml
│ │ ├── [1.5K] local_stateful.yaml
│ │ ├── [ 951] localfs_statemgr.yaml
│ │ ├── [1001] localfs_uploader.yaml
│ │ ├── [7.3K] metrics_sinks.yaml
│ │ └── [ 800] roundrobin_packing.yaml
│ ├── [2.3K] heron_tracker.yaml
│ ├── [ 352] kubernetes
│ │ ├── [ 979] client.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.3K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [1.5K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.7K] statemgr.yaml
│ │ └── [1.1K] uploader.yaml
│ ├── [ 384] local
│ │ ├── [ 984] client.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [2.2K] healthmgr.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [1.3K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.3K] statemgr.yaml
│ │ └── [1.2K] uploader.yaml
│ ├── [ 352] localzk
│ │ ├── [ 915] client.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [1.3K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [2.5K] statemgr.yaml
│ │ └── [1.2K] uploader.yaml
│ ├── [ 320] marathon
│ │ ├── [1.1K] client.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [1.5K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.3K] statemgr.yaml
│ │ └── [1.1K] uploader.yaml
│ ├── [ 320] mesos
│ │ ├── [ 800] client.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [2.0K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.3K] statemgr.yaml
│ │ └── [1.2K] uploader.yaml
│ ├── [ 416] nomad
│ │ ├── [1.1K] client.yaml
│ │ ├── [1.0K] cluster.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [1.8K] heron_nomad.sh
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [2.7K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.7K] statemgr.yaml
│ │ └── [1.1K] uploader.yaml
│ ├── [ 384] sandbox
│ │ ├── [ 984] client.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [2.2K] healthmgr.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [1.3K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.3K] statemgr.yaml
│ │ └── [1.2K] uploader.yaml
│ ├── [ 352] slurm
│ │ ├── [ 915] client.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [1.4K] scheduler.yaml
│ │ ├── [1.2K] slurm.sh
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.2K] statemgr.yaml
│ │ └── [1.2K] uploader.yaml
│ ├── [ 512] standalone
│ │ ├── [ 984] client.yaml
│ │ ├── [1.0K] cluster.yaml
│ │ ├── [1.1K] downloader.yaml
│ │ ├── [ 12K] heron_internals.yaml
│ │ ├── [1.8K] heron_nomad.sh
│ │ ├── [ 845] inventory.yaml
│ │ ├── [6.4K] metrics_sinks.yaml
│ │ ├── [1.0K] packing.yaml
│ │ ├── [ 96] resources
│ │ │ └── [1015] master.hcl
│ │ ├── [2.5K] scheduler.yaml
│ │ ├── [1.6K] stateful.yaml
│ │ ├── [1.7K] statemgr.yaml
│ │ ├── [ 256] templates
│ │ │ ├── [1.5K] apiserver.template.hcl
│ │ │ ├── [1.6K] heron_tools.template.hcl
│ │ │ ├── [2.4K] scheduler.template.yaml
│ │ │ ├── [1.1K] slave.template.hcl
│ │ │ ├── [1.7K] statemgr.template.yaml
│ │ │ └── [1.0K] uploader.template.yaml
│ │ └── [1.0K] uploader.yaml
│ ├── [ 160] test
│ │ ├── [ 11K] test_heron_internals.yaml
│ │ ├── [8.1K] test_metrics_sinks.yaml
│ │ └── [ 816] test_override.yaml
│ └── [ 384] yarn
│ ├── [ 915] client.yaml
│ ├── [1.1K] downloader.yaml
│ ├── [2.2K] healthmgr.yaml
│ ├── [ 12K] heron_internals.yaml
│ ├── [6.4K] metrics_sinks.yaml
│ ├── [1.0K] packing.yaml
│ ├── [1.6K] scheduler.yaml
│ ├── [1.6K] stateful.yaml
│ ├── [1.2K] statemgr.yaml
│ └── [1.1K] uploader.yaml
├── [ 160] dist
│ ├── [ 160] heron-core
│ │ ├── [ 160] heron-core
│ │ │ ├── [ 384] bin
│ │ │ │ ├── [6.3M] heron-cpp-instance
│ │ │ │ ├── [1.2K] heron-downloader
│ │ │ │ ├── [ 931] heron-downloader-config
│ │ │ │ ├── [ 931] heron-downloader-config.sh
│ │ │ │ ├── [1.2K] heron-downloader.sh
│ │ │ │ ├── [1.7M] heron-executor
│ │ │ │ ├── [1.8M] heron-python-instance
│ │ │ │ ├── [2.5M] heron-shell
│ │ │ │ ├── [7.4M] heron-stmgr
│ │ │ │ └── [7.7M] heron-tmaster
│ │ │ └── [ 416] lib
│ │ │ ├── [ 96] ckptmgr
│ │ │ │ └── [5.4M] heron-ckptmgr.jar
│ │ │ ├── [ 96] downloaders
│ │ │ │ └── [ 16M] heron-downloader.jar
│ │ │ ├── [ 96] healthmgr
│ │ │ │ └── [ 32M] heron-healthmgr.jar
│ │ │ ├── [ 96] instance
│ │ │ │ └── [3.3M] heron-instance.jar
│ │ │ ├── [ 96] metricscachemgr
│ │ │ │ └── [8.8M] heron-metricscachemgr.jar
│ │ │ ├── [ 96] metricsmgr
│ │ │ │ └── [6.9M] heron-metricsmgr.jar
│ │ │ ├── [ 128] packing
│ │ │ │ ├── [5.3M] heron-binpacking-packing.jar
│ │ │ │ └── [5.3M] heron-roundrobin-packing.jar
│ │ │ ├── [ 288] scheduler
│ │ │ │ ├── [ 16M] heron-kubernetes-scheduler.jar
│ │ │ │ ├── [7.5M] heron-local-scheduler.jar
│ │ │ │ ├── [7.5M] heron-marathon-scheduler.jar
│ │ │ │ ├── [9.4M] heron-mesos-scheduler.jar
│ │ │ │ ├── [ 13M] heron-nomad-scheduler.jar
│ │ │ │ ├── [7.3M] heron-scheduler.jar
│ │ │ │ └── [7.5M] heron-slurm-scheduler.jar
│ │ │ ├── [ 160] statefulstorage
│ │ │ │ ├── [ 18M] heron-dlog-statefulstorage.jar
│ │ │ │ ├── [5.0M] heron-hdfs-statefulstorage.jar
│ │ │ │ └── [5.0M] heron-localfs-statefulstorage.jar
│ │ │ └── [ 128] statemgr
│ │ │ ├── [5.3M] heron-localfs-statemgr.jar
│ │ │ └── [6.6M] heron-zookeeper-statemgr.jar
│ │ └── [ 285] release.yaml
│ └── [186M] heron-core.tar.gz
├── [ 512] examples
│ ├── [2.0K] fibonacci.yaml
│ ├── [3.3M] heron-api-examples.jar
│ ├── [4.4M] heron-eco-examples.jar
│ ├── [1.6K] heron-stateful-windowing.yaml
│ ├── [1.2K] heron-stateful-word-count.yaml
│ ├── [ 12M] heron-streamlet-examples.jar
│ ├── [ 31M] heron-streamlet-scala-examples.jar
│ ├── [2.1K] heron_fibonacci.yaml
│ ├── [1.9K] heron_windowing.yaml
│ ├── [2.0K] heron_wordcount.yaml
│ ├── [ 860] sample.properties
│ ├── [1.8K] simple_windowing.yaml
│ ├── [2.0K] simple_wordcount.yaml
│ └── [4.4M] storm-eco-examples.jar
├── [ 352] include
│ ├── [ 288] bolt
│ │ ├── [2.0K] base-basic-bolt.h
│ │ ├── [1.3K] base-rich-bolt.h
│ │ ├── [2.2K] basic-bolt-collector.h
│ │ ├── [2.8K] ibasic-bolt.h
│ │ ├── [3.0K] ibolt-output-collector.h
│ │ ├── [4.3K] ibolt.h
│ │ └── [1.2K] irich-bolt.h
│ ├── [ 96] config
│ │ └── [ 12K] config.h
│ ├── [ 160] exceptions
│ │ ├── [1.2K] already-alive-exception.h
│ │ ├── [1.3K] invalid-topology-exception.h
│ │ └── [1.3K] serialization-exception.h
│ ├── [ 320] metric
│ │ ├── [1.4K] assignable-metric.h
│ │ ├── [1.5K] count-metric.h
│ │ ├── [1.2K] imetric.h
│ │ ├── [1.6K] imetrics-registrar.h
│ │ ├── [1.3K] imulti-metric.h
│ │ ├── [1.6K] mean-metric.h
│ │ ├── [1.9K] multi-count-metric.h
│ │ └── [1.9K] multi-mean-metric.h
│ ├── [ 192] serializer
│ │ ├── [1.6K] cereal-serializer.h
│ │ ├── [1.9K] ipluggable-serializer.h
│ │ ├── [1.5K] string-serializer.h
│ │ └── [6.2K] tuple-serializer-utils.h
│ ├── [ 192] spout
│ │ ├── [1.4K] base-rich-spout.h
│ │ ├── [1.3K] irich-spout.h
│ │ ├── [2.4K] ispout-output-collector.h
│ │ └── [5.6K] ispout.h
│ ├── [ 192] topology
│ │ ├── [1.4K] base-rich-spout.h
│ │ ├── [1.3K] irich-spout.h
│ │ ├── [2.4K] ispout-output-collector.h
│ │ └── [5.6K] ispout.h
│ ├── [ 128] tuple
│ │ ├── [2.9K] fields.h
│ │ └── [2.8K] tuple.h
│ └── [ 96] utils
│ └── [2.4K] utils.h
├── [ 384] lib
│ ├── [ 224] api
│ │ ├── [ 43K] api-scala.jar
│ │ ├── [ 46M] heron-apiserver.jar
│ │ ├── [110K] libapi-java.jar
│ │ ├── [ 60K] libcxx-api.a
│ │ └── [ 61K] libcxx-api.pic.a
│ ├── [ 96] downloaders
│ │ └── [ 16M] heron-downloader.jar
│ ├── [ 96] metricscachemgr
│ │ └── [8.8M] heron-metricscachemgr.jar
│ ├── [ 128] packing
│ │ ├── [5.3M] heron-binpacking-packing.jar
│ │ └── [5.3M] heron-roundrobin-packing.jar
│ ├── [ 416] scheduler
│ │ ├── [7.5M] heron-aurora-scheduler.jar
│ │ ├── [5.3M] heron-binpacking-packing.jar
│ │ ├── [ 16M] heron-kubernetes-scheduler.jar
│ │ ├── [7.5M] heron-local-scheduler.jar
│ │ ├── [7.5M] heron-marathon-scheduler.jar
│ │ ├── [9.4M] heron-mesos-scheduler.jar
│ │ ├── [ 13M] heron-nomad-scheduler.jar
│ │ ├── [5.3M] heron-roundrobin-packing.jar
│ │ ├── [7.3M] heron-scheduler.jar
│ │ ├── [7.5M] heron-slurm-scheduler.jar
│ │ └── [9.0M] heron-yarn-scheduler.jar
│ ├── [ 96] simulator
│ │ └── [ 45K] libsimulator-java.jar
│ ├── [ 128] statemgr
│ │ ├── [5.3M] heron-localfs-statemgr.jar
│ │ └── [6.6M] heron-zookeeper-statemgr.jar
│ ├── [ 160] third_party
│ │ ├── [1.3M] libprotobuf_java.jar
│ │ ├── [ 29K] slf4j-api-1.7.7.jar
│ │ └── [7.7K] slf4j-jdk14-1.7.7.jar
│ └── [ 320] uploader
│ ├── [ 16M] heron-dlog-uploader.jar
│ ├── [4.8M] heron-gcs-uploader.jar
│ ├── [2.5M] heron-hdfs-uploader.jar
│ ├── [3.9M] heron-http-uploader.jar
│ ├── [2.5M] heron-localfs-uploader.jar
│ ├── [2.5M] heron-null-uploader.jar
│ ├── [7.2M] heron-s3-uploader.jar
│ └── [2.5M] heron-scp-uploader.jar
└── [ 285] release.yaml
53 directories, 253 files
On Sun, Mar 10, 2019 at 11:54 PM Ning Wang <wa...@gmail.com> wrote:
> Thanks Dave!
>
> Yeah. I will get a list of content in packages.
>
> For the docker image, I think it should be ok. Let me try to publish it to
> Apache docker hub and see if there is anything missing.
>
>
>
> On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <da...@comcast.net> wrote:
>
>> Hi -
>>
>> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wa...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I have been trying to release Heron 0.20.1 (being distracted time by
>> time
>> > though) and the most recent question I am having is where to put the
>> binary
>> > packages.
>> >
>> > The binary packages are (when we were doing releases on github):
>> > - tar.gz packages for osx, centos and ubuntu, each one includes all
>> modules
>> > like core, lib, tools, etc.
>> > - .sh packages for the three platforms. which is an installer for the
>> > modules in the tar.gz packages.
>> > - docker image (dockerhub, not github most of the times)
>>
>> Let’s discuss the components in each binary package and how big that they
>> really are and need to be.
>>
>> >
>> > Currently each package is more than 400MB.
>>
>> When packages of this size are released from dev to the release area it
>> requires replication to the Apache Mirror system. When the size exceeds an
>> aggregate of 1GB then Infra needs to manually handle things to avoid
>> impacting the mirrors. (There are 250 projects using the mirrors.)
>>
>>
>>
>> >
>> > I was trying to understand the Apache rules and my impression was that
>> > these package should be on dist.apache.org like the src packages (I
>> might
>> > be wrong about the rules though) and it looks like Apache Storm has a
>> > binary package in their release.
>>
>> Make the case for Heron without comparison to other projects.
>>
>> There is a place to make Apache Docker releases on docker hub. Let’s
>> figure out this if it is a valid distribution that Heron could make.
>>
>> >
>> > However it seems Apache infra has a byte limit of 500MB for each
>> release. I
>> > guess it means that the binary packages are not "required" to be on
>> Apache
>> > infra?
>>
>> Yes and no. Let’s discuss the packages first.
>>
>> >
>> > The binary packages are convenient for users. So I think they should be
>> > included in release. The question is where should we put them?
>> >
>> > So far it looks like the options are:
>> > - ask for an exception and publish them to dist.apache.org. It seems
>> like
>> > Apache infra guys don't suggest this solution.
>> > - publish only src package to dist.apache.org and publish the binary
>> > packages on github (or is there any other suggestion?). This is
>> convenient
>> > for us and there is no problem so far (we have binary packages for all
>> the
>> > previous releases and github hasn't complained). The question about this
>> > option is more like if this is acceptable (or ever better suggested) by
>> > Apache?
>> >
>> > What do you think about the two options above and any other options we
>> > should consider?
>>
>> Let’s discuss the packages.
>>
>> Next we will also need to discuss the website.
>>
>> >
>> > Thanks.
>> > --ning
>>
>>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Ning Wang <wa...@gmail.com>.
Thanks Dave!
Yeah. I will get a list of content in packages.
For the docker image, I think it should be ok. Let me try to publish it to
Apache docker hub and see if there is anything missing.
On Fri, Mar 8, 2019 at 2:14 PM Dave Fisher <da...@comcast.net> wrote:
> Hi -
>
> > On Mar 8, 2019, at 1:58 PM, Ning Wang <wa...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have been trying to release Heron 0.20.1 (being distracted time by time
> > though) and the most recent question I am having is where to put the
> binary
> > packages.
> >
> > The binary packages are (when we were doing releases on github):
> > - tar.gz packages for osx, centos and ubuntu, each one includes all
> modules
> > like core, lib, tools, etc.
> > - .sh packages for the three platforms. which is an installer for the
> > modules in the tar.gz packages.
> > - docker image (dockerhub, not github most of the times)
>
> Let’s discuss the components in each binary package and how big that they
> really are and need to be.
>
> >
> > Currently each package is more than 400MB.
>
> When packages of this size are released from dev to the release area it
> requires replication to the Apache Mirror system. When the size exceeds an
> aggregate of 1GB then Infra needs to manually handle things to avoid
> impacting the mirrors. (There are 250 projects using the mirrors.)
>
>
>
> >
> > I was trying to understand the Apache rules and my impression was that
> > these package should be on dist.apache.org like the src packages (I
> might
> > be wrong about the rules though) and it looks like Apache Storm has a
> > binary package in their release.
>
> Make the case for Heron without comparison to other projects.
>
> There is a place to make Apache Docker releases on docker hub. Let’s
> figure out this if it is a valid distribution that Heron could make.
>
> >
> > However it seems Apache infra has a byte limit of 500MB for each
> release. I
> > guess it means that the binary packages are not "required" to be on
> Apache
> > infra?
>
> Yes and no. Let’s discuss the packages first.
>
> >
> > The binary packages are convenient for users. So I think they should be
> > included in release. The question is where should we put them?
> >
> > So far it looks like the options are:
> > - ask for an exception and publish them to dist.apache.org. It seems
> like
> > Apache infra guys don't suggest this solution.
> > - publish only src package to dist.apache.org and publish the binary
> > packages on github (or is there any other suggestion?). This is
> convenient
> > for us and there is no problem so far (we have binary packages for all
> the
> > previous releases and github hasn't complained). The question about this
> > option is more like if this is acceptable (or ever better suggested) by
> > Apache?
> >
> > What do you think about the two options above and any other options we
> > should consider?
>
> Let’s discuss the packages.
>
> Next we will also need to discuss the website.
>
> >
> > Thanks.
> > --ning
>
>
Re: [DISCUSS] Hosting Heron binary release packages
Posted by Dave Fisher <da...@comcast.net>.
Hi -
> On Mar 8, 2019, at 1:58 PM, Ning Wang <wa...@gmail.com> wrote:
>
> Hi,
>
> I have been trying to release Heron 0.20.1 (being distracted time by time
> though) and the most recent question I am having is where to put the binary
> packages.
>
> The binary packages are (when we were doing releases on github):
> - tar.gz packages for osx, centos and ubuntu, each one includes all modules
> like core, lib, tools, etc.
> - .sh packages for the three platforms. which is an installer for the
> modules in the tar.gz packages.
> - docker image (dockerhub, not github most of the times)
Let’s discuss the components in each binary package and how big that they really are and need to be.
>
> Currently each package is more than 400MB.
When packages of this size are released from dev to the release area it requires replication to the Apache Mirror system. When the size exceeds an aggregate of 1GB then Infra needs to manually handle things to avoid impacting the mirrors. (There are 250 projects using the mirrors.)
>
> I was trying to understand the Apache rules and my impression was that
> these package should be on dist.apache.org like the src packages (I might
> be wrong about the rules though) and it looks like Apache Storm has a
> binary package in their release.
Make the case for Heron without comparison to other projects.
There is a place to make Apache Docker releases on docker hub. Let’s figure out this if it is a valid distribution that Heron could make.
>
> However it seems Apache infra has a byte limit of 500MB for each release. I
> guess it means that the binary packages are not "required" to be on Apache
> infra?
Yes and no. Let’s discuss the packages first.
>
> The binary packages are convenient for users. So I think they should be
> included in release. The question is where should we put them?
>
> So far it looks like the options are:
> - ask for an exception and publish them to dist.apache.org. It seems like
> Apache infra guys don't suggest this solution.
> - publish only src package to dist.apache.org and publish the binary
> packages on github (or is there any other suggestion?). This is convenient
> for us and there is no problem so far (we have binary packages for all the
> previous releases and github hasn't complained). The question about this
> option is more like if this is acceptable (or ever better suggested) by
> Apache?
>
> What do you think about the two options above and any other options we
> should consider?
Let’s discuss the packages.
Next we will also need to discuss the website.
>
> Thanks.
> --ning