You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gobblin.apache.org by ab...@apache.org on 2018/01/03 18:57:59 UTC
[1/5] incubator-gobblin git commit: [GOBBLIN-355] Updated CHANGELOG
for 0.12.0 release
Repository: incubator-gobblin
Updated Branches:
refs/heads/0.12.0 f2e7c0647 -> 693c7f602
[GOBBLIN-355] Updated CHANGELOG for 0.12.0 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/commit/50b0bcfb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/tree/50b0bcfb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/diff/50b0bcfb
Branch: refs/heads/0.12.0
Commit: 50b0bcfb414b8355fe45ce5c0f5b151ca0024172
Parents: f2e7c06
Author: Abhishek Tiwari <ab...@gmail.com>
Authored: Wed Jan 3 19:27:04 2018 +0530
Committer: Abhishek Tiwari <ab...@gmail.com>
Committed: Wed Jan 3 19:27:04 2018 +0530
----------------------------------------------------------------------
CHANGELOG.md | 220 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 220 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/50b0bcfb/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e697552..c6c262f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,223 @@
+GOBBLIN 0.12.0
+-------------
+
+###Created Date: 1/03/2018
+
+## HIGHLIGHTS
+
+* First Apache Release.
+* Improved Gobblin-as-a-Service.
+* Improved Global Throttling.
+* Improved Gobblin Cluster.
+* Enhanced stream processing.
+* New Converters: JsonToParquet, GrokToJson, JsonToAvro.
+* New Sources: RegexPartitionedAvroFileSource, new SalesforceWriter.
+* New Extractors: PostgresqlExtractor, EnvelopePayloadExtractor.
+* New Writers: ParquetHdfsDataWriter, eventually consistent FS support.
+
+## NEW FEATURES
+
+* [State Store] [GOBBLIN-199] GOBBLIN-56 Add state store entry listing API
+* [State Store] [GOBBLIN-200] GOBBLIN-56 State store dataset cleaner using state store listing API
+* [Extractor] [GOBBLIN-203] Postgresql Extractor
+* [Extractor] [GOBBLIN-238] Implement EnvelopePayloadExtractor and EnvelopePayloadDeserializer
+* [Converter] [GOBBLIN-248] Converter for Json to Parquet
+* [Converter] [GOBBLIN-231] Grok to Json Converter
+* [Converter] [GOBBLIN-221] Add Json to Avro converter
+* [Writer] [GOBBLIN-255] ParquetHdfsDataWriter
+* [Writer] [GOBBLIN-36] New salesforce writer
+* [Encryption] [GOBBLIN-224] Gobblin doesn't support keyring based GPG file decryption
+* [Kafka] [GOBBLIN-190] Kafka Sink replication factor and partition creation.
+* [Avro-to-ORC] [GOBBLIN-181] Modify Avro2ORC flow to materialize Hive views
+
+## IMPROVEMENTS
+
+* [GaaS] [GOBBLIN-232] Create Azkaban Orchestrator for Gobblin-as-a-Service
+* [GaaS] [GOBBLIN-213] Add scheduler service to GobblinServiceManager
+* [GaaS] [GOBBLIN-3] Implementation of Flow compiler with multiple hops
+* [GaaS] [GOBBLIN-280] Add new SpecCompiler compatible constructor to AzkabanSpecExecutor
+* [GaaS] [GOBBLIN-299] Add deletion support to Azkaban Orchestrator
+* [GaaS] [GOBBLIN-262] Make multihopcompiler use the first user specified template
+* [GaaS] [GOBBLIN-204] Add a service that fetches GaaS flow configs from a git repository
+* [GaaS] [GOBBLIN-292] Add kafka09 support for service and cluster job spec communication
+* [GaaS] [GOBBLIN-281] Fix logging in gobblin-service
+* [GaaS] [GOBBLIN-273] Add failure monitoring
+* [GaaS] [GOBBLIN-304] Remove versioning from Gobblin-as-a-Service flow specs
+* [Global Throttling] [GOBBLIN-334] Implement SharedResourceFactory for LineageInfo
+* [Global Throttling] [GOBBLIN-287] Support service-level throttling quotas
+* [Global Throttling] [GOBBLIN-264] Add a SharedResourceFactory for creating shared DataPublishers
+* [Global Throttling] [GOBBLIN-251] Having UpdateProviderFactory able to instantiate FileSystem with URI
+* [Global Throtlting] [GOBBLIN-236] Add a ControlMessage injector as a RecordStreamProcessor
+* [Global Throttling] [GOBBLIN-24] Allow disabling global throttling. Fix a race condition in BatchedPer…
+* [Cluster] [GOBBLIN-329] Add a basic cluster integration test
+* [Cluster] [GOBBLIN-325] Add a Source and Extractor for stress testing
+* [Cluster] [GOBBLIN-324] Add a configuration to configure the cluster working directory
+* [Cluster] [GOBBLIN-257] Remove old jobs' run data
+* [Cluster] [GOBBLIN-202] Add better metrics to gobblin to support AWS autoscaling
+* [Cluster] [GOBBLIN-320] Add metrics to GobblinHelixJobScheduler
+* [Cluster] [GOBBLIN-185] Design for gobblin job level gracefully shutdown
+* [Cluster] [GOBBLIN-11] Fix for #1822 and #1823
+* [Cluster] [GOBBLIN-10] Fix_for_#1850_and_#1851
+* [Cluster] [GOBBLIN-349] Add guages for gobblin cluster metrics
+* [Core] [GOBBLIN-177] Allow error limit to skip records which are not convertible
+* [Core] [GOBBLIN-333] Remove reference to log4j in WriterUtils
+* [Core] [GOBBLIN-332] Implement fetching hive tokens in tokenUtils
+* [Core] [GOBBLIN-330] Generate Kerberos Principal dynamically
+* [Core] [GOBBLIN-319] Add DatasetResolver to transform raw Gobblin dataset to application specific dataset
+* [Core] [GOBBLIN-317] Add dynamic configuration injection in the mappers
+* [Core] [GOBBLIN-310] Skip rerunning completed tasks on mapper reattempts
+* [Core] [GOBBLIN-300] Use 1.7.7 form of Schema.createUnion() API that takes in a list
+* [Core] [GOBBLIN-294] Change logging level of refection utilities
+* [Core] [GOBBLIN-271] Move the grok converter to the gobblin-grok module
+* [Core] [GOBBLIN-252] Add some azkaban related constants
+* [Core] [GOBBLIN-240] Adding three more Azkaban tags
+* [Core] [GOBBLIN-186] Add support for using the Kerberos authentication plugin without a GobblinDriverInstance
+* [Core] [GOBBLIN-179] Make migrated Gobblin code work with old state files
+* [Core] [GOBBLIN-178] Migrate Gobblin codebase from gobblin to org.apache.gobblin package
+* [State Store] [GOBBLIN-335] Increase blob size in MySQL state store
+* [State Store] [GOBBLIN-270] State Migration script
+* [State Store] [GOBBLIN-230] Convert old package name to new name in old states
+* [Source] [GOBBLIN-296] Kafka json source and writer
+* [Source] [GOBBLIN-245] Create topic specific extract of a WorkUnit in KafkaSource
+* [Source] [GOBBLIN-210] Implement a source based on Dataset Finder
+* [Extractor] [GOBBLIN-197] Modify JDBCExtractor to support reading clob columns as strings
+* [Converter] [GOBBLIN-228] Add config property to ignore fields in JsonRecordAvroSchemaToAvroConverter
+* [Converter] [GOBBLIN-226] Nested schema support in JsonStringToJsonIntermediateConverter and JsonIntermediateToAvroConverter
+* [Writer] [GOBBLIN-314] Validate filesize when copying in writer
+* [Writer] [GOBBLIN-171] Add a writer wrapper that closes the wrapped writer and creates a new one
+* [Writer] [GOBBLIN-6] Support eventual consistent filesystems like S3
+* [Compaction] [GOBBLIN-354] Support DynamicConfig in AzkabanCompactionJobLauncher
+* [Retention] [GOBBLIN-348] Hdfs Modified Time based Version Finder for Hive Tables
+* [Hive-Registration] [GOBBLIN-342] Option to set hive metastore uri in Hiveregister
+* [Kafka] [GOBBLIN-331] Add sharedConfig support for the KafkaDataWriters
+* [Kafka] [GOBBLIN-312] Pass extra kafka configuration to the KafkaConsumer in KafkaSimpleStreamingSource
+* [Kafka] [GOBBLIN-198] Configuration to disable switching the Kafka topic's and Avro schema's names before registering schema
+* [Kafka] [GOBBLIN-195] Ability to switch Avro schema namespace switch before registering with Kafka Avro Schema registry
+* [Avro-to-ORC] [GOBBLIN-313] Option to explicitly set group name for staging and final destination directories for Avro-To-Orc conversion
+* [Avro-to-ORC] [GOBBLIN-297] Changing access modifier to Protected for HiveSource and Watermarker classes
+* [Metrics] [GOBBLIN-326] Gobblin metrics constructor only provides default constructor for Codhale metrics
+* [Metrics] [GOBBLIN-189] Add additional information in events for gobblintrackingevent_distcp_ng to show published dataset path
+* [Metrics] [GOBBLIN-307] Implement lineage event as LineageEventBuilder in gobblin
+* [Metrics] [GOBBLIN-261] Add kafka lineage event
+* [Metrics] [GOBBLIN-182] Emit Lineage Events for Query Based Sources
+* [Metrics] [GOBBLIN-22] Graphite prefix in configuration
+* [Salesforce] [GOBBLIN-288] Add finer-grain dynamic partition generation for Salesforce
+* [Salesforce] [GOBBLIN-265] Add support for PK chunking to gobblin-salesforce
+* [Compaction] [GOBBLIN-256] Improve logging for gobblin compaction
+* [Hive Registration] [GOBBLIN-266] Improve Hive Task setup
+* [Hive Registration] [GOBBLIN-253] Hive materializer enhancements
+* [Hive Registration] [GOBBLIN-172] Pipelined Hive Registration thru. TastStateCollectorService
+* [Config] [GOBBLIN-209] Add support for HOCO global files
+* [DistcpNG] [GOBBLIN-173] Add pattern support for job-level blacklist in distcpNG/replication
+* [DistcpNG] [GOBBLIN-8] Add simple distcp job publishing to S3 as an example
+* [DistcpNG] [GOBBLIN-5] Make Watermark checking configurable in distcpNG-replication
+* [Documentation] [GOBBLIN-282] Support templates on Gobblin Azkaban launcher
+* [Documentation] [GOBBLIN-170] Updating documentation to include Apache with Gobblin
+* [Documentation] [GOBBLIN-25] Gobblin data-management run script and example configuration
+* [Documentation] [GOBBLIN-339] Example to illustrate how to build custom source and extractor in Gobblin.
+* [Documentation] [GOBBLIN-305] Add csv-kafka and kafka-hdfs template
+* [Apache] [GOBBLIN-169] Ability to curate licenses of all Gobblin dependencies
+* [Apache] [GOBBLIN-168] Standardize Github PR template for Gobblin
+* [Apache] [GOBBLIN-167] Add dev tooling for signing releases
+* [Apache] [GOBBLIN-166] Add dev tooling for simplifying the Github PR workflow
+* [Apache] [GOBBLIN-163] Setup Wiki for Gobblin
+* [Apache] [GOBBLIN-162] Setup new PR process for Gobblin
+* [Apache] [GOBBLIN-161] Migrate all Gobblin issues from Github to Apache
+* [Apache] [GOBBLIN-160] Move mailing lists to Apache
+* [Apache] [GOBBLIN-65] Add com.linkedin.gobblin to alias resolver
+* [Apache] [GOBBLIN-38] Create workunitstream for CompactionSource
+* [Apache] [GOBBLIN-2] Setup Apache Gobblin's website
+* [Apache] [GOBBLIN-1] Move Gobblin codebase to Apache
+* [AdminUI] [GOBBLIN-9] Improve AdminUI and RestService with better sorting, filtering, auto-updates, etc.
+* [Streaming] [GOBBLIN-4] Added control messages to Gobblin stream.
+
+## BUGS FIXES
+
+* [Bug] [GOBBLIN-353] Fix low watermark overridden by high watermark in SalesforceSource
+* [Bug] [GOBBLIN-347] KafkaPusher is not closed when GobblinMetrics.stopReporting is called
+* [Bug] [GOBBLIN-344] Fix help method getResolver in LineageInfo is private
+* [Bug] [GOBBLIN-343] Table and db regexp does not work in HiverRegistrationPolicyBase
+* [Bug] [GOBBLIN-341] Fix logger name to correct class prefix after apache package change
+* [Bug] [GOBBLIN-338] HiveAvroManagerSerde failed if external table was on different fs
+* [Bug] [GOBBLIN-337] HiveConf token signature bug
+* [Bug] [GOBBLIN-328] GobblinClusterKillTest failed. Not able to find expected output files.
+* [Bug] [GOBBLIN-322] Cluster mode failed to start. Failed to find a log4j config file
+* [Bug] [GOBBLIN-321] CSV to HDFS ISSUE
+* [Bug] [GOBBLIN-315] Fix shaded avro is used in LineageEventBuilder
+* [Bug] [GOBBLIN-309] Bug fixing for contention of adding jar file into HDFS
+* [Bug] [GOBBLIN-308] Gobblin cluster bootup hangs
+* [Bug] [GOBBLIN-306] Exception when using fork followed by converters with EmbeddedGoblin
+* [Bug] [GOBBLIN-303] Compaction can generate zero sized output when MR is in speculative mode
+* [Bug] [GOBBLIN-301] Fix the key GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS
+* [Bug] [GOBBLIN-295] Make missing nullable fields default to null in json to avro converter
+* [Bug] [GOBBLIN-291] Remove unnecessary listing and reading of flowSpecs
+* [Bug] [GOBBLIN-289] Gobblin only partially decrypt the PGP file using keyring
+* [Bug] [GOBBLIN-286] Fix bug where non hive dataset throw NPE during dataset publish
+* [Bug] [GOBBLIN-285] KafkaExtractor does not compute avgMillisPerRecord when partition pull is interrupted
+* [Bug] [GOBBLIN-284] Add retry in SalesforceExtractor to handle transient network errors
+* [Bug] [GOBBLIN-283] Refactor EnvelopePayloadConverter to support multi fields conversion
+* [Bug] [GOBBLIN-279] pull file unable to reuse the json property.
+* [Bug] [GOBBLIN-278] Fix sending lineage event for KafkaSource
+* [Bug] [GOBBLIN-276] Change setActive order to prevent flow spec loss
+* [Bug] [GOBBLIN-275] Use listStatus instead of globStatus for finding persisted files
+* [Bug] [GOBBLIN-274] Fix wait for salesforce batch completion
+* [Bug] [GOBBLIN-268] Unique job uri and job name generation for GaaS
+* [Bug] [GOBBLIN-267] HiveSource creates workunit even when update time is before maxLookBackDays
+* [Bug] [GOBBLIN-263] TaskExecutor metrics are calculated incorrectly
+* [Bug] [GOBBLIN-260] Salesforce dynamic partitioning bugs
+* [Bug] [GOBBLIN-259] Support writing Kafka messages to db/table file path
+* [Bug] [GOBBLIN-258] Try to remove the tmp output path from wrong fs before compaction
+* [Bug] [GOBBLIN-254] Add config key to update watermark when a partition is empty
+* [Bug] [GOBBLIN-247] avro-to-orc conversion validation job should fail only on data mismatch
+* [Bug] [GOBBLIN-244] Need additional info for gobblin tracking hourly-deduped
+* [Bug] [GOBBLIN-241] Allow multiple datasets send different lineage event for kafka
+* [Bug] [GOBBLIN-237] Update property names in JsonRecordAvroSchemaToAvroConverter
+* [Bug] [GOBBLIN-235] Prevent log warnings when TaskStateCollectorService has no task states detected
+* [Bug] [GOBBLIN-234] Add a ControlMessageInjector that generates metadata update control messages
+* [Bug] [GOBBLIN-233] Add concurrent map to avoid multiple job submission from GobblinHelixJobScheduler
+* [Bug] [GOBBLIN-229] Gobblin cluster doesn't clean up job state file upon job completion
+* [Bug] [GOBBLIN-225] Fix cloning of ControlMessages in PartitionDataWriterMessageHandler
+* [Bug] [GOBBLIN-223] CsvToJsonConverter should throw DataConversionException
+* [Bug] [GOBBLIN-222] Fix silent failure in loading incompatible state store
+* [Bug] [GOBBLIN-220] FileAwareInputDataStreamWriter only logs file names when a copy completes successfully
+* [Bug] [GOBBLIN-219] Check for copyright header
+* [Bug] [GOBBLIN-218] Ensure runImmediately is honored in Gobblin as a Service
+* [Bug] [GOBBLIN-217] Fix gobblin-admin module to use correct idString
+* [Bug] [GOBBLIN-215] hasJoinOperation failed when SQL statement has limit keyword
+* [Bug] [GOBBLIN-214] Filtering doesn't work in FileListUtils:listFilesRecursively
+* [Bug] [GOBBLIN-212] Exception handling of TaskStateCollectorServiceHandler
+* [Bug] [GOBBLIN-208] JobCatalogs should fallback to system configuration
+* [Bug] [GOBBLIN-206] Remove extra close of CloseOnFlushWriterWrapper
+* [Bug] [GOBBLIN-205] Fix Replication bug in Push Mode
+* [Bug] [GOBBLIN-194] NPE in BaseDataPublisher if writer partitions are enabled and metadata filename is not set
+* [Bug] [GOBBLIN-193] AbstractAvroToOrcConverter throws NoObjectException when trying to fetch partition info from table when partition doesn't exist
+* [Bug] [GOBBLIN-192] Gobblin AWS hardcodes the log4j config
+* [Bug] [GOBBLIN-191] Make sure cron scheduler works and tune schedule period
+* [Bug] [GOBBLIN-184] Call the flush method of CloseOnFlushWriterWrapper when a FlushControlMessage is received
+* [Bug] [GOBBLIN-183] Gobblin data management copy empty directories
+* [Bug] [GOBBLIN-176] Gobblin build is failing with missing dependency jetty-http
+* [Bug] [GOBBLIN-175] String is not escaped while creating hive query for avro_to_orc conversion.
+* [Bug] [GOBBLIN-174] fix distcp-ng so it does not remove existing target files
+* [Bug] [GOBBLIN-165] Fix URI is not absolute issue in SFTP
+* [Bug] [GOBBLIN-159] Gobblin Cluster graceful shutdown of master and workers
+* [Bug] [GOBBLIN-129] AdminUI performs too many requests when update is pressed
+* [Bug] [GOBBLIN-127] Admin UI duration chart is sorted incorrectly
+* [Bug] [GOBBLIN-109] Remove need for current.jst
+* [Bug] [GOBBLIN-87] Gobblin runOnce not working correctly
+* [Bug] [GOBBLIN-79] Add config to specify database for JDBC source
+* [Bug] [GOBBLIN-54] How to use oozie to schedule gobblin with mapreduce mode, not the local mode
+* [Bug] [GOBBLIN-48] java.lang.IllegalArgumentException when using extract.limit.enabled
+* [Bug] [GOBBLIN-40] Job History DB Schema had not been updated to reflect new LauncherType
+* [Bug] [GOBBLIN-39] JobHistoryDB migration files have been incorrectly modified
+* [Bug] [GOBBLIN-37] Gobblin-Master Build failed
+* [Bug] [GOBBLIN-33] StateStores persists Task and WorkUnit state to state.store.fs.uri
+* [Bug] [GOBBLIN-32] StateStores created with rootDir that is incompatible with state.store.type
+* [Bug] [GOBBLIN-31] Reflections concurrency issue
+* [Bug] [GOBBLIN-30] Reflections errors when scanning classpath and encountering missing/invalid file paths.
+* [Bug] [GOBBLIN-29] GobblinHelixJobScheduler should be able to be run without default configuration manager
+* [Bug] [GOBBLIN-27] SQL Server - incomplete JDBC URL
+
+
GOBBLIN 0.11.0
-------------
[5/5] incubator-gobblin git commit: [GOBBLIN-355] Add helper script
to publish archives on Nexus, and wire gradle tasks
Posted by ab...@apache.org.
[GOBBLIN-355] Add helper script to publish archives on Nexus, and wire gradle tasks
Project: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/commit/693c7f60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/tree/693c7f60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/diff/693c7f60
Branch: refs/heads/0.12.0
Commit: 693c7f60277b4f2423c4ea7a9d4cc429c7c77e23
Parents: fbc4cba
Author: Abhishek Tiwari <ab...@gmail.com>
Authored: Thu Jan 4 00:27:42 2018 +0530
Committer: Abhishek Tiwari <ab...@gmail.com>
Committed: Thu Jan 4 00:27:42 2018 +0530
----------------------------------------------------------------------
build.gradle | 7 +++
maven-nexus/maven-install.sh | 58 ++++++++++++++++++
maven-nexus/maven-nexus.sh | 115 ++++++++++++++++++++++++++++++++++++
maven-nexus/upload-to-nexus.sh | 71 ++++++++++++++++++++++
4 files changed, 251 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/693c7f60/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index 74a64e7..0eae590 100644
--- a/build.gradle
+++ b/build.gradle
@@ -22,6 +22,8 @@ buildscript {
apply from: 'gradle/scripts/defaultBuildProperties.gradle'
apply from: 'gradle/scripts/computeVersions.gradle'
+ apply from: file('gradle/scripts/buildscript.gradle'), to: buildscript
+
buildscript.repositories.addAll(project.repositories)
dependencies {
@@ -128,9 +130,11 @@ apply from: 'gradle/scripts/javadoc.gradle'
apply from: 'gradle/scripts/sourcesJar.gradle'
apply from: 'gradle/scripts/mavenPublishing.gradle'
+apply from: 'gradle/scripts/nexusPublishing.gradle'
apply from: 'gradle/scripts/javaVersionCheck.gradle'
apply from: 'gradle/scripts/rat.gradle'
+apply from: 'gradle/scripts/release.gradle'
task wrapper(type: Wrapper) { gradleVersion = '2.13' }
@@ -209,6 +213,9 @@ rat {
'**/grok/**',
'**/WebmasterPerformanceTuningMetrics',
'**/*.template',
+ '**/mainGeneratedRest/**',
+ '**/mainGeneratedDataTemplate/**',
+ '**/gen-java/**',
'**/package-list'
]
}
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/693c7f60/maven-nexus/maven-install.sh
----------------------------------------------------------------------
diff --git a/maven-nexus/maven-install.sh b/maven-nexus/maven-install.sh
new file mode 100755
index 0000000..d31064f
--- /dev/null
+++ b/maven-nexus/maven-install.sh
@@ -0,0 +1,58 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#group is overiden to support forked repositories
+
+function print_usage(){
+ echo "maven-install.sh --version VERSION [--group GROUP]"
+}
+
+for i in "$@"
+do
+ case "$1" in
+ --version)
+ VERSION="$2"
+ shift
+ ;;
+ --group)
+ GROUP="$2"
+ shift
+ ;;
+ --help)
+ print_usage
+ exit 0
+ ;;
+ *)
+ ;;
+ esac
+ shift
+done
+
+if [ -z "$VERSION" ]; then
+ print_usage
+ exit
+fi
+echo VERSION=$VERSION
+
+if [ -z "$GROUP" ]; then
+ GROUP="org.apache.gobblin"
+fi
+
+
+./gradlew install -Dorg.gradle.parallel=false -Pversion=$VERSION -Pgroup=$GROUP
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/693c7f60/maven-nexus/maven-nexus.sh
----------------------------------------------------------------------
diff --git a/maven-nexus/maven-nexus.sh b/maven-nexus/maven-nexus.sh
new file mode 100755
index 0000000..1e636c0
--- /dev/null
+++ b/maven-nexus/maven-nexus.sh
@@ -0,0 +1,115 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+script_dir=$(dirname $0)
+script_name=$(basename $0)
+GRADLE="$script_dir/../gradlew"
+
+function print_usage() {
+ echo -e "USAGE: $0 [-remote|-local] [-noclean] [gradle_args]"
+ echo
+ echo -e "Publishes signed maven artifacts locally ($HOME/.m2/repository) or remotely (Nexus)."
+ echo -e "\t-local Publish to local repository"
+ echo -e "\t-noclean Don't run gradlew clean (useful if re-running)"
+ echo -e "\t-remote Publish to Nexus repository"
+ echo -e "\t-packages a comma-separated list of gradle paths to publish (e.g. :gobblin-api,:gobblin-core)"
+ echo
+ echo -e "NOTES:"
+ echo -e "\t1. You need the Gobblin PGP key to sign the artifacts. If you don't have it,"
+ echo -e "\t talk to another committer to get it. You also need to add the following to"
+ echo -e "\t your $HOME/.gradle/gradle.properties file:"
+ echo
+ echo -e "signing.keyId=<PGP key hex id>"
+ echo -e "signing.password=<PGP key password>"
+ echo -e "signing.secretKeyRingFile=$HOME/.gnupg/secring.gpg"
+ echo
+ echo -e "\t2. To upload remotely, you'll need a Nexus account. Visit "
+ echo -e "\t https://repository.apache.org/ to set it up. After"
+ echo -e "\t that add to your $HOME/.gradle/gradle.properties file:"
+ echo
+ echo -e "nexusUsername=<Nexus username>"
+ echo -e "nexusPassword=<Nexus password>"
+ echo
+ echo -e "\t3. Uploading remotely will upload only to the Nexus staging directory. "
+ echo -e "\t4. Don't forget to create a gobblin_<version> tag before publishing remotely!"
+ echo -e "\t5. Sometimes build with fail with an error"
+ echo -e "\t '... Failed to interpolate field: private java.lang.String ...'"
+ echo -e "\t Just re-run with -noclean"
+}
+
+if [ "$#" -eq 0 ] ; then
+ print_usage
+ exit
+fi
+
+install_target=
+gradle_args=
+noclean=
+declare -a packages
+
+# Parse command line
+while [ "$#" -gt 0 ] ; do
+ A="$1"
+ case "$A" in
+ -local)
+ install_target=install
+ ;;
+ -noclean)
+ noclean="1"
+ ;;
+ -remote)
+ install_target=uploadArchives
+ ;;
+ -h|--help|-help)
+ print_usage
+ exit
+ ;;
+ -packages)
+ shift
+ packages=( ${1//,/ } )
+ ;;
+ *)
+ gradle_args="$gradle_args $A"
+ ;;
+ esac
+ shift
+done
+
+if [ -z "${install_target}" ] ; then
+ echo "${script_name}: missing install target"
+ exit 1
+fi
+
+declare -a package_targets
+for P in "${packages[@]}" ; do
+ ptarget="$P:${install_target}"
+ if [ "${ptarget:0:1}" != ":" ] ; then
+ ptarget=":$ptarget"
+ fi
+ package_targets+=( "$ptarget" )
+done
+
+if [ "${#packages[@]}" -gt 0 ] ; then
+ install_target="${package_targets[@]}"
+fi
+
+if [ -z "$noclean" ] ; then
+ $GRADLE clean
+fi
+$GRADLE -PpublishToNexus -Porg.gradle.parallel=false -Porg.gradle.daemon=false -xtest $gradle_args $install_target 2>&1 | tee /tmp/${script_name}.out
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/693c7f60/maven-nexus/upload-to-nexus.sh
----------------------------------------------------------------------
diff --git a/maven-nexus/upload-to-nexus.sh b/maven-nexus/upload-to-nexus.sh
new file mode 100755
index 0000000..4847f22
--- /dev/null
+++ b/maven-nexus/upload-to-nexus.sh
@@ -0,0 +1,71 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+PROG=$(basename $0)
+
+function usage() {
+ echo -e "USAGE: $PROG"
+}
+
+# main()
+
+if [ "$#" -eq 0 ] ; then
+ usage
+ exit
+fi
+
+while [ "$#" -gt 0 ] ; do
+ A="$1"
+ case "$A" in
+ -h|--help)
+ usage
+ exit
+ ;;
+ *)
+ echo "$PROG: unknown option: $A"
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+echo "CLEANING"
+./gradlew clean
+
+upload_all=0
+
+for P in :gobblin-admin :gobblin-api :gobblin-azkaban :gobblin-compaction :gobblin-config-management:gobblin-config-core :gobblin-config-management:gobblin-config-client :gobblin-core :gobblin-data-management :gobblin-distribution :gobblin-example :gobblin-metastore :gobblin-metrics :gobblin-oozie :gobblin-rest-service:gobblin-rest-api :gobblin-rest-service:gobblin-rest-client :gobblin-rest-service:gobblin-rest-server :gobblin-runtime :gobblin-salesforce :gobblin-scheduler :gobblin-test :gobblin-test-harness :gobblin-utility :gobblin-yarn ; do
+ echo "----------------------------"
+ if [ "$upload_all" -eq 0 ] ; then
+ read -p "UPLOAD $P [y(es)|n(o)|a(ll)]" ans
+ ans=$(echo "$ans" | tr '[:upper:]' '[:lower:]')
+ if [ "$ans" == "a" -o "$ans" == "all" ] ; then
+ upload_all=1
+ ans="y"
+ fi
+ else
+ ans="y"
+ fi
+ if [ "$ans" == "y" -o "$ans" == "yes" ] ; then
+ ./maven-nexus/maven-nexus.sh -remote -noclean -packages $P
+ else
+ echo "Skipping $P"
+ fi
+ done
+
[3/5] incubator-gobblin git commit: [GOBBLIN-355] Add HEADER as per
release process
Posted by ab...@apache.org.
[GOBBLIN-355] Add HEADER as per release process
Project: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/commit/20818ac2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/tree/20818ac2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/diff/20818ac2
Branch: refs/heads/0.12.0
Commit: 20818ac25a2a6c7a2e282d49091c138bcdddb92b
Parents: a85d8a8
Author: Abhishek Tiwari <ab...@gmail.com>
Authored: Wed Jan 3 19:34:49 2018 +0530
Committer: Abhishek Tiwari <ab...@gmail.com>
Committed: Wed Jan 3 19:34:49 2018 +0530
----------------------------------------------------------------------
HEADER | 14 ++++++++++++++
1 file changed, 14 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/20818ac2/HEADER
----------------------------------------------------------------------
diff --git a/HEADER b/HEADER
new file mode 100644
index 0000000..8853bce
--- /dev/null
+++ b/HEADER
@@ -0,0 +1,14 @@
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
[4/5] incubator-gobblin git commit: [GOBBLIN-355] Add git scripts to
publish to Nexus and generate signed sources
Posted by ab...@apache.org.
[GOBBLIN-355] Add git scripts to publish to Nexus and generate signed sources
Project: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/commit/fbc4cba3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/tree/fbc4cba3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/diff/fbc4cba3
Branch: refs/heads/0.12.0
Commit: fbc4cba3157d6962ab2acc23892dcdf2f1310980
Parents: 20818ac
Author: Abhishek Tiwari <ab...@gmail.com>
Authored: Thu Jan 4 00:25:37 2018 +0530
Committer: Abhishek Tiwari <ab...@gmail.com>
Committed: Thu Jan 4 00:25:37 2018 +0530
----------------------------------------------------------------------
gobblin-metastore/build.gradle | 2 +-
.../gobblin-rest-api/build.gradle | 2 +-
.../gobblin-rest-client/build.gradle | 2 +-
.../gobblin-rest-server/build.gradle | 2 +-
gobblin-restli/api.gradle | 2 +-
gobblin-runtime/build.gradle | 2 +-
gobblin-service/build.gradle | 2 +-
gradle.properties | 1 +
gradle/scripts/buildscript.gradle | 32 ++++
gradle/scripts/defaultBuildProperties.gradle | 13 +-
gradle/scripts/mavenPublishing.gradle | 24 +--
gradle/scripts/nexusPublishing.gradle | 188 +++++++++++++++++++
gradle/scripts/release.gradle | 177 +++++++++++++++++
13 files changed, 426 insertions(+), 23 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-metastore/build.gradle
----------------------------------------------------------------------
diff --git a/gobblin-metastore/build.gradle b/gobblin-metastore/build.gradle
index bf8e998..a78df44 100644
--- a/gobblin-metastore/build.gradle
+++ b/gobblin-metastore/build.gradle
@@ -49,7 +49,7 @@ def installer = install.repositories.mavenInstaller
[installer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-rest-client'
}
-if (rootProject.publishToMaven) {
+if (rootProject.publishToMaven || rootProject.publishToNexus) {
def deployer = uploadArchives.repositories.mavenDeployer
[deployer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-rest-client'
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-rest-service/gobblin-rest-api/build.gradle
----------------------------------------------------------------------
diff --git a/gobblin-rest-service/gobblin-rest-api/build.gradle b/gobblin-rest-service/gobblin-rest-api/build.gradle
index 43832ed..6745aa3 100644
--- a/gobblin-rest-service/gobblin-rest-api/build.gradle
+++ b/gobblin-rest-service/gobblin-rest-api/build.gradle
@@ -31,7 +31,7 @@ artifacts {
archives mainDataTemplateJar
}
-if (rootProject.ext.publishToMaven) {
+if (rootProject.ext.publishToMaven || rootProject.ext.publishToNexus) {
configure(uploadArchives.repositories.mavenDeployer) {
addFilter('gobblin-rest-api-rest-client') {artifact, file ->
artifact.name == 'gobblin-rest-api-rest-client'
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-rest-service/gobblin-rest-client/build.gradle
----------------------------------------------------------------------
diff --git a/gobblin-rest-service/gobblin-rest-client/build.gradle b/gobblin-rest-service/gobblin-rest-client/build.gradle
index 551edad..aea8d95 100644
--- a/gobblin-rest-service/gobblin-rest-client/build.gradle
+++ b/gobblin-rest-service/gobblin-rest-client/build.gradle
@@ -31,7 +31,7 @@ def installer = install.repositories.mavenInstaller
[installer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-rest-client'
}
-if (rootProject.publishToMaven) {
+if (rootProject.publishToMaven || rootProject.publishToNexus) {
def deployer = uploadArchives.repositories.mavenDeployer
[deployer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-rest-client'
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-rest-service/gobblin-rest-server/build.gradle
----------------------------------------------------------------------
diff --git a/gobblin-rest-service/gobblin-rest-server/build.gradle b/gobblin-rest-service/gobblin-rest-server/build.gradle
index c6a319b..9b99d29 100644
--- a/gobblin-rest-service/gobblin-rest-server/build.gradle
+++ b/gobblin-rest-service/gobblin-rest-server/build.gradle
@@ -43,7 +43,7 @@ def installer = install.repositories.mavenInstaller
[installer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-data-template'
}
-if (rootProject.publishToMaven) {
+if (rootProject.publishToMaven || rootProject.publishToNexus) {
def deployer = uploadArchives.repositories.mavenDeployer
[deployer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-data-template'
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-restli/api.gradle
----------------------------------------------------------------------
diff --git a/gobblin-restli/api.gradle b/gobblin-restli/api.gradle
index 095e7d9..688dae8 100644
--- a/gobblin-restli/api.gradle
+++ b/gobblin-restli/api.gradle
@@ -109,7 +109,7 @@ install {
}
}
-if (rootProject.ext.publishToMaven) {
+if (rootProject.ext.publishToMaven || rootProject.ext.publishToNexus) {
uploadArchives {
repositories {
mavenDeployer {
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-runtime/build.gradle
----------------------------------------------------------------------
diff --git a/gobblin-runtime/build.gradle b/gobblin-runtime/build.gradle
index 422d6f3..b453a26 100644
--- a/gobblin-runtime/build.gradle
+++ b/gobblin-runtime/build.gradle
@@ -105,7 +105,7 @@ def installer = install.repositories.mavenInstaller
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-rest-client'
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-data-template'
}
-if (rootProject.publishToMaven) {
+if (rootProject.publishToMaven || rootProject.publishToNexus) {
def deployer = uploadArchives.repositories.mavenDeployer
[deployer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-rest-api' }.artifactId = 'gobblin-rest-api-rest-client'
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gobblin-service/build.gradle
----------------------------------------------------------------------
diff --git a/gobblin-service/build.gradle b/gobblin-service/build.gradle
index 0b461bd..268bc70 100644
--- a/gobblin-service/build.gradle
+++ b/gobblin-service/build.gradle
@@ -90,7 +90,7 @@ def installer = install.repositories.mavenInstaller
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-flow-config-service-api' }.artifactId = 'gobblin-flow-config-service-api-rest-client'
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-flow-config-service-api' }.artifactId = 'gobblin-flow-config-service-api-data-template'
}
-if (rootProject.publishToMaven) {
+if (rootProject.publishToMaven || rootProject.publishToNexus) {
def deployer = uploadArchives.repositories.mavenDeployer
[deployer]*.pom*.whenConfigured {pom ->
pom.dependencies.find {dep -> dep.groupId == project.group && dep.artifactId == 'gobblin-flow-config-service-api' }.artifactId = 'gobblin-flow-config-service-api-rest-client'
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gradle.properties
----------------------------------------------------------------------
diff --git a/gradle.properties b/gradle.properties
index 7aaf18d..88d130e 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -33,4 +33,5 @@ ide.recursive=true
# Apache release specific
version=0.12.0
group=org.apache.gobblin
+release=false
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gradle/scripts/buildscript.gradle
----------------------------------------------------------------------
diff --git a/gradle/scripts/buildscript.gradle b/gradle/scripts/buildscript.gradle
new file mode 100644
index 0000000..3ad13c5
--- /dev/null
+++ b/gradle/scripts/buildscript.gradle
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+repositories {
+ repositories {
+ maven {
+ // For gradle-nexus-plugin
+ url 'http://jcenter.bintray.com/'
+ }
+ }
+}
+
+dependencies {
+ classpath 'org.gradle.api.plugins:gradle-nexus-plugin:0.7.1'
+ classpath 'com.fizzpod:gradle-sweeney-plugin:1.0+'
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gradle/scripts/defaultBuildProperties.gradle
----------------------------------------------------------------------
diff --git a/gradle/scripts/defaultBuildProperties.gradle b/gradle/scripts/defaultBuildProperties.gradle
index 7f25b22..ce95473 100644
--- a/gradle/scripts/defaultBuildProperties.gradle
+++ b/gradle/scripts/defaultBuildProperties.gradle
@@ -19,8 +19,10 @@ import org.apache.gobblin.gradle.BuildProperties
import org.apache.gobblin.gradle.BuildProperty
def BuildProperties BUILD_PROPERTIES = new BuildProperties(project)
- .register(new BuildProperty("artifactRepository", "https://oss.sonatype.org/service/local/staging/deploy/maven2/", "Maven repository to publish artifacts"))
- .register(new BuildProperty("artifactSnapshotRepository", "https://oss.sonatype.org/content/repositories/snapshots/", "Maven repository to publish artifacts"))
+ .register(new BuildProperty("sonatypeArtifactRepository", "https://oss.sonatype.org/service/local/staging/deploy/maven2/", "Maven repository to publish artifacts"))
+ .register(new BuildProperty("sonatypeArtifactSnapshotRepository", "https://oss.sonatype.org/content/repositories/snapshots/", "Maven repository to publish artifacts"))
+ .register(new BuildProperty("nexusArtifactRepository", "https://repository.apache.org/service/local/staging/deploy/maven2", "Maven repository to publish artifacts"))
+ .register(new BuildProperty("nexusArtifactSnapshotRepository", "https://repository.apache.org/content/repositories/snapshots", "Maven repository to publish artifacts"))
.register(new BuildProperty("avroVersion", "1.8.1", "Avro dependencies version"))
.register(new BuildProperty("awsVersion", "1.11.8", "AWS dependencies version"))
.register(new BuildProperty("bytemanVersion", "2.2.1", "Byteman dependencies version"))
@@ -35,6 +37,7 @@ def BuildProperties BUILD_PROPERTIES = new BuildProperties(project)
.register(new BuildProperty("kafka09Version", "0.9.0.1", "Kafka 0.9 dependencies version"))
.register(new BuildProperty("pegasusVersion", "11.0.0", "Pegasus dependencies version"))
.register(new BuildProperty("publishToMaven", false, "Enable publishing of artifacts to a central Maven repository"))
+ .register(new BuildProperty("publishToNexus", false, "Enable publishing of artifacts to Nexus"))
.register(new BuildProperty("salesforceVersion", "37.0.3", "Salesforce dependencies version"))
task buildProperties(description: 'Lists main properties that can be used to customize the build') << {
@@ -47,8 +50,10 @@ BUILD_PROPERTIES.ensureDefined('gobblinFlavor')
// Compiler compatibility
BUILD_PROPERTIES.ensureDefined('jdkVersion')
-BUILD_PROPERTIES.ensureDefined('artifactRepository')
-BUILD_PROPERTIES.ensureDefined('artifactSnapshotRepository')
+BUILD_PROPERTIES.ensureDefined('sonatypeArtifactRepository')
+BUILD_PROPERTIES.ensureDefined('sonatypeArtifactSnapshotRepository')
+BUILD_PROPERTIES.ensureDefined('nexusArtifactRepository')
+BUILD_PROPERTIES.ensureDefined('nexusArtifactSnapshotRepository')
BUILD_PROPERTIES.ensureDefined('doNotSignArtifacts')
// Library dependencies versions
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gradle/scripts/mavenPublishing.gradle
----------------------------------------------------------------------
diff --git a/gradle/scripts/mavenPublishing.gradle b/gradle/scripts/mavenPublishing.gradle
index 044b4fc..799464c 100644
--- a/gradle/scripts/mavenPublishing.gradle
+++ b/gradle/scripts/mavenPublishing.gradle
@@ -21,7 +21,7 @@ import java.util.concurrent.locks.ReentrantLock
// Set group for Maven
allprojects {
if (!project.hasProperty('group') || project.group.length() == 0) {
- project.ext.group = 'com.linkedin.gobblin'
+ project.ext.group = 'org.apache.gobblin'
project.group = project.ext.group
}
}
@@ -65,16 +65,16 @@ gradle.taskGraph.afterTask {
}
ext.pomAttributes = {
- name "${project.name}"
packaging 'jar'
// optionally artifactId can be defined here
- description 'Gobblin Ingestion Framework'
- url 'https://github.com/linkedin/gobblin/'
+ name 'Apache Gobblin (incubating)'
+ description 'A distributed data integration framework for streaming and batch data ecosystems.'
+ url 'https://gobblin.apache.org/'
scm {
- connection 'scm:git:git@github.com:linkedin/gobblin.git'
- developerConnection 'scm:git:git@github.com:linkedin/gobblin.git'
- url 'git@github.com:linkedin/gobblin.git'
+ url 'https://git-wip-us.apache.org/repos/asf?p=incubator-gobblin.git;a=tree'
+ connection 'scm:http://git-wip-us.apache.org/repos/asf/incubator-gobblin.git'
+ developerConnection 'scm:https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git'
}
licenses {
@@ -141,10 +141,10 @@ subprojects {
mavenInstaller {
mavenLocal()
pom.project {
- name "${project.name}"
packaging 'jar'
- description 'Gobblin Ingestion Framework'
- url 'https://github.com/linkedin/gobblin/'
+ name 'Apache Gobblin (incubating)'
+ description 'A distributed data integration framework for streaming and batch data ecosystems.'
+ url 'https://gobblin.apache.org/'
}
}
}
@@ -165,11 +165,11 @@ subprojects {
}
}
- repository(url: rootProject.artifactRepository) {
+ repository(url: rootProject.sonatypeArtifactRepository) {
authentication(userName: rootProject.ext.ossrhUsername, password: rootProject.ext.ossrhPassword)
}
- snapshotRepository(url: rootProject.artifactSnapshotRepository) {
+ snapshotRepository(url: rootProject.sonatypeArtifactSnapshotRepository) {
authentication(userName: rootProject.ext.ossrhUsername, password: rootProject.ext.ossrhPassword)
}
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gradle/scripts/nexusPublishing.gradle
----------------------------------------------------------------------
diff --git a/gradle/scripts/nexusPublishing.gradle b/gradle/scripts/nexusPublishing.gradle
new file mode 100644
index 0000000..86d89b3
--- /dev/null
+++ b/gradle/scripts/nexusPublishing.gradle
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.locks.ReentrantLock
+
+// Set group for Maven
+allprojects {
+ if (!project.hasProperty('group') || project.group.length() == 0) {
+ project.ext.group = 'org.apache.gobblin'
+ project.group = project.ext.group
+ }
+}
+
+// Nexus publishing
+ext.buildProperties.ensureDefined('publishToNexus')
+if (ext.publishToNexus) {
+ plugins.apply('maven')
+ // Workaround for a bug in gradle's "maven" plugin. See https://discuss.gradle.org/t/error-in-parallel-build/7215/3
+ project.setProperty("org.gradle.parallel", "false")
+}
+
+ext.signArtifacts = !project.doNotSignArtifacts
+
+// Maven POM generation is not thread safe, so serialize all the Upload tasks we can use `--parallel`.
+// https://issues.gradle.org/browse/GRADLE-2492
+// When we start building with 2.3 and later we should remove this and just add a common output dir for all tasks and let Gradle serialize them
+def lock = new ReentrantLock()
+def available = lock.newCondition()
+def busy = new AtomicBoolean()
+def serializedTasks = []
+allprojects {
+ tasks.matching { it.name == "generatePom" || it instanceof Upload }.all {
+ serializedTasks << it
+ doFirst {
+ lock.lock()
+ while (busy.get()) {
+ available.await()
+ }
+ busy.set(true)
+ }
+ }
+}
+
+gradle.taskGraph.afterTask {
+ if (it in serializedTasks && lock.heldByCurrentThread) {
+ busy.set(false)
+ available.signal()
+ lock.unlock()
+ }
+}
+
+ext.pomAttributes = {
+ packaging 'jar'
+ // optionally artifactId can be defined here
+ name 'Apache Gobblin (incubating)'
+ description 'A distributed data integration framework for streaming and batch data ecosystems.'
+ url 'https://gobblin.apache.org/'
+
+ scm {
+ url 'https://git-wip-us.apache.org/repos/asf?p=incubator-gobblin.git;a=tree'
+ connection 'scm:http://git-wip-us.apache.org/repos/asf/incubator-gobblin.git'
+ developerConnection 'scm:https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git'
+ }
+
+ licenses {
+ license {
+ name 'The Apache License, Version 2.0'
+ url 'http://www.apache.org/licenses/LICENSE-2.0.txt'
+ }
+ }
+
+ developers {
+ developer {
+ name 'Abhishek Tiwari'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Chavdar Botev'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Issac Buenrostro'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Min Tu'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Narasimha Veeramreddy'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Pradhan Cadabam'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Sahil Takiar'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Shirshanka Das'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Yinan Li'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Ying Dai'
+ organization 'LinkedIn'
+ }
+ developer {
+ name 'Ziyang Liu'
+ organization 'LinkedIn'
+ }
+ }
+}
+
+subprojects {
+ plugins.withType(JavaPlugin) {
+ plugins.apply('maven')
+
+ install {
+ repositories {
+ mavenInstaller {
+ mavenLocal()
+ pom.project {
+ packaging 'jar'
+ name 'Apache Gobblin (incubating)'
+ description 'A distributed data integration framework for streaming and batch data ecosystems.'
+ url 'https://gobblin.apache.org/'
+ }
+ }
+ }
+ }
+
+ // Publishing of maven artifacts for subprojects
+ if (rootProject.ext.publishToNexus) {
+ if (rootProject.ext.signArtifacts) {
+ plugins.apply('signing')
+ }
+
+ uploadArchives {
+ repositories {
+ mavenDeployer {
+ beforeDeployment { MavenDeployment deployment ->
+ if (rootProject.ext.signArtifacts) {
+ signing.signPom(deployment)
+ }
+ }
+
+ repository(url: rootProject.nexusArtifactRepository) {
+ authentication(userName: rootProject.ext.nexusUsername, password: rootProject.ext.nexusPassword)
+ }
+
+ snapshotRepository(url: rootProject.nexusArtifactSnapshotRepository) {
+ authentication(userName: rootProject.ext.nexusUsername, password: rootProject.ext.nexusPassword)
+ }
+
+ pom.project pomAttributes
+ }
+ }
+ }
+
+ if (rootProject.ext.signArtifacts) {
+ signing {
+ sign configurations.archives
+ }
+ }
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/fbc4cba3/gradle/scripts/release.gradle
----------------------------------------------------------------------
diff --git a/gradle/scripts/release.gradle b/gradle/scripts/release.gradle
new file mode 100644
index 0000000..98195b1
--- /dev/null
+++ b/gradle/scripts/release.gradle
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+project(':') {
+ apply plugin: 'base'
+ apply plugin: 'signing'
+}
+
+def isRelease = ext.release.toBoolean()
+
+def releaseVersion = project.version
+if (!isRelease) {
+ releaseVersion += "-SNAPSHOT"
+}
+
+// Modify the gradle.properties to indicate whether this is a release. This results in the
+// source releases generating artifacts without -SNAPSHOT appended to the version when they are
+// built.
+task prepare_release_config(type: Copy) {
+ from "$rootDir/gradle.properties"
+ into "$rootDir"
+ rename { filename ->
+ filename + ".release" }
+ filter { line ->
+ if (isRelease && line.startsWith("release=")) {
+ "release=true"
+ }
+ else {
+ line
+ }
+ }
+}
+
+task sourceRelease(type: Tar, dependsOn: prepare_release_config) {
+ description = "Build a source release, specifically excluding the build directories and gradle wrapper files"
+ compression = Compression.GZIP
+
+ baseName "apache-gobblin-incubating-sources-${releaseVersion}"
+
+ from(project.rootDir) {
+ // don't include gradle.properties because we use a modified version
+ exclude "gradle.properties"
+ exclude '**/build'
+ exclude 'buildSrc'
+ exclude '.gradle'
+ exclude '.github'
+ exclude 'gradlew'
+ exclude 'gradlew.bat'
+ exclude 'gradle'
+ exclude 'maven-sonatype'
+ exclude 'travis'
+ exclude '.classpath*'
+ exclude '.project*'
+ exclude '.settings'
+ exclude '**/.idea'
+ exclude '**/*.iml'
+ exclude '**/*.iws'
+ exclude '**/*.ipr'
+ exclude '**/.classpath'
+ exclude '**/.project'
+ exclude '**/.settings'
+ exclude '**/*.swp'
+ exclude '**/*.swo'
+ exclude '**/*.log'
+ exclude '**/build/'
+ exclude '**/.gradle'
+ exclude 'test-output'
+ exclude '**/test-output'
+ exclude 'dist'
+ exclude 'target'
+ exclude 'tmp'
+ exclude 'out'
+ exclude '**/out'
+ exclude 'output'
+ exclude 'gobblin-test/basicTest'
+ exclude 'gobblin-test/jobOutput'
+ exclude 'gobblin-test/state-store'
+ exclude 'gobblin-tesTaskt/metrics'
+ exclude 'gobblin-test/byteman'
+ exclude 'gobblin-test/locks'
+ exclude 'gobblin-test/mr-jobs'
+ exclude '**/mainGeneratedDataTemplate'
+ exclude '**/mainGeneratedRest'
+ exclude 'gobblin-dist'
+ exclude 'metastore_db'
+ exclude 'GobblinKey_*.pem'
+ exclude 'node_modules'
+ exclude 'package-lock.json'
+ exclude '**/gen-java'
+ exclude '**/generated-gobblin-cluster.conf'
+ }
+
+ // rename gradle.properties.release to gradle.properties
+ rename { filename ->
+ if (filename.equals("gradle.properties.release")) {
+ "gradle.properties"
+ }
+ else {
+ filename
+ }
+ }
+
+ into(baseName)
+
+ // Set destination directory.
+ destinationDir = file("${project.buildDir}/distribution/source")
+
+ archiveName = "${baseName}.tgz"
+ doLast {
+ ant.checksum file:"$destinationDir/$archiveName", algorithm: "MD5", fileext: ".md5"
+ ant.checksum file:"$destinationDir/$archiveName", algorithm: "SHA-512", fileext: ".sha512"
+ }
+}
+
+// Signing requires a user ~/.gradle/gradle.properties file with signing information.
+// See: http://www.gradle.org/docs/current/userguide/signing_plugin.html
+signing {
+ // Sign the source archive if the proper configuration is in place to do so. Otherwise
+ // skip the signing process (it isn't required). This archive can be manually signed if
+ // needed.
+ required false
+ sign sourceRelease
+}
+
+task release(dependsOn: signSourceRelease)
+
+// Publishing to Apache's Maven repository (Nexus). To install the archives in the
+// local repository, run the 'install' task.
+//subprojects {
+// apply plugin: 'nexus'
+//
+// nexus {
+// attachSources = false
+// attachTests = false
+// attachJavadoc = false
+// sign = true
+// repositoryUrl = 'https://repository.apache.org/service/local/staging/deploy/maven2'
+// snapshotRepositoryUrl = 'https://repository.apache.org/content/repositories/snapshots'
+// }
+//
+// modifyPom {
+// project {
+// name 'Apache Gobblin (incubating)'
+// description 'A distributed data integration framework for streaming and batch data ecosystems.'
+// url 'https://gobblin.apache.org/'
+//
+// scm {
+// url 'https://git-wip-us.apache.org/repos/asf?p=incubator-gobblin.git;a=tree'
+// connection 'scm:http://git-wip-us.apache.org/repos/asf/incubator-gobblin.git'
+// developerConnection 'scm:https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git'
+// }
+//
+// licenses {
+// license {
+// name 'The Apache Software License, Version 2.0'
+// url 'http://www.apache.org/licenses/LICENSE-2.0.txt'
+// }
+// }
+// }
+// }
+//}
\ No newline at end of file
[2/5] incubator-gobblin git commit: [GOBBLIN-355] Added DISCLAIMER
for Apache Incubation
Posted by ab...@apache.org.
[GOBBLIN-355] Added DISCLAIMER for Apache Incubation
Project: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/commit/a85d8a8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/tree/a85d8a8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-gobblin/diff/a85d8a8c
Branch: refs/heads/0.12.0
Commit: a85d8a8c03413754e8bcab14890666886c49104f
Parents: 50b0bcf
Author: Abhishek Tiwari <ab...@gmail.com>
Authored: Wed Jan 3 19:30:31 2018 +0530
Committer: Abhishek Tiwari <ab...@gmail.com>
Committed: Wed Jan 3 19:30:31 2018 +0530
----------------------------------------------------------------------
DISCLAIMER | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-gobblin/blob/a85d8a8c/DISCLAIMER
----------------------------------------------------------------------
diff --git a/DISCLAIMER b/DISCLAIMER
new file mode 100644
index 0000000..2177b17
--- /dev/null
+++ b/DISCLAIMER
@@ -0,0 +1 @@
+Apache Gobblin is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.