You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by ja...@apache.org on 2018/11/27 09:48:08 UTC
[01/50] samza git commit: edits to LinkedIn case-study
Repository: samza
Updated Branches:
refs/heads/master 6f40414d7 -> 6cdcdefd4
edits to LinkedIn case-study
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/3dc92511
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/3dc92511
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/3dc92511
Branch: refs/heads/master
Commit: 3dc92511636cfbde50e29bd646d57f27b3c8ffa0
Parents: b00035b
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 15 16:14:07 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 15 16:14:07 2018 -0700
----------------------------------------------------------------------
docs/_case-studies/linkedin.md | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/3dc92511/docs/_case-studies/linkedin.md
----------------------------------------------------------------------
diff --git a/docs/_case-studies/linkedin.md b/docs/_case-studies/linkedin.md
index a52ada4..0f5fa6d 100644
--- a/docs/_case-studies/linkedin.md
+++ b/docs/_case-studies/linkedin.md
@@ -38,14 +38,10 @@ Any service that wants to send out a notification to members writes its request
- **Partitioner**: _Partitioners_ read incoming communication requests from Kafka and distribute them across _Pipeline_ instances based on the hash of the recipient. It also does some
filtering early-on to drop malformed messages.
-- **Relevance processor**: The _Relevance processors_ read personalized machine-learning models and stores them in Samza's RocksDb store. It uses them to evaluate incoming requests and determine the right channel (eg: drop it vs sending an email vs push notification vs badge) for the notification.
-- **Pipeline**: The _pipeline_ processors aggregate the output of the _Relevance_ and the _Partitioners_, thereby making the final call. It heavily leverages Samza's local state to
-store and merge notifications. It decides the frequency of notifications (eg: duplicate notifications are merged, notifications are capped at a certain threshold). The _Pipeline_ also implements a _scheduler_ on top of Samza's local-store so that it can schedule messages for delivery later (For eg: it makes no sense to send notifications to a member at midnight)
+- **Relevance processor**: The _Relevance processors_ read personalized machine-learning models from Kafka and stores them in Samza's RocksDb store for evaluating them later. It uses them to score incoming requests and determine the right channel (eg: drop it vs sending an email vs push notification vs badge) for the notification.
+- **Pipeline**: The _pipeline_ processors aggregate the output of the _Relevance_ and the _Partitioners_, thereby making the final determination on the notification. It heavily leverages Samza's local state to batch and aggregate notifications. It decides the frequency of notifications (eg: duplicate notifications are merged, notifications are capped at a certain threshold). The _Pipeline_ also implements a _scheduler_ on top of Samza's local-store so that it can schedule messages for delivery later (For eg: it makes no sense to send notifications to a member at midnight)
-Handle partitioned communication requests which performs aggregation and consults with the relevance model to determine delivery time
-- **Relevance processor**: Provide insights on how relevant is the content to the user, the right delivery time, etc.
-
ATC, leverages Samza extensively and uses a lot of features including but not limited to:
- **Stateful processing**: The ML models in the relevance module are stored locally in RocksDb which are updated realtime time based on user feedback.
[27/50] samza git commit: Revert un-necessary changes
Posted by ja...@apache.org.
Revert un-necessary changes
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/796aa400
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/796aa400
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/796aa400
Branch: refs/heads/master
Commit: 796aa4004069ca56923adcf6c544f7a399f72b22
Parents: 424b042
Author: Jagadish <jv...@linkedin.com>
Authored: Wed Oct 24 14:36:04 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Wed Oct 24 14:36:04 2018 -0700
----------------------------------------------------------------------
.../grouper/task/SingleContainerGrouperFactory.java | 13 ++-----------
.../samza/standalone/PassthroughJobCoordinator.java | 9 +--------
.../controlmessages/EndOfStreamIntegrationTest.java | 2 +-
.../test/controlmessages/WatermarkIntegrationTest.java | 2 +-
4 files changed, 5 insertions(+), 21 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/796aa400/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java b/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
index 90c031a..ee962d5 100644
--- a/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
+++ b/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
@@ -19,8 +19,6 @@
package org.apache.samza.container.grouper.task;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
import org.apache.samza.config.Config;
import org.apache.samza.config.JobConfig;
import org.apache.samza.container.TaskName;
@@ -29,7 +27,6 @@ import org.apache.samza.job.model.TaskModel;
import java.util.Collections;
import java.util.HashMap;
-import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -44,22 +41,16 @@ class SingleContainerGrouper implements TaskNameGrouper {
private final String containerId;
SingleContainerGrouper(String containerId) {
- this.containerId = null;
+ this.containerId = containerId;
}
@Override
public Set<ContainerModel> group(Set<TaskModel> taskModels) {
- return group(taskModels, ImmutableList.of(this.containerId));
- }
-
- @Override
- public Set<ContainerModel> group(Set<TaskModel> taskModels, List<String> containersIds) {
- Preconditions.checkState(containersIds.size() == 1);
Map<TaskName, TaskModel> taskNameTaskModelMap = new HashMap<>();
for (TaskModel taskModel: taskModels) {
taskNameTaskModelMap.put(taskModel.getTaskName(), taskModel);
}
- ContainerModel containerModel = new ContainerModel(containersIds.get(0), taskNameTaskModelMap);
+ ContainerModel containerModel = new ContainerModel(containerId, taskNameTaskModelMap);
return Collections.singleton(containerModel);
}
}
http://git-wip-us.apache.org/repos/asf/samza/blob/796aa400/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
index bcf2085..737ac3e 100644
--- a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
+++ b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
@@ -18,14 +18,11 @@
*/
package org.apache.samza.standalone;
-import com.google.common.collect.ImmutableMap;
import org.apache.samza.checkpoint.CheckpointManager;
import org.apache.samza.config.ApplicationConfig;
import org.apache.samza.config.Config;
import org.apache.samza.config.ConfigException;
import org.apache.samza.config.JobConfig;
-import org.apache.samza.config.MapConfig;
-import org.apache.samza.config.TaskConfig;
import org.apache.samza.config.TaskConfigJava;
import org.apache.samza.coordinator.JobCoordinator;
import org.apache.samza.coordinator.JobModelManager;
@@ -72,7 +69,7 @@ public class PassthroughJobCoordinator implements JobCoordinator {
public PassthroughJobCoordinator(Config config) {
this.processorId = createProcessorId(config);
- this.config = new MapConfig(config, ImmutableMap.of(TaskConfig.GROUPER_FACTORY(), "org.apache.samza.container.grouper.task.SingleContainerGrouperFactory"));
+ this.config = config;
}
@Override
@@ -122,11 +119,7 @@ public class PassthroughJobCoordinator implements JobCoordinator {
SystemAdmins systemAdmins = new SystemAdmins(config);
StreamMetadataCache streamMetadataCache = new StreamMetadataCache(systemAdmins, 5000, SystemClock.instance());
systemAdmins.start();
-<<<<<<< Updated upstream
String containerId = Integer.toString(config.getInt(JobConfig.PROCESSOR_ID()));
-=======
- String containerId = this.processorId;
->>>>>>> Stashed changes
/** TODO:
Locality Manager seems to be required in JC for reading locality info and grouping tasks intelligently and also,
http://git-wip-us.apache.org/repos/asf/samza/blob/796aa400/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
----------------------------------------------------------------------
diff --git a/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java b/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
index f967672..6f381e2 100644
--- a/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
+++ b/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
@@ -78,7 +78,7 @@ public class EndOfStreamIntegrationTest extends AbstractIntegrationTestHarness {
configs.put("streams.PageView.partitionCount", String.valueOf(partitionCount));
configs.put(JobConfig.JOB_NAME(), "test-eos-job");
- //configs.put(JobConfig.PROCESSOR_ID(), "1");
+ configs.put(JobConfig.PROCESSOR_ID(), "1");
configs.put(JobCoordinatorConfig.JOB_COORDINATOR_FACTORY, PassthroughJobCoordinatorFactory.class.getName());
configs.put(TaskConfig.GROUPER_FACTORY(), SingleContainerGrouperFactory.class.getName());
http://git-wip-us.apache.org/repos/asf/samza/blob/796aa400/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
----------------------------------------------------------------------
diff --git a/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java b/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
index 088278d..74c32b4 100644
--- a/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
+++ b/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
@@ -131,7 +131,7 @@ public class WatermarkIntegrationTest extends AbstractIntegrationTestHarness {
configs.put("streams.PageView.partitionCount", String.valueOf(PARTITION_COUNT));
configs.put(JobConfig.JOB_NAME(), "test-watermark-job");
- //configs.put(JobConfig.PROCESSOR_ID(), "hello");
+ configs.put(JobConfig.PROCESSOR_ID(), "1");
configs.put(JobCoordinatorConfig.JOB_COORDINATOR_FACTORY, PassthroughJobCoordinatorFactory.class.getName());
configs.put(TaskConfig.GROUPER_FACTORY(), SingleContainerGrouperFactory.class.getName());
[40/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/841b8faf
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/841b8faf
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/841b8faf
Branch: refs/heads/master
Commit: 841b8fafe0d75cc7e46b96584f7959c9baecd2ec
Parents: 018a6e4 3f9bbb0
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 30 18:46:53 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 30 18:46:53 2018 -0700
----------------------------------------------------------------------
docs/_releases/1.0.0.md | 500 +++++++++++++++++++
.../org/apache/samza/sql/SamzaSqlRelRecord.java | 13 +-
.../sql/client/impl/SamzaExecutorTest.java | 2 +-
.../apache/samza/sql/avro/AvroRelConverter.java | 59 ++-
.../samza/sql/avro/AvroRelSchemaProvider.java | 5 +
.../samza/sql/data/SamzaSqlCompositeKey.java | 82 ---
.../samza/sql/data/SamzaSqlRelMessage.java | 52 +-
.../apache/samza/sql/fn/ConvertToStringUdf.java | 39 ++
.../org/apache/samza/sql/fn/GetSqlFieldUdf.java | 95 ++++
.../sql/impl/ConfigBasedIOResolverFactory.java | 14 +-
.../apache/samza/sql/planner/QueryPlanner.java | 2 +-
.../sql/runner/SamzaSqlApplicationRunner.java | 55 +-
.../samza/sql/translator/JoinTranslator.java | 33 +-
.../SamzaSqlRelMessageJoinFunction.java | 23 +-
.../samza/sql/avro/TestAvroRelConversion.java | 10 +-
.../samza/sql/data/TestSamzaSqlRelMessage.java | 30 ++
.../samza/sql/fn/TestConvertToStringUdf.java | 62 +++
.../apache/samza/sql/fn/TestGetSqlFieldUdf.java | 166 ++++++
.../samza/sql/system/TestAvroSystemFactory.java | 6 +-
.../samza/sql/testutil/SamzaSqlTestConfig.java | 1 -
.../sql/testutil/TestIOResolverFactory.java | 13 +-
.../TestSamzaSqlRelMessageJoinFunction.java | 21 +-
.../descriptors/InMemoryInputDescriptor.java | 5 +
.../descriptors/InMemoryOutputDescriptor.java | 5 +
.../tools/avro/AvroSchemaGenRelConverter.java | 12 +-
25 files changed, 1113 insertions(+), 192 deletions(-)
----------------------------------------------------------------------
[29/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/48f5f280
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/48f5f280
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/48f5f280
Branch: refs/heads/master
Commit: 48f5f280784fcf52c3148dac8cbece6ae9d3d41e
Parents: c21593b 59bc23c
Author: Jagadish <jv...@linkedin.com>
Authored: Wed Oct 24 15:18:41 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Wed Oct 24 15:18:41 2018 -0700
----------------------------------------------------------------------
----------------------------------------------------------------------
[44/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/e0e5e871
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/e0e5e871
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/e0e5e871
Branch: refs/heads/master
Commit: e0e5e871477fd2894bf37b74a763c4a35bc4849f
Parents: f8470b1 4e59070
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Nov 13 19:17:26 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Nov 13 19:17:26 2018 -0800
----------------------------------------------------------------------
build.gradle | 36 +-
.../versioned/api/high-level-api.md | 411 ++++++++++---------
.../versioned/api/low-level-api.md | 351 ++++++++--------
.../versioned/api/programming-model.md | 118 +++---
.../documentation/versioned/api/samza-sql.md | 2 +-
.../documentation/versioned/api/table-api.md | 11 +-
.../versioned/connectors/eventhubs.md | 2 +-
.../documentation/versioned/connectors/kafka.md | 6 +-
.../versioned/core-concepts/core-concepts.md | 14 +-
.../documentation/versioned/hadoop/overview.md | 2 +-
docs/learn/documentation/versioned/index.html | 16 +-
.../versioned/jobs/samza-configurations.md | 2 +-
.../versioned/operations/monitoring.md | 10 +-
.../versioned/hello-samza-high-level-code.md | 2 +-
.../versioned/hello-samza-high-level-yarn.md | 4 +-
.../versioned/samza-event-hubs-standalone.md | 2 +-
docs/learn/tutorials/versioned/samza-sql.md | 2 +-
docs/startup/code-examples/versioned/index.md | 2 +-
.../org/apache/samza/table/TableProvider.java | 61 +++
.../samza/table/TableProviderFactory.java | 35 ++
.../table/descriptors/BaseTableDescriptor.java | 110 +++++
.../descriptors/CachingTableDescriptor.java | 173 ++++++++
.../descriptors/GuavaCacheTableDescriptor.java | 79 ++++
.../descriptors/HybridTableDescriptor.java | 48 +++
.../table/descriptors/LocalTableDescriptor.java | 168 ++++++++
.../descriptors/RemoteTableDescriptor.java | 296 +++++++++++++
.../samza/table/descriptors/TableProvider.java | 62 ---
.../table/descriptors/TableProviderFactory.java | 36 --
.../samza/table/remote/TableRateLimiter.java | 167 ++++++++
.../samza/table/remote/TableReadFunction.java | 111 +++++
.../samza/table/remote/TableWriteFunction.java | 159 +++++++
.../samza/table/retry/TableRetryPolicy.java | 257 ++++++++++++
.../apache/samza/table/utils/SerdeUtils.java | 66 +++
.../org/apache/samza/task/ClosableTask.java | 7 +
.../table/remote/TestTableRateLimiter.java | 103 +++++
.../descriptors/ApplicationDescriptorImpl.java | 6 +-
.../apache/samza/config/JavaSystemConfig.java | 25 ++
.../apache/samza/config/JavaTableConfig.java | 5 +-
.../apache/samza/table/BaseTableProvider.java | 71 ++++
.../samza/table/TableConfigGenerator.java | 2 -
.../org/apache/samza/table/TableManager.java | 2 -
.../table/caching/CachingTableProvider.java | 98 +++++
.../caching/CachingTableProviderFactory.java | 34 ++
.../descriptors/CachingTableDescriptor.java | 166 --------
.../descriptors/CachingTableProvider.java | 105 -----
.../CachingTableProviderFactory.java | 34 --
.../caching/guava/GuavaCacheTableProvider.java | 59 +++
.../guava/GuavaCacheTableProviderFactory.java | 34 ++
.../descriptors/GuavaCacheTableDescriptor.java | 75 ----
.../descriptors/GuavaCacheTableProvider.java | 60 ---
.../GuavaCacheTableProviderFactory.java | 34 --
.../descriptors/BaseHybridTableDescriptor.java | 48 ---
.../table/descriptors/BaseTableDescriptor.java | 110 -----
.../samza/table/remote/RemoteTableProvider.java | 190 +++++++++
.../remote/RemoteTableProviderFactory.java | 38 ++
.../samza/table/remote/TableRateLimiter.java | 167 --------
.../samza/table/remote/TableReadFunction.java | 111 -----
.../samza/table/remote/TableWriteFunction.java | 159 -------
.../descriptors/RemoteTableDescriptor.java | 275 -------------
.../remote/descriptors/RemoteTableProvider.java | 202 ---------
.../descriptors/RemoteTableProviderFactory.java | 38 --
.../table/retry/RetriableReadFunction.java | 2 +-
.../table/retry/RetriableWriteFunction.java | 2 +-
.../samza/table/retry/TableRetryPolicy.java | 257 ------------
.../apache/samza/table/utils/SerdeUtils.java | 66 ---
.../utils/descriptors/BaseTableProvider.java | 73 ----
.../org/apache/samza/zk/ZkJobCoordinator.java | 61 ++-
.../apache/samza/checkpoint/OffsetManager.scala | 8 +-
.../org/apache/samza/config/SystemConfig.scala | 2 -
.../TestJobNodeConfigurationGenerator.java | 4 +-
.../apache/samza/table/TestTableManager.java | 2 -
.../samza/table/caching/TestCachingTable.java | 18 +-
.../table/remote/TestTableRateLimiter.java | 103 -----
.../descriptors/TestRemoteTableDescriptor.java | 14 +-
.../apache/samza/zk/TestZkJobCoordinator.java | 76 +++-
.../samza/config/KafkaConsumerConfig.java | 78 ++--
.../samza/system/kafka/KafkaSystemAdmin.java | 91 +++-
.../samza/system/kafka/KafkaSystemConsumer.java | 1 +
.../kafka/KafkaSystemAdminUtilsScala.scala | 93 +----
.../samza/config/TestKafkaConsumerConfig.java | 114 +++++
.../system/kafka/TestKafkaSystemAdminJava.java | 37 +-
.../kafka/TestKafkaCheckpointManager.scala | 1 +
.../system/kafka/TestKafkaSystemAdmin.scala | 2 +-
.../descriptors/InMemoryTableDescriptor.java | 4 +-
.../descriptors/InMemoryTableProvider.java | 4 +-
.../InMemoryTableProviderFactory.java | 4 +-
.../descriptors/TestInMemoryTableProvider.java | 2 +-
.../kv/descriptors/RocksDbTableDescriptor.java | 3 +-
.../kv/descriptors/RocksDbTableProvider.java | 3 +-
.../RocksDbTableProviderFactory.java | 4 +-
.../descriptors/TestRocksDbTableDescriptor.java | 10 +-
.../descriptors/TestRocksDbTableProvider.java | 2 +-
.../samza/storage/kv/LocalReadWriteTable.java | 149 +++++++
.../samza/storage/kv/LocalReadableTable.java | 108 +++++
.../kv/LocalStoreBackedReadWriteTable.java | 149 -------
.../kv/LocalStoreBackedReadableTable.java | 108 -----
.../samza/storage/kv/LocalTableProvider.java | 148 +++++++
.../BaseLocalStoreBackedTableDescriptor.java | 168 --------
.../BaseLocalStoreBackedTableProvider.java | 149 -------
.../TestBaseLocalStoreBackedTableProvider.java | 150 -------
.../kv/descriptors/TestLocalTableProvider.java | 152 +++++++
.../apache/samza/sql/avro/AvroRelConverter.java | 30 +-
.../interfaces/SamzaRelTableKeyConverter.java | 39 ++
.../SamzaRelTableKeyConverterFactory.java | 39 ++
.../samza/sql/interfaces/SqlIOConfig.java | 48 ++-
.../samza/sql/runner/SamzaSqlApplication.java | 15 +-
.../sql/runner/SamzaSqlApplicationConfig.java | 19 +-
.../sql/runner/SamzaSqlApplicationRunner.java | 4 +-
.../samza/sql/translator/JoinInputNode.java | 76 ++++
.../samza/sql/translator/JoinTranslator.java | 161 +++++---
.../samza/sql/translator/ModifyTranslator.java | 4 +-
.../samza/sql/translator/QueryTranslator.java | 11 +-
.../SamzaSqlLocalTableJoinFunction.java | 54 +++
.../SamzaSqlRelMessageJoinFunction.java | 126 ------
.../SamzaSqlRemoteTableJoinFunction.java | 79 ++++
.../translator/SamzaSqlTableJoinFunction.java | 121 ++++++
.../samza/sql/translator/ScanTranslator.java | 16 +-
.../samza/sql/translator/TranslatorContext.java | 13 +-
.../samza/sql/e2e/TestSamzaSqlRemoteTable.java | 115 ++++++
.../apache/samza/sql/e2e/TestSamzaSqlTable.java | 68 ---
.../runner/TestSamzaSqlApplicationRunner.java | 8 +-
.../samza/sql/system/TestAvroSystemFactory.java | 53 ++-
.../RemoteStoreIOResolverTestFactory.java | 147 +++++++
.../testutil/SampleRelTableKeyConverter.java | 36 ++
.../SampleRelTableKeyConverterFactory.java | 41 ++
.../samza/sql/testutil/SamzaSqlTestConfig.java | 46 ++-
.../sql/testutil/TestIOResolverFactory.java | 234 -----------
.../sql/translator/TestJoinTranslator.java | 60 ++-
.../sql/translator/TestQueryTranslator.java | 40 +-
.../TestSamzaSqlLocalTableJoinFunction.java | 161 ++++++++
.../TestSamzaSqlRelMessageJoinFunction.java | 121 ------
.../TestSamzaSqlRemoteTableJoinFunction.java | 103 +++++
.../processor/TestZkLocalApplicationRunner.java | 65 ++-
.../test/samzasql/TestSamzaSqlEndToEnd.java | 106 +++++
.../apache/samza/test/table/TestLocalTable.java | 4 +-
.../samza/test/table/TestRemoteTable.java | 6 +-
settings.gradle | 23 +-
137 files changed, 5431 insertions(+), 4362 deletions(-)
----------------------------------------------------------------------
[24/50] samza git commit: Clean-up the quickstart page
Posted by ja...@apache.org.
Clean-up the quickstart page
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/d8336be5
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/d8336be5
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/d8336be5
Branch: refs/heads/master
Commit: d8336be5c5eaf17f4a0454a02053a133c3e1ebfc
Parents: 14e6220
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:34:13 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:34:13 2018 -0700
----------------------------------------------------------------------
docs/startup/quick-start/versioned/index.md | 38 +++++++-----------------
1 file changed, 10 insertions(+), 28 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/d8336be5/docs/startup/quick-start/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/quick-start/versioned/index.md b/docs/startup/quick-start/versioned/index.md
index 0c77a41..30add8a 100644
--- a/docs/startup/quick-start/versioned/index.md
+++ b/docs/startup/quick-start/versioned/index.md
@@ -23,7 +23,7 @@ In this tutorial, we will create our first Samza application - `WordCount`. This
### Setting up a Java Project
-First let’s create the project structure as follows:
+Observe the project structure as follows:
{% highlight bash %}
wordcount
@@ -38,7 +38,7 @@ wordcount
|-- WordCount.java
{% endhighlight %}
-You can copy build.gradle and gradle.properties files from the downloaded tutorial tgz file. The WordCount class is just an empty class for now. Once finishing this setup, you can build the project by:
+You can build the project anytime by running:
{% highlight bash %}
> cd wordcount
@@ -81,11 +81,13 @@ public class WordCount implements StreamApplication {
@Override
public void describe(StreamApplicationDescriptor streamApplicationDescriptor) {
+ // Create a KafkaSystemDescriptor providing properties of the cluster
KafkaSystemDescriptor kafkaSystemDescriptor = new KafkaSystemDescriptor(KAFKA_SYSTEM_NAME)
.withConsumerZkConnect(KAFKA_CONSUMER_ZK_CONNECT)
.withProducerBootstrapServers(KAFKA_PRODUCER_BOOTSTRAP_SERVERS)
.withDefaultStreamConfigs(KAFKA_DEFAULT_STREAM_CONFIGS);
+ // For each input or output stream, create a KafkaInput/Output descriptor
KafkaInputDescriptor<KV<String, String>> inputDescriptor =
kafkaSystemDescriptor.getInputDescriptor(INPUT_STREAM_ID,
KVSerde.of(new StringSerde(), new StringSerde()));
@@ -93,6 +95,7 @@ public class WordCount implements StreamApplication {
kafkaSystemDescriptor.getOutputDescriptor(OUTPUT_STREAM_ID,
KVSerde.of(new StringSerde(), new StringSerde()));
+ // Obtain a handle to a MessageStream that you can chain operations on
MessageStream<KV<String, String>> lines = streamApplicationDescriptor.getInputStream(inputDescriptor);
OutputStream<KV<String, String>> counts = streamApplicationDescriptor.getOutputStream(outputDescriptor);
}
@@ -177,7 +180,7 @@ For more details on Samza's configs, feel free to check out the latest [configur
### Run your application
-Let’s now add a `main()` function to the `WordCount` class. The function reads the config file and factory from the args, and creates a `LocalApplicationRunner` that run the application locally.
+We are ready to add a `main()` function to the `WordCount` class. It parses the command-line arguments and instantiates a `LocalApplicationRunner` to execute the application locally.
{% highlight java %}
public static void main(String[] args) {
@@ -190,13 +193,6 @@ public static void main(String[] args) {
}
{% endhighlight %}
-In your "build.gradle" file, please add the following so we can use gradle to run it:
-
-{% highlight jproperties %}
-apply plugin:'application'
-
-mainClassName = "samzaapp.WordCount"
-{% endhighlight %}
Before running `main()`, we will create our input Kafka topic and populate it with sample data. You can download the scripts to interact with Kafka along with the sample data from [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
@@ -211,7 +207,7 @@ Before running `main()`, we will create our input Kafka topic and populate it wi
> ./deploy/kafka/bin/kafka-console-producer.sh --topic sample-text --broker localhost:9092 < ./sample-text.txt
{% endhighlight %}
-Now let’s kick off our application and use gradle to run it. Alternately, you can also run it directly from your IDE, with the same program arguments.
+Let’s kick off our application and use gradle to run it. Alternately, you can also run it directly from your IDE, with the same program arguments.
{% highlight bash %}
> export BASE_DIR=`pwd`
@@ -219,7 +215,7 @@ Now let’s kick off our application and use gradle to run it. Alternately, you
{% endhighlight %}
-The application will output to a Kafka topic named "word-count-output". Let’s fire up a Kafka consumer to read from this topic:
+The application will output to a Kafka topic named "word-count-output". We will now fire up a Kafka consumer to read from this topic:
{% highlight bash %}
> ./deploy/kafka/bin/kafka-console-consumer.sh --topic word-count-output --zookeeper localhost:2181 --from-beginning
@@ -244,20 +240,6 @@ and: 243
from: 16
{% endhighlight %}
-### More Examples
-
-The [hello-samza](https://github.com/apache/samza-hello-samza) project contains a lot of more examples to help you create your Samza job. To checkout the hello-samza project:
-
-{% highlight bash %}
-> git clone https://git.apache.org/samza-hello-samza.git hello-samza
-{% endhighlight %}
-
-There are four main categories of examples in this project, including:
-
-1. [Wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
-
-2. [Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
-
-3. [Azure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): This example shows how to build an application that consumes input streams from Azure EventHubs.
+Congratulations! You've successfully run your first Samza application.
-4. [Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): This example shows how to consume from Kinesis streams.
\ No newline at end of file
+### [More Examples >>](/startup/code-examples/{{site.version}})
\ No newline at end of file
[17/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/590eb14b
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/590eb14b
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/590eb14b
Branch: refs/heads/master
Commit: 590eb14b707eccd28bfbe64c8768ea353dc214ac
Parents: 50fa974 e203db2
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 22 16:30:01 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 22 16:30:01 2018 -0700
----------------------------------------------------------------------
build.gradle | 28 +
gradle/dependency-versions.gradle | 2 +
.../JobNodeConfigurationGenerator.java | 6 +-
samza-sql-shell/conf/samza-sql-shell-log4j.xml | 49 ++
samza-sql-shell/conf/shell-defaults.conf | 28 +
samza-sql-shell/scripts/samza-sql-shell.sh | 42 +
.../apache/samza/sql/client/cli/CliCommand.java | 53 ++
.../samza/sql/client/cli/CliCommandType.java | 78 ++
.../samza/sql/client/cli/CliConstants.java | 57 ++
.../samza/sql/client/cli/CliEnvironment.java | 130 +++
.../samza/sql/client/cli/CliHighlighter.java | 89 ++
.../apache/samza/sql/client/cli/CliShell.java | 821 +++++++++++++++++++
.../apache/samza/sql/client/cli/CliView.java | 29 +
.../org/apache/samza/sql/client/cli/Main.java | 117 +++
.../sql/client/cli/QueryResultLogView.java | 291 +++++++
.../sql/client/impl/AvroSqlSchemaConverter.java | 112 +++
.../client/impl/CliLoggingSystemFactory.java | 117 +++
.../FileSystemAvroRelSchemaProviderFactory.java | 75 ++
.../samza/sql/client/impl/SamzaExecutor.java | 511 ++++++++++++
.../sql/client/impl/SamzaSqlFieldType.java | 94 +++
.../sql/client/impl/SamzaSqlUdfDisplayInfo.java | 71 ++
.../sql/client/interfaces/ExecutionContext.java | 41 +
.../client/interfaces/ExecutionException.java | 40 +
.../sql/client/interfaces/ExecutionStatus.java | 30 +
.../sql/client/interfaces/NonQueryResult.java | 62 ++
.../sql/client/interfaces/QueryResult.java | 49 ++
.../sql/client/interfaces/SqlExecutor.java | 171 ++++
.../sql/client/interfaces/SqlFunction.java | 55 ++
.../samza/sql/client/interfaces/SqlSchema.java | 56 ++
.../sql/client/interfaces/SqlSchemaBuilder.java | 63 ++
.../samza/sql/client/util/CliException.java | 41 +
.../apache/samza/sql/client/util/CliUtil.java | 43 +
.../sql/client/util/RandomAccessQueue.java | 96 +++
.../sql/client/impl/SamzaExecutorTest.java | 79 ++
.../sql/client/util/RandomAccessQueueTest.java | 89 ++
.../src/test/resources/ProfileChangeStream.avsc | 51 ++
.../sql/runner/SamzaSqlApplicationConfig.java | 2 +-
settings.gradle | 1 +
38 files changed, 3764 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[39/50] samza git commit: Close timeseries store iterators on deletes
Posted by ja...@apache.org.
Close timeseries store iterators on deletes
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/018a6e45
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/018a6e45
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/018a6e45
Branch: refs/heads/master
Commit: 018a6e45ea04a150155788fc32ec7a1ffb81d625
Parents: ff6032c
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 30 18:45:37 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 30 18:45:37 2018 -0700
----------------------------------------------------------------------
.../org/apache/samza/operators/impl/WindowOperatorImpl.java | 4 ++--
.../samza/operators/impl/store/TimeSeriesStoreImpl.java | 9 ++++++---
2 files changed, 8 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/018a6e45/samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java b/samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java
index c09c5f8..0241d9e 100644
--- a/samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java
+++ b/samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java
@@ -379,15 +379,15 @@ public class WindowOperatorImpl<M, K> extends OperatorImpl<M, WindowPane<K, Obje
* @return a list of all elements returned by the iterator
*/
static <V> List<V> toList(ClosableIterator<V> iterator) {
+ Preconditions.checkNotNull(iterator);
+
List<V> values = new ArrayList<>();
try {
while (iterator.hasNext()) {
values.add(iterator.next());
}
} finally {
- if (iterator != null) {
iterator.close();
- }
}
return Collections.unmodifiableList(values);
}
http://git-wip-us.apache.org/repos/asf/samza/blob/018a6e45/samza-core/src/main/java/org/apache/samza/operators/impl/store/TimeSeriesStoreImpl.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/operators/impl/store/TimeSeriesStoreImpl.java b/samza-core/src/main/java/org/apache/samza/operators/impl/store/TimeSeriesStoreImpl.java
index 10a5967..b8cd82f 100644
--- a/samza-core/src/main/java/org/apache/samza/operators/impl/store/TimeSeriesStoreImpl.java
+++ b/samza-core/src/main/java/org/apache/samza/operators/impl/store/TimeSeriesStoreImpl.java
@@ -154,10 +154,13 @@ public class TimeSeriesStoreImpl<K, V> implements TimeSeriesStore<K, V> {
List<TimeSeriesKey<K>> keysToDelete = new LinkedList<>();
KeyValueIterator<TimeSeriesKey<K>, V> range = kvStore.range(fromKey, toKey);
- while (range.hasNext()) {
- keysToDelete.add(range.next().getKey());
+ try {
+ while (range.hasNext()) {
+ keysToDelete.add(range.next().getKey());
+ }
+ } finally {
+ range.close();
}
-
kvStore.deleteAll(keysToDelete);
}
[33/50] samza git commit: Cleanup the connectors overview and Kafka
connector sections. Use System/Stream Descriptors
Posted by ja...@apache.org.
Cleanup the connectors overview and Kafka connector sections. Use System/Stream Descriptors
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/c9596b24
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/c9596b24
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/c9596b24
Branch: refs/heads/master
Commit: c9596b243b8c12c502212a0acb413c5f27b3f52e
Parents: 7d4c2ed
Author: Jagadish <jv...@linkedin.com>
Authored: Sun Oct 28 11:26:46 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sun Oct 28 11:26:46 2018 -0700
----------------------------------------------------------------------
.../documentation/versioned/connectors/kafka.md | 168 +++++++++++--------
.../versioned/connectors/overview.md | 38 ++---
2 files changed, 112 insertions(+), 94 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/c9596b24/docs/learn/documentation/versioned/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/kafka.md b/docs/learn/documentation/versioned/connectors/kafka.md
index b6e78e4..447bfdc 100644
--- a/docs/learn/documentation/versioned/connectors/kafka.md
+++ b/docs/learn/documentation/versioned/connectors/kafka.md
@@ -19,108 +19,132 @@ title: Kafka Connector
limitations under the License.
-->
-## Overview
-Samza offers built-in integration with Apache Kafka for stream processing. A common pattern in Samza applications is to read messages from one or more Kafka topics, process it and emit results to other Kafka topics or external systems.
+### Kafka I/O : QuickStart
+Samza offers built-in integration with Apache Kafka for stream processing. A common pattern in Samza applications is to read messages from one or more Kafka topics, process them and emit results to other Kafka topics or databases.
-## Consuming from Kafka
+The `hello-samza` project includes multiple examples on interacting with Kafka from your Samza jobs. Each example also includes instructions on how to run them and view results.
-### <a name="kafka-basic-configuration"></a>Basic Configuration
+- [High-level API Example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) with a corresponding [tutorial](/learn/documentation/{{site.version}}/deployment/yarn.html#starting-your-application-on-yarn)
-The example below provides a basic example for configuring a system called `kafka-cluster-1` that uses the provided KafkaSystemFactory.
+- [Low-level API Example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/wikipedia/task/application/WikipediaParserTaskApplication.java) with a corresponding [tutorial](https://github.com/apache/samza-hello-samza#hello-samza)
-{% highlight jproperties %}
-# Set the SystemFactory implementation to instantiate KafkaSystemConsumer, KafkaSystemProducer and KafkaSystemAdmin
-systems.kafka-cluster-1.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
-# Define the default key and message SerDe.
-systems.kafka-cluster-1.default.stream.samza.key.serde=string
-systems.kafka-cluster-1.default.stream.samza.msg.serde=json
+### Concepts
-# Zookeeper nodes of the Kafka cluster
-systems.kafka-cluster-1.consumer.zookeeper.connect=localhost:2181
+####KafkaSystemDescriptor
-# List of network endpoints where Kafka brokers are running. Also needed by consumers for querying metadata.
-systems.kafka-cluster-1.producer.bootstrap.servers=localhost:9092,localhost:9093
+Samza refers to any IO source (eg: Kafka) it interacts with as a _system_, whose properties are set using a corresponding `SystemDescriptor`. The `KafkaSystemDescriptor` allows you to describe the Kafka cluster you are interacting with and specify its properties.
+
+{% highlight java %}
+ KafkaSystemDescriptor kafkaSystemDescriptor =
+ new KafkaSystemDescriptor("kafka").withConsumerZkConnect(KAFKA_CONSUMER_ZK_CONNECT)
+ .withProducerBootstrapServers(KAFKA_PRODUCER_BOOTSTRAP_SERVERS)
+ .withDefaultStreamConfigs(KAFKA_DEFAULT_STREAM_CONFIGS);
{% endhighlight %}
-Samza provides a built-in KafkaSystemDescriptor to consume from and produce to Kafka from the StreamApplication (High-level API) or the TaskApplication (Low-level API).
-Below is an example of how to use the descriptors in the describe method of a StreamApplication.
+####KafkaInputDescriptor
+A Kafka cluster usually has multiple topics (a.k.a _streams_). The `KafkaInputDescriptor` allows you to specify the properties of each Kafka topic your application should read from. For each of your input topics, you should create a corresponding instance of `KafkaInputDescriptor`
+by providing a topic-name and a serializer.
{% highlight java %}
-public class PageViewFilter implements StreamApplication {
- @Override
- public void describe(StreamApplicationDescriptor appDesc) {
- // add input and output streams
- KafkaSystemDescriptor ksd = new KafkaSystemDescriptor("kafka-cluster-1");
- KafkaInputDescriptor<PageView> isd = ksd.getInputDescriptor("myinput", new JsonSerdeV2<>(PageView.class));
- KafkaOutputDescriptor<DecoratedPageView> osd = ksd.getOutputDescriptor("myout", new JsonSerdeV2<>(DecordatedPageView.class));
-
- MessageStream<PageView> ms = appDesc.getInputStream(isd);
- OutputStream<DecoratedPageView> os = appDesc.getOutputStream(osd);
-
- ms.filter(this::isValidPageView)
- .map(this::addProfileInformation)
- .sendTo(os);
- }
-}
+ KafkaInputDescriptor<PageView> pageViewStreamDescriptor = kafkaSystemDescriptor.getInputDescriptor("page-view-topic", new JsonSerdeV2<>(PageView.class));
{% endhighlight %}
-Below is an example of how to use the descriptors in the describe method of a TaskApplication
+The above example describes an input Kafka stream from the "page-view-topic" which Samza de-serializes into a JSON payload. Samza provides default serializers for common data-types like string, avro, bytes, integer etc.
+
+####KafkaOutputDescriptor
+
+Similarly, the `KafkaOutputDescriptor` allows you to specify the output streams for your application. For each output topic you write to, you should create an instance of `KafkaOutputDescriptor`.
{% highlight java %}
-public class PageViewFilterTask implements TaskApplication {
- @Override
- public void describe(TaskApplicationDescriptor appDesc) {
- // add input and output streams
- KafkaSystemDescriptor ksd = new KafkaSystemDescriptor("kafka-cluster-1");
- KafkaInputDescriptor<String> isd = ksd.getInputDescriptor("myinput", new StringSerde());
- KafkaOutputDescriptor<String> osd = ksd.getOutputDescriptor("myout", new StringSerde());
-
- appDesc.addInputStream(isd);
- appDesc.addOutputStream(osd);
- appDesc.addTable(td);
-
- appDesc.withTaskFactory((StreamTaskFactory) () -> new MyStreamTask());
- }
-}
+ KafkaOutputDescriptor<DecoratedPageView> decoratedPageView = kafkaSystemDescriptor.getOutputDescriptor("my-output-topic", new JsonSerdeV2<>(DecoratedPageView.class));
{% endhighlight %}
-### Advanced Configuration
-Prefix the configuration with `systems.system-name.consumer.` followed by any of the Kafka consumer configurations. See [Kafka Consumer Configuration Documentation](http://kafka.apache.org/documentation.html#consumerconfigs)
+### Configuration
-{% highlight jproperties %}
-systems.kafka-cluster-1.consumer.security.protocol=SSL
-systems.kafka-cluster-1.consumer.max.partition.fetch.bytes=524288
+#####Configuring Kafka producer and consumer
+
+The `KafkaSystemDescriptor` allows you to specify any [Kafka producer](https://kafka.apache.org/documentation/#producerconfigs) or [Kafka consumer](https://kafka.apache.org/documentation/#consumerconfigs)) property which are directly passed over to the underlying Kafka client. This allows for
+precise control over the KafkaProducer and KafkaConsumer used by Samza.
+
+{% highlight java %}
+ KafkaSystemDescriptor kafkaSystemDescriptor =
+ new KafkaSystemDescriptor("kafka").withConsumerZkConnect(..)
+ .withProducerBootstrapServers(..)
+ .withConsumerConfigs(..)
+ .withProducerConfigs(..)
{% endhighlight %}
-## Producing to Kafka
-### Basic Configuration
+####Accessing an offset which is out-of-range
+This setting determines the behavior if a consumer attempts to read an offset that is outside of the current valid range maintained by the broker. This could happen if the topic does not exist, or if a checkpoint is older than the maximum message history retained by the brokers.
-The basic configuration is the same as [Consuming from Kafka](#kafka-basic-configuration).
+{% highlight java %}
+ KafkaSystemDescriptor kafkaSystemDescriptor =
+ new KafkaSystemDescriptor("kafka").withConsumerZkConnect(..)
+ .withProducerBootstrapServers(..)
+ .withConsumerAutoOffsetReset("largest")
+{% endhighlight %}
-### Advanced Configuration
-#### Changelog to Kafka for State Stores
+#####Ignoring checkpointed offsets
+Samza periodically persists the last processed Kafka offsets as a part of its checkpoint. During startup, Samza resumes consumption from the previously checkpointed offsets by default. You can over-ride this behavior and configure Samza to ignore checkpoints with `KafkaInputDescriptor#shouldResetOffset()`.
+Once there are no checkpoints for a stream, the `#withOffsetDefault(..)` determines whether we start consumption from the oldest or newest offset.
-For Samza processors that have local state and is configured with a changelog for durability, if the changelog is configured to use Kafka, there are Kafka specific configuration parameters.
-See section on `TODO: link to state management section` State Management `\TODO` for more details.
+{% highlight java %}
+KafkaInputDescriptor<PageView> pageViewStreamDescriptor =
+ kafkaSystemDescriptor.getInputDescriptor("page-view-topic", new JsonSerdeV2<>(PageView.class))
+ .shouldResetOffset()
+ .withOffsetDefault(OffsetType.OLDEST);
-{% highlight jproperties %}
-stores.store-name.changelog=kafka-cluster-2.changelog-topic-name
-stores.store-name.changelog.replication.factor=3
-stores.store-name.changelog.kafka.cleanup.policy=compact
{% endhighlight %}
-#### Performance Tuning
+The above example configures Samza to ignore checkpointed offsets for `page-view-topic` and consume from the oldest available offset during startup. You can configure this behavior to apply to all topics in the Kafka cluster by using `KafkaSystemDescriptor#withDefaultStreamOffsetDefault`.
+
+
-Increasing the consumer fetch buffer thresholds may improve throughput at the expense of memory by buffering more messages. Run some performance analysis to find the optimal values.
+### Code walkthrough
+
+In this section, we walk through a complete example.
+
+#### High-level API
+{% highlight java %}
+// Define coordinates of the Kafka cluster using the KafkaSystemDescriptor
+1 KafkaSystemDescriptor kafkaSystemDescriptor = new KafkaSystemDescriptor("kafka")
+2 .withConsumerZkConnect(KAFKA_CONSUMER_ZK_CONNECT)
+3 .withProducerBootstrapServers(KAFKA_PRODUCER_BOOTSTRAP_SERVERS)
+
+// Create an KafkaInputDescriptor for your input topic and a KafkaOutputDescriptor for the output topic
+4 KVSerde<String, PageView> serde = KVSerde.of(new StringSerde(), new JsonSerdeV2<>(PageView.class));
+5 KafkaInputDescriptor<KV<String, PageView>> inputDescriptor =
+6 kafkaSystemDescriptor.getInputDescriptor("page-views", serde);
+7 KafkaOutputDescriptor<KV<String, PageView>> outputDescriptor =
+8 kafkaSystemDescriptor.getOutputDescriptor("filtered-page-views", serde);
+
+
+// Obtain a message stream the input topic
+9 MessageStream<KV<String, PageView>> pageViews = appDescriptor.getInputStream(inputDescriptor);
+
+// Obtain an output stream for the topic
+10 OutputStream<KV<String, PageView>> filteredPageViews = appDescriptor.getOutputStream(outputDescriptor);
+
+// write results to the output topic
+11 pageViews
+12 .filter(kv -> !INVALID_USER_ID.equals(kv.value.userId))
+13 .sendTo(filteredPageViews);
-{% highlight jproperties %}
-# Max number of messages to buffer across all Kafka input topic partitions per container. Default is 50000 messages.
-systems.kafka-cluster-1.samza.fetch.threshold=10000
-# Max buffer size by bytes. This configuration takes precedence over the above configuration if value is not -1. Default is -1.
-systems.kafka-cluster-1.samza.fetch.threshold.bytes=-1
{% endhighlight %}
+
+- Lines 1-3 create a KafkaSystemDescriptor defining the coordinates of our Kafka cluster
+
+- Lines 4-6 defines a KafkaInputDescriptor for our input topic - `page-views`
+
+- Lines 7-9 defines a KafkaOutputDescriptor for our output topic - `filtered-page-views`
+
+- Line 9 creates a MessageStream for the input topic so that you can chain operations on it later
+
+- Line 10 creates an OuputStream for the output topic
+
+- Lines 11-13 define a simple pipeline that reads from the input stream and writes filtered results to the output stream
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/c9596b24/docs/learn/documentation/versioned/connectors/overview.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/overview.md b/docs/learn/documentation/versioned/connectors/overview.md
index 5b6ba39..6697b9c 100644
--- a/docs/learn/documentation/versioned/connectors/overview.md
+++ b/docs/learn/documentation/versioned/connectors/overview.md
@@ -20,33 +20,27 @@ title: Connectors overview
-->
Stream processing applications often read data from external sources like Kafka or HDFS. Likewise, they require processed
-results to be written to external system or data stores. As of the 1.0 release, Samza integrates with the following systems
-out-of-the-box:
+results to be written to external system or data stores. Samza is pluggable and designed to support a variety of [producers](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/system/SystemProducer.html) and [consumers](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/system/SystemConsumer.html) for your data. You can
+integrate Samza with any streaming system by implementing the [SystemFactory](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/system/SystemFactory.html) interface.
-- [Apache Kafka](kafka) (consumer/producer)
-- [Microsoft Azure Eventhubs](eventhubs) (consumer/producer)
-- [Amazon AWS Kinesis Streams](kinesis) (consumer)
-- [Hadoop Filesystem](hdfs) (consumer/producer)
-- [Elasticsearch](https://github.com/apache/samza/blob/master/samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemProducer.java) (producer)
+The following integrations are supported out-of-the-box:
-Instructions on how to use these connectors can be found in the corresponding subsections. Please note that the
-connector API is different from [Samza Table API](../api/table-api), where the data could be read from and written to
-data stores.
+Consumers:
-Samza is pluggable and designed to support a variety of producers and consumers. You can provide your own producer or
-consumer by implementing the SystemFactory interface.
+- [Apache Kafka](kafka)
-To associate a system with a Samza Connector, the user needs to set the following config:
+- [Microsoft Azure Eventhubs](eventhubs)
-{% highlight jproperties %}
-systems.<system-name>.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
-{% endhighlight %}
+- [Amazon AWS Kinesis Streams](kinesis)
-Any system specific configs, could be defined as below:
+- [Hadoop Filesystem](hdfs)
-{% highlight jproperties %}
-systems.<system-name>.param1=value1
-systems.<system-name>.consumer.param2=value2
-systems.<system-name>.producer.param3=value3
-{% endhighlight %}
+Producers:
+- [Apache Kafka](kafka)
+
+- [Microsoft Azure Eventhubs](eventhubs)
+
+- [Hadoop Filesystem](hdfs)
+
+- [Elasticsearch](https://github.com/apache/samza/blob/master/samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemProducer.java)
[48/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/32af8e07
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/32af8e07
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/32af8e07
Branch: refs/heads/master
Commit: 32af8e07e1e467a85a6f6a46a813337bc933a886
Parents: d6b8988 f529722
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Nov 27 00:27:11 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Nov 27 00:27:11 2018 -0800
----------------------------------------------------------------------
.../samza/table/TableProviderFactory.java | 4 +-
.../apache/samza/table/BaseTableProvider.java | 8 +-
.../org/apache/samza/table/TableManager.java | 2 +-
.../table/caching/CachingTableProvider.java | 15 +-
.../caching/CachingTableProviderFactory.java | 5 +-
.../caching/guava/GuavaCacheTableProvider.java | 9 +-
.../guava/GuavaCacheTableProviderFactory.java | 5 +-
.../samza/table/remote/RemoteTableProvider.java | 23 +-
.../remote/RemoteTableProviderFactory.java | 5 +-
.../apache/samza/table/TestTableManager.java | 3 +-
.../samza/table/caching/TestCachingTable.java | 8 +-
.../descriptors/TestLocalTableDescriptor.java | 2 +-
.../table/remote/TestRemoteReadWriteTable.java | 458 +++++++++++++++++++
.../samza/table/remote/TestRemoteTable.java | 458 -------------------
.../descriptors/TestRemoteTableDescriptor.java | 19 +-
.../retry/TestRetriableTableFunctions.java | 4 +-
.../samza/storage/kv/LocalTableProvider.java | 6 +-
.../storage/kv/LocalTableProviderFactory.java | 5 +-
.../kv/descriptors/TestLocalTableProvider.java | 3 +-
.../samza/test/table/TestRemoteTable.java | 9 +-
20 files changed, 524 insertions(+), 527 deletions(-)
----------------------------------------------------------------------
[37/50] samza git commit: Cleanup documentation for Kinesis
Posted by ja...@apache.org.
Cleanup documentation for Kinesis
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/0b4c6893
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/0b4c6893
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/0b4c6893
Branch: refs/heads/master
Commit: 0b4c6893502e737816e1ec8c4b47681f2b9e58b4
Parents: 7d74781
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 30 12:05:49 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 30 12:05:49 2018 -0700
----------------------------------------------------------------------
.../versioned/connectors/kinesis.md | 106 +++++++++++--------
1 file changed, 63 insertions(+), 43 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/0b4c6893/docs/learn/documentation/versioned/connectors/kinesis.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/kinesis.md b/docs/learn/documentation/versioned/connectors/kinesis.md
index a866484..4f667d2 100644
--- a/docs/learn/documentation/versioned/connectors/kinesis.md
+++ b/docs/learn/documentation/versioned/connectors/kinesis.md
@@ -19,11 +19,18 @@ title: Kinesis Connector
limitations under the License.
-->
-## Overview
+## Kinesis I/O: Quickstart
-The Samza Kinesis connector provides access to [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams),
-Amazon’s data streaming service. A Kinesis Data Stream is similar to a Kafka topic and can have multiple partitions.
-Each message consumed from a Kinesis Data Stream is an instance of [Record](http://docs.aws.amazon.com/goto/WebAPI/kinesis-2013-12-02/Record).
+The Samza Kinesis connector allows you to interact with [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams),
+Amazon’s data streaming service.
+
+The `hello-samza` project includes an example of processing Kinesis streams using Samza. Here is the complete [source code](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/kinesis/KinesisHelloSamza.java) and [configs](https://github.com/apache/samza-hello-samza/blob/master/src/main/config/kinesis-hello-samza.properties).
+You can build and run this example using this [tutorial](https://github.com/apache/samza-hello-samza#hello-samza).
+
+
+###Data Format
+Like a Kafka topic, a Kinesis stream can have multiple shards with producers and consumers.
+Each message consumed from the stream is an instance of a Kinesis [Record](http://docs.aws.amazon.com/goto/WebAPI/kinesis-2013-12-02/Record).
Samza’s [KinesisSystemConsumer](https://github.com/apache/samza/blob/master/samza-aws/src/main/java/org/apache/samza/system/kinesis/consumer/KinesisSystemConsumer.java)
wraps the Record into a [KinesisIncomingMessageEnvelope](https://github.com/apache/samza/blob/master/samza-aws/src/main/java/org/apache/samza/system/kinesis/consumer/KinesisIncomingMessageEnvelope.java).
@@ -31,90 +38,103 @@ wraps the Record into a [KinesisIncomingMessageEnvelope](https://github.com/apac
### Basic Configuration
-You can configure your Samza jobs to process data from Kinesis Streams. To configure Samza job to consume from Kinesis
-streams, please add the below configuration:
+Here is the required configuration for consuming messages from Kinesis.
{% highlight jproperties %}
-// define a kinesis system factory with your identifier. eg: kinesis-system
-systems.kinesis-system.samza.factory=org.apache.samza.system.eventhub.KinesisSystemFactory
+// Define a Kinesis system factory with your identifier. eg: kinesis-system
+systems.kinesis-system.samza.factory=org.apache.samza.system.kinesis.KinesisSystemFactory
-// kinesis system consumer works with only AllSspToSingleTaskGrouperFactory
+// Kinesis consumer works with only AllSspToSingleTaskGrouperFactory
job.systemstreampartition.grouper.factory=org.apache.samza.container.grouper.stream.AllSspToSingleTaskGrouperFactory
-// define your streams
+// Define your streams
task.inputs=kinesis-system.input0
-// define required properties for your streams
+// Define required properties for your streams
systems.kinesis-system.streams.input0.aws.region=YOUR-STREAM-REGION
systems.kinesis-system.streams.input0.aws.accessKey=YOUR-ACCESS_KEY
sensitive.systems.kinesis-system.streams.input0.aws.secretKey=YOUR-SECRET-KEY
{% endhighlight %}
-The tuple required to access the Kinesis data stream must be provided, namely the following fields:<br>
-**YOUR-STREAM-REGION**, **YOUR-ACCESS-KEY**, **YOUR-SECRET-KEY**.
+####Coordination
+The Kinesis system consumer does not rely on Samza's coordination mechanism. Instead, it uses the Kinesis client library (KCL) for coordination and distributing available shards among available instances. Hence, you should
+set your `grouper` configuration to `AllSspToSingleTaskGrouperFactory`.
+{% highlight jproperties %}
+job.systemstreampartition.grouper.factory=org.apache.samza.container.grouper.stream.AllSspToSingleTaskGrouperFactory
+{% endhighlight %}
-### Advanced Configuration
+####Security
-#### AWS Client configs
-You can configure any [AWS client config](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html)
-with the prefix **systems.system-name.aws.clientConfig.***
+Each Kinesis stream in a given AWS [region](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) can be accessed by providing an [access key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). An Access key consists of two parts: an access key ID (for example, `AKIAIOSFODNN7EXAMPLE`) and a secret access key (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`) which you can use to send programmatic requests to AWS.
{% highlight jproperties %}
-systems.system-name.aws.clientConfig.CONFIG-PARAM=CONFIG-VALUE
+systems.kinesis-system.streams.input0.aws.region=YOUR-STREAM-REGION
+systems.kinesis-system.streams.input0.aws.accessKey=YOUR-ACCESS_KEY
+sensitive.systems.kinesis-system.streams.input0.aws.secretKey=YOUR-SECRET-KEY
{% endhighlight %}
-As an example, to set a *proxy host* and *proxy port* for the AWS Client:
+### Advanced Configuration
+
+#### Kinesis Client Library Configs
+Samza Kinesis Connector uses the [Kinesis Client Library](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html#kinesis-record-processor-overview-kcl)
+(KCL) to access the Kinesis data streams. You can set any [KCL Configuration](https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/coordinator/KinesisClientLibConfiguration.java)
+for a stream by configuring it with the **systems.system-name.streams.stream-name.aws.kcl.*** prefix.
{% highlight jproperties %}
-systems.system-name.aws.clientConfig.ProxyHost=my-proxy-host.com
-systems.system-name.aws.clientConfig.ProxyPort=my-proxy-port
+systems.system-name.streams.stream-name.aws.kcl.CONFIG-PARAM=CONFIG-VALUE
{% endhighlight %}
-#### Kinesis Client Library Configs
-Samza Kinesis Connector uses [Kinesis Client Library](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html#kinesis-record-processor-overview-kcl)
-(KCL) to access the Kinesis data streams. You can set any [Kinesis Client Lib Configuration](https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/coordinator/KinesisClientLibConfiguration.java)
-for a stream by configuring it under **systems.system-name.streams.stream-name.aws.kcl.***
+As an example, the below configuration is equivalent to invoking `kclClient#WithTableName(myTable)` on the KCL instance.
+{% highlight jproperties %}
+systems.system-name.streams.stream-name.aws.kcl.TableName=myTable
+{% endhighlight %}
+
+#### AWS Client configs
+Samza allows you to specify any [AWS client configs](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html) to connect to your Kinesis instance.
+You can configure any [AWS client configuration](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html) with the `systems.your-system-name.aws.clientConfig.*` prefix.
{% highlight jproperties %}
-systems.system-name.streams.stream-name.aws.kcl.CONFIG-PARAM=CONFIG-VALUE
+systems.system-name.aws.clientConfig.CONFIG-PARAM=CONFIG-VALUE
{% endhighlight %}
-Obtain the config param from the public functions in [Kinesis Client Lib Configuration](https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/coordinator/KinesisClientLibConfiguration.java)
-by removing the *"with"* prefix. For example: config param corresponding to **withTableName()** is **TableName**.
+As an example, to set the *proxy host* and *proxy port* to be used by the Kinesis Client:
+{% highlight jproperties %}
+systems.system-name.aws.clientConfig.ProxyHost=my-proxy-host.com
+systems.system-name.aws.clientConfig.ProxyPort=my-proxy-port
+{% endhighlight %}
### Resetting Offsets
-The source of truth for checkpointing while using Kinesis Connector is not the Samza checkpoint topic but Kinesis itself.
-The Kinesis Client Library (KCL) [uses DynamoDB](https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html)
-to store it’s checkpoints. By default, Kinesis Connector reads from the latest offset in the stream.
-
-To reset the checkpoints and consume from earliest/latest offset of a Kinesis data stream, please change the KCL TableName
-and set the appropriate starting position for the stream as shown below.
+Unlike other connectors where Samza stores and manages checkpointed offsets, Kinesis checkpoints are stored in a [DynamoDB](https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html) table.
+These checkpoints are stored and managed by the KCL library internally. You can reset the checkpoints by configuring a different name for the DynamoDB table.
{% highlight jproperties %}
-// change the TableName to a unique name to reset checkpoint.
+// change the TableName to a unique name to reset checkpoints.
systems.kinesis-system.streams.input0.aws.kcl.TableName=my-app-table-name
+{% endhighlight %}
+
+When you reset checkpoints, you can configure your job to start consuming from either the earliest or latest offset in the stream.
+
+{% highlight jproperties %}
// set the starting position to either TRIM_HORIZON (oldest) or LATEST (latest)
-systems.kinesis-system.streams.input0.aws.kcl.InitialPositionInStream=my-start-position
+systems.kinesis-system.streams.input0.aws.kcl.InitialPositionInStream=LATEST
{% endhighlight %}
-To manipulate checkpoints to start from a particular position in the Kinesis stream, in lieu of Samza CheckpointTool,
-please login to the AWS Console and change the offsets in the DynamoDB Table with the table name that you have specified
-in the config above. By default, the table name has the following format:
-"\<job name\>-\<job id\>-\<kinesis stream\>".
+Alternately, if you want to start from a particular offset in the Kinesis stream, you can login to the [AWS console](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ConsoleDynamoDB.html) and edit the offsets in your DynamoDB Table.
+By default, the table-name has the following format: "\<job name\>-\<job id\>-\<kinesis stream\>".
### Known Limitations
-The following limitations apply to Samza jobs consuming from Kinesis streams using the Samza consumer:
+The following limitations apply to Samza jobs consuming from Kinesis streams :
- Stateful processing (eg: windows or joins) is not supported on Kinesis streams. However, you can accomplish this by
chaining two Samza jobs where the first job reads from Kinesis and sends to Kafka while the second job processes the
data from Kafka.
- Kinesis streams cannot be configured as [bootstrap](https://samza.apache.org/learn/documentation/latest/container/streams.html)
or [broadcast](https://samza.apache.org/learn/documentation/latest/container/samza-container.html) streams.
-- Kinesis streams must be used ONLY with the [AllSspToSingleTaskGrouperFactory](https://github.com/apache/samza/blob/master/samza-core/src/main/java/org/apache/samza/container/grouper/stream/AllSspToSingleTaskGrouperFactory.java)
-as the Kinesis consumer does the partition management by itself. No other grouper is supported.
+- Kinesis streams must be used only with the [AllSspToSingleTaskGrouperFactory](https://github.com/apache/samza/blob/master/samza-core/src/main/java/org/apache/samza/container/grouper/stream/AllSspToSingleTaskGrouperFactory.java)
+as the Kinesis consumer does the partition management by itself. No other grouper is currently supported.
- A Samza job that consumes from Kinesis cannot consume from any other input source. However, you can send your results
to any destination (eg: Kafka, EventHubs), and have another Samza job consume them.
[28/50] samza git commit: Clean up the deployment models section
Posted by ja...@apache.org.
Clean up the deployment models section
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/c21593bb
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/c21593bb
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/c21593bb
Branch: refs/heads/master
Commit: c21593bb466a40440e6211821e550e2ad9628d83
Parents: 796aa40
Author: Jagadish <jv...@linkedin.com>
Authored: Wed Oct 24 15:18:04 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Wed Oct 24 15:18:04 2018 -0700
----------------------------------------------------------------------
.../versioned/deployment/deployment-model.md | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/c21593bb/docs/learn/documentation/versioned/deployment/deployment-model.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/deployment/deployment-model.md b/docs/learn/documentation/versioned/deployment/deployment-model.md
index 81de2eb..ba100da 100644
--- a/docs/learn/documentation/versioned/deployment/deployment-model.md
+++ b/docs/learn/documentation/versioned/deployment/deployment-model.md
@@ -1,6 +1,6 @@
---
layout: page
-title: Deployment model
+title: Deployment options
---
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -20,22 +20,24 @@ title: Deployment model
-->
### Overview
-One unique thing about Samza is that it provides multiple ways to deploy an application. Each deployment model comes with its own benefits, so you have flexibility in choosing which model best fits your needs. Samza supports “write once, run anywhere”, so application logic is the same regardless of where you choose to deploy your application.
+A unique thing about Samza is that it provides multiple ways to run your applications. Each deployment model comes with its own benefits, so you have flexibility in choosing whichever fits your needs. Since Samza supports “Write Once, Run Anywhere”, your application logic does not change depending on where you deploy it.
### Running Samza on YARN
-Samza integrates with Apache YARN for running stream-processing as a managed service. Samza leverages YARN for multi-tenancy, resource-management, isolation and deployment for your applications. In this mode, you write your Samza application and submit it to be scheduled on a YARN cluster. You also specify its resource requirements - the number of containers needed, number of cores and memory per-container. Samza then works with YARN to provision resources for your application and run it across a cluster of machines. It also handles failures of individual instances and restarts them.
+Samza integrates with [Apache YARN](learn/documentation/{{site.version}}/deployment/yarn.html) for running stream-processing as a managed service. We leverage YARN for isolation, multi-tenancy, resource-management and deployment for your applications. In this mode, you write your Samza application and submit it to be scheduled on a YARN cluster. You also specify its resource requirement - the number of containers needed, number of cores and memory required per-container. Samza then works with YARN to provision resources for your application and run it across a cluster of machines. It also handles failures of individual instances and automatically restarts them.
-When multiple applications share the same YARN cluster, they need to be isolated from each other. For this purpose, Samza works with YARN to enforce cpu and memory limits. Any application that uses more than its requested share of memory or cpu is terminated - thereby, enabling multi-tenancy. Just like you would for any YARN-based application, you can use YARN's web UI to manage your Samza jobs, view their logs etc.
+When multiple applications share the same YARN cluster, they need to be isolated from each other. For this purpose, Samza works with YARN to enforce cpu and memory limits. Any application that uses more than its requested share of memory or cpu is automatically terminated - thereby, allowing multi-tenancy. Just like you would for any YARN-based application, you can use YARN's [web UI](/learn/documentation/{{site.version}}/deployment/yarn.html#application-master-ui) to manage your Samza jobs, view their logs etc.
### Running Samza in standalone mode
-Often you want to embed Samza as a component in a larger application. To enable this, Samza supports a standalone mode of operation. In this mode, Samza can be used like a library within your application. This is very similar to Kafka Streams and offers greater control over the application life-cycle. You can increase capacity by spinning up multiple instances. The instances will dynamically coordinate among themselves to distribute work. If any instance fails, the tasks running on it will be re-assigned to the remaining ones. By default, Samza uses Zookeeper for coordination across instances. The coordination logic by itself is pluggable.
+Often you want to embed and integrate Samza as a component within a larger application. To enable this, Samza supports a [standalone mode](learn/documentation/{{site.version}}/deployment/standalone.html) of deployment allowing greater control over your application's life-cycle. In this model, Samza can be used just like any library you import within your Java application. This is identical to using a [high-level Kafka consumer](https://kafka.apache.org/) to process your streams.
-This mode allows you to run Samza with any cluster-manager of your choice - including Kubernetes, Marathon or on any hosting environment. You are free to control memory-limits, multi-tenancy for your application on your own - since Samza now acts as a light-weight library used by your application.
+You can increase your application's capacity by spinning up multiple instances. These instances will then dynamically coordinate with each other and distribute work among themselves. If an instance fails for some reason, the tasks running on it will be re-assigned to the remaining ones. By default, Samza uses [Zookeeper](https://zookeeper.apache.org/) for coordination across individual instances. The coordination logic by itself is pluggable and hence, can integrate with other frameworks.
+
+This mode allows you to bring any cluster-manager or hosting-environment of your choice(eg: [Kubernetes](https://kubernetes.io/), [Marathon](https://mesosphere.github.io/marathon/)) to run your application. You are also free to control memory-limits, multi-tenancy on your own - since Samza is used as a light-weight library.
### Choosing a deployment model
-A common question that we get asked is - "Where should I run my Samza application?". Here are some guidelines when choosing your deployment model. Since your application logic does not change, it is easy to port from one deployment model to the other.
+A common question that we get asked is - "Should I use YARN or standalone?". Here are some guidelines when choosing your deployment model. Since your application logic does not change, it is quite easy to port from one to the other.
* Would you like Samza to be embedded as a component of a larger application?
* If so, then you should use standalone.
@@ -45,4 +47,4 @@ A common question that we get asked is - "Where should I run my Samza applicatio
* If so, then you should use standalone.
* Would you like to run centrally-managed tools and dashboards?
* If so, then you should use YARN.
- * Note: You can still have tools and dashboards when using standalone, but you will need to run them yourself wherever your application is deployed.
+ * Note: You can still have tools and dashboards when using standalone, but you will need to run them yourself wherever your application is deployed.
\ No newline at end of file
[41/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/429277b7
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/429277b7
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/429277b7
Branch: refs/heads/master
Commit: 429277b706a7f7fd3c531e38777099aee9c4530c
Parents: 841b8fa 8723c3f
Author: Jagadish <jv...@linkedin.com>
Authored: Wed Oct 31 14:22:21 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Wed Oct 31 14:22:21 2018 -0700
----------------------------------------------------------------------
RELEASE.md | 47 ++++++--------
checkstyle/checkstyle-suppressions.xml | 13 ++--
checkstyle/checkstyle.xml | 9 ++-
.../samza/application/SamzaApplication.java | 25 ++++----
.../samza/application/StreamApplication.java | 52 ++++++++-------
.../samza/application/TaskApplication.java | 27 ++++----
.../descriptors/ApplicationDescriptor.java | 63 +++++++++++--------
.../StreamApplicationDescriptor.java | 13 ++--
.../descriptors/TaskApplicationDescriptor.java | 22 ++++---
.../context/ApplicationContainerContext.java | 28 +++++----
.../ApplicationContainerContextFactory.java | 12 ++--
.../samza/context/ApplicationTaskContext.java | 19 +++---
.../context/ApplicationTaskContextFactory.java | 12 ++--
.../apache/samza/context/ContainerContext.java | 4 +-
.../java/org/apache/samza/context/Context.java | 10 +--
.../org/apache/samza/context/TaskContext.java | 11 ++--
.../samza/system/ExtendedSystemAdmin.java | 37 -----------
.../org/apache/samza/system/SystemAdmin.java | 12 ++++
.../system/descriptors/StreamExpander.java | 7 ++-
.../TransformingInputDescriptorProvider.java | 10 +--
.../main/java/org/apache/samza/table/Table.java | 36 ++++++-----
.../table/descriptors/TableDescriptor.java | 28 +++++----
.../apache/samza/system/TestSystemAdmin.java | 2 +-
.../system/inmemory/InMemorySystemProducer.java | 22 +++++--
.../samza/system/StreamMetadataCache.scala | 4 +-
.../apache/samza/system/MockSystemFactory.java | 9 +--
.../system/inmemory/TestInMemorySystem.java | 28 +++++++++
.../samza/checkpoint/TestCheckpointTool.scala | 2 +-
.../samza/system/kafka/KafkaSystemAdmin.java | 39 +-----------
.../kafka_deprecated/KafkaSystemAdmin.scala | 66 +-------------------
.../system/kafka/TestKafkaSystemAdmin.scala | 19 +++---
.../kafka_deprecated/TestKafkaSystemAdmin.scala | 41 ------------
.../descriptors/InMemoryInputDescriptor.java | 2 +-
.../descriptors/InMemoryOutputDescriptor.java | 2 +-
34 files changed, 319 insertions(+), 414 deletions(-)
----------------------------------------------------------------------
[08/50] samza git commit: Delete flickity.js since it's on GPL,
a license which is incompatible with Apache
Posted by ja...@apache.org.
Delete flickity.js since it's on GPL, a license which is incompatible with Apache
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/b729f294
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/b729f294
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/b729f294
Branch: refs/heads/master
Commit: b729f294186a5a04d4140b97311450ae340d2db3
Parents: 88a4a7c
Author: Jagadish <jv...@linkedin.com>
Authored: Thu Oct 18 18:03:59 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Thu Oct 18 18:03:59 2018 -0700
----------------------------------------------------------------------
docs/_layouts/case-study.html | 4 +--
docs/_layouts/default.html | 4 +--
docs/css/main.new.css | 71 --------------------------------------
docs/js/flickity.js | 13 -------
4 files changed, 2 insertions(+), 90 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/b729f294/docs/_layouts/case-study.html
----------------------------------------------------------------------
diff --git a/docs/_layouts/case-study.html b/docs/_layouts/case-study.html
index a4274f4..75a1570 100644
--- a/docs/_layouts/case-study.html
+++ b/docs/_layouts/case-study.html
@@ -93,7 +93,7 @@
<div class="case-studies">
- <div class="case-studies-list js-flickity" data-flickity-options='{ "wrapAround": true, "autoPlay": 1500 }'>
+ <div class="case-studies-list">
{% assign case-studies_sorted = site.case-studies | sort: 'priority' %}
{% for study in case-studies_sorted %}
@@ -124,8 +124,6 @@
{% include footer.html %}
- <script type="text/javascript" src="/js/flickity.js"></script>
-
</body>
</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/b729f294/docs/_layouts/default.html
----------------------------------------------------------------------
diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html
index b8cce84..988f536 100644
--- a/docs/_layouts/default.html
+++ b/docs/_layouts/default.html
@@ -196,7 +196,7 @@
Case Studies
</div>
- <div class="section__items js-flickity" data-flickity-options='{ "wrapAround": true, "autoPlay": 1500 }'>
+ <div class="section__items" >
{% assign case-studies_sorted = site.case-studies | sort: 'priority' %}
{% for study in case-studies_sorted %}
@@ -220,8 +220,6 @@
{% include footer.html %}
- <script type="text/javascript" src="/js/flickity.js"></script>
-
</body>
</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/b729f294/docs/css/main.new.css
----------------------------------------------------------------------
diff --git a/docs/css/main.new.css b/docs/css/main.new.css
index 43f2f9f..6773cc6 100644
--- a/docs/css/main.new.css
+++ b/docs/css/main.new.css
@@ -750,72 +750,6 @@ footer .side-by-side > * {
border: none;
}
-.flickity-page-dots {
- display: none;
-}
-
-.js-flickity {
- position: relative;
- display: block;
- margin: auto;
- width: 80%;
- width: calc(100% - 100px);
- margin-top: 40px;
- outline: none;
-}
-
-.js-flickity::after {
- content: '';
- position: absolute;
- top: 0;
- right: 0;
- width: 30%;
- height: 100%;
- background-image: linear-gradient(to right, rgba(255,255,255,0) 0%, transparent 20%, white 100%);
- pointer-events: none;
-}
-
-.js-flickity::before {
- content: '';
- position: absolute;
- top: 0;
- left: 0;
- width: 30%;
- height: 100%;
- background-image: linear-gradient(to left, rgba(255,255,255,0) 0%, transparent 20%, white 100%);
- pointer-events: none;
- z-index: 1;
-}
-
-.flickity-viewport {
- overflow: hidden;
-}
-
-.flickity-button {
- background: transparent;
- position: absolute;
- width: 50px;
- height: 50px;
- top: 50%;
- border: none;
- color: #333;
- transform: translateY(-50%);
- transition: fill .3s ease 0s;
- cursor: pointer;
-}
-
-.flickity-button:hover {
- fill: #eb1c23;
-}
-
-.flickity-button.previous {
- left: -50px;
-}
-
-.flickity-button.next {
- right: -50px;
-}
-
/********************************************************************
*
* WEIRD MESH NODE CSS THING
@@ -900,7 +834,6 @@ footer .side-by-side > * {
}
-
.page .section {
box-sizing: border-box;
padding: 20px;
@@ -1539,10 +1472,6 @@ ul.case-studies {
justify-content: center;
}
-.case-studies-list.js-flickity {
- display: block;
-}
-
@media only screen and (min-width: 768px) {
.case-studies-list {
justify-content: start;
http://git-wip-us.apache.org/repos/asf/samza/blob/b729f294/docs/js/flickity.js
----------------------------------------------------------------------
diff --git a/docs/js/flickity.js b/docs/js/flickity.js
deleted file mode 100644
index e9dac24..0000000
--- a/docs/js/flickity.js
+++ /dev/null
@@ -1,13 +0,0 @@
-/*!
- * Flickity PACKAGED v2.1.2
- * Touch, responsive, flickable carousels
- *
- * Licensed GPLv3 for open source use
- * or Flickity Commercial License for commercial use
- *
- * https://flickity.metafizzy.co
- * Copyright 2015-2018 Metafizzy
- */
-
-!function(t,e){"function"==typeof define&&define.amd?define("jquery-bridget/jquery-bridget",["jquery"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?module.exports=e(t,require("jquery")):t.jQueryBridget=e(t,t.jQuery)}(window,function(t,e){"use strict";function i(i,o,a){function l(t,e,n){var s,o="$()."+i+'("'+e+'")';return t.each(function(t,l){var h=a.data(l,i);if(!h)return void r(i+" not initialized. Cannot call methods, i.e. "+o);var c=h[e];if(!c||"_"==e.charAt(0))return void r(o+" is not a valid method");var d=c.apply(h,n);s=void 0===s?d:s}),void 0!==s?s:t}function h(t,e){t.each(function(t,n){var s=a.data(n,i);s?(s.option(e),s._init()):(s=new o(n,e),a.data(n,i,s))})}a=a||e||t.jQuery,a&&(o.prototype.option||(o.prototype.option=function(t){a.isPlainObject(t)&&(this.options=a.extend(!0,this.options,t))}),a.fn[i]=function(t){if("string"==typeof t){var e=s.call(arguments,1);return l(this,t,e)}return h(this,t),this},n(a))}function n(t){!t||t&&t.bridget||(t.bridget=
i)}var s=Array.prototype.slice,o=t.console,r="undefined"==typeof o?function(){}:function(t){o.error(t)};return n(e||t.jQuery),i}),function(t,e){"function"==typeof define&&define.amd?define("ev-emitter/ev-emitter",e):"object"==typeof module&&module.exports?module.exports=e():t.EvEmitter=e()}("undefined"!=typeof window?window:this,function(){function t(){}var e=t.prototype;return e.on=function(t,e){if(t&&e){var i=this._events=this._events||{},n=i[t]=i[t]||[];return n.indexOf(e)==-1&&n.push(e),this}},e.once=function(t,e){if(t&&e){this.on(t,e);var i=this._onceEvents=this._onceEvents||{},n=i[t]=i[t]||{};return n[e]=!0,this}},e.off=function(t,e){var i=this._events&&this._events[t];if(i&&i.length){var n=i.indexOf(e);return n!=-1&&i.splice(n,1),this}},e.emitEvent=function(t,e){var i=this._events&&this._events[t];if(i&&i.length){i=i.slice(0),e=e||[];for(var n=this._onceEvents&&this._onceEvents[t],s=0;s<i.length;s++){var o=i[s],r=n&&n[o];r&&(this.off(t,o),delete n[o]),o.apply(this,e)}return t
his}},e.allOff=function(){delete this._events,delete this._onceEvents},t}),function(t,e){"function"==typeof define&&define.amd?define("get-size/get-size",e):"object"==typeof module&&module.exports?module.exports=e():t.getSize=e()}(window,function(){"use strict";function t(t){var e=parseFloat(t),i=t.indexOf("%")==-1&&!isNaN(e);return i&&e}function e(){}function i(){for(var t={width:0,height:0,innerWidth:0,innerHeight:0,outerWidth:0,outerHeight:0},e=0;e<h;e++){var i=l[e];t[i]=0}return t}function n(t){var e=getComputedStyle(t);return e||a("Style returned "+e+". Are you running this code in a hidden iframe on Firefox? See https://bit.ly/getsizebug1"),e}function s(){if(!c){c=!0;var e=document.createElement("div");e.style.width="200px",e.style.padding="1px 2px 3px 4px",e.style.borderStyle="solid",e.style.borderWidth="1px 2px 3px 4px",e.style.boxSizing="border-box";var i=document.body||document.documentElement;i.appendChild(e);var s=n(e);r=200==Math.round(t(s.width)),o.isBoxSizeOuter=r,i.r
emoveChild(e)}}function o(e){if(s(),"string"==typeof e&&(e=document.querySelector(e)),e&&"object"==typeof e&&e.nodeType){var o=n(e);if("none"==o.display)return i();var a={};a.width=e.offsetWidth,a.height=e.offsetHeight;for(var c=a.isBorderBox="border-box"==o.boxSizing,d=0;d<h;d++){var u=l[d],f=o[u],p=parseFloat(f);a[u]=isNaN(p)?0:p}var g=a.paddingLeft+a.paddingRight,v=a.paddingTop+a.paddingBottom,m=a.marginLeft+a.marginRight,y=a.marginTop+a.marginBottom,b=a.borderLeftWidth+a.borderRightWidth,E=a.borderTopWidth+a.borderBottomWidth,S=c&&r,C=t(o.width);C!==!1&&(a.width=C+(S?0:g+b));var x=t(o.height);return x!==!1&&(a.height=x+(S?0:v+E)),a.innerWidth=a.width-(g+b),a.innerHeight=a.height-(v+E),a.outerWidth=a.width+m,a.outerHeight=a.height+y,a}}var r,a="undefined"==typeof console?e:function(t){console.error(t)},l=["paddingLeft","paddingRight","paddingTop","paddingBottom","marginLeft","marginRight","marginTop","marginBottom","borderLeftWidth","borderRightWidth","borderTopWidth","borderBott
omWidth"],h=l.length,c=!1;return o}),function(t,e){"use strict";"function"==typeof define&&define.amd?define("desandro-matches-selector/matches-selector",e):"object"==typeof module&&module.exports?module.exports=e():t.matchesSelector=e()}(window,function(){"use strict";var t=function(){var t=window.Element.prototype;if(t.matches)return"matches";if(t.matchesSelector)return"matchesSelector";for(var e=["webkit","moz","ms","o"],i=0;i<e.length;i++){var n=e[i],s=n+"MatchesSelector";if(t[s])return s}}();return function(e,i){return e[t](i)}}),function(t,e){"function"==typeof define&&define.amd?define("fizzy-ui-utils/utils",["desandro-matches-selector/matches-selector"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?module.exports=e(t,require("desandro-matches-selector")):t.fizzyUIUtils=e(t,t.matchesSelector)}(window,function(t,e){var i={};i.extend=function(t,e){for(var i in e)t[i]=e[i];return t},i.modulo=function(t,e){return(t%e+e)%e};var n=Array.prototype.slice;i.makeA
rray=function(t){if(Array.isArray(t))return t;if(null===t||void 0===t)return[];var e="object"==typeof t&&"number"==typeof t.length;return e?n.call(t):[t]},i.removeFrom=function(t,e){var i=t.indexOf(e);i!=-1&&t.splice(i,1)},i.getParent=function(t,i){for(;t.parentNode&&t!=document.body;)if(t=t.parentNode,e(t,i))return t},i.getQueryElement=function(t){return"string"==typeof t?document.querySelector(t):t},i.handleEvent=function(t){var e="on"+t.type;this[e]&&this[e](t)},i.filterFindElements=function(t,n){t=i.makeArray(t);var s=[];return t.forEach(function(t){if(t instanceof HTMLElement){if(!n)return void s.push(t);e(t,n)&&s.push(t);for(var i=t.querySelectorAll(n),o=0;o<i.length;o++)s.push(i[o])}}),s},i.debounceMethod=function(t,e,i){i=i||100;var n=t.prototype[e],s=e+"Timeout";t.prototype[e]=function(){var t=this[s];clearTimeout(t);var e=arguments,o=this;this[s]=setTimeout(function(){n.apply(o,e),delete o[s]},i)}},i.docReady=function(t){var e=document.readyState;"complete"==e||"interactiv
e"==e?setTimeout(t):document.addEventListener("DOMContentLoaded",t)},i.toDashed=function(t){return t.replace(/(.)([A-Z])/g,function(t,e,i){return e+"-"+i}).toLowerCase()};var s=t.console;return i.htmlInit=function(e,n){i.docReady(function(){var o=i.toDashed(n),r="data-"+o,a=document.querySelectorAll("["+r+"]"),l=document.querySelectorAll(".js-"+o),h=i.makeArray(a).concat(i.makeArray(l)),c=r+"-options",d=t.jQuery;h.forEach(function(t){var i,o=t.getAttribute(r)||t.getAttribute(c);try{i=o&&JSON.parse(o)}catch(a){return void(s&&s.error("Error parsing "+r+" on "+t.className+": "+a))}var l=new e(t,i);d&&d.data(t,n,l)})})},i}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/cell",["get-size/get-size"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?module.exports=e(t,require("get-size")):(t.Flickity=t.Flickity||{},t.Flickity.Cell=e(t,t.getSize))}(window,function(t,e){function i(t,e){this.element=t,this.parent=e,this.create()}var n=i.prototype;ret
urn n.create=function(){this.element.style.position="absolute",this.element.setAttribute("aria-selected","false"),this.x=0,this.shift=0},n.destroy=function(){this.element.style.position="";var t=this.parent.originSide;this.element.removeAttribute("aria-selected"),this.element.style[t]=""},n.getSize=function(){this.size=e(this.element)},n.setPosition=function(t){this.x=t,this.updateTarget(),this.renderPosition(t)},n.updateTarget=n.setDefaultTarget=function(){var t="left"==this.parent.originSide?"marginLeft":"marginRight";this.target=this.x+this.size[t]+this.size.width*this.parent.cellAlign},n.renderPosition=function(t){var e=this.parent.originSide;this.element.style[e]=this.parent.getPositionValue(t)},n.wrapShift=function(t){this.shift=t,this.renderPosition(this.x+this.parent.slideableWidth*t)},n.remove=function(){this.element.parentNode.removeChild(this.element)},i}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/slide",e):"object"==typeof module&&module.exp
orts?module.exports=e():(t.Flickity=t.Flickity||{},t.Flickity.Slide=e())}(window,function(){"use strict";function t(t){this.parent=t,this.isOriginLeft="left"==t.originSide,this.cells=[],this.outerWidth=0,this.height=0}var e=t.prototype;return e.addCell=function(t){if(this.cells.push(t),this.outerWidth+=t.size.outerWidth,this.height=Math.max(t.size.outerHeight,this.height),1==this.cells.length){this.x=t.x;var e=this.isOriginLeft?"marginLeft":"marginRight";this.firstMargin=t.size[e]}},e.updateTarget=function(){var t=this.isOriginLeft?"marginRight":"marginLeft",e=this.getLastCell(),i=e?e.size[t]:0,n=this.outerWidth-(this.firstMargin+i);this.target=this.x+this.firstMargin+n*this.parent.cellAlign},e.getLastCell=function(){return this.cells[this.cells.length-1]},e.select=function(){this.changeSelected(!0)},e.unselect=function(){this.changeSelected(!1)},e.changeSelected=function(t){var e=t?"add":"remove";this.cells.forEach(function(i){i.element.classList[e]("is-selected"),i.element.setAttr
ibute("aria-selected",t.toString())})},e.getCellElements=function(){return this.cells.map(function(t){return t.element})},t}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/animate",["fizzy-ui-utils/utils"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?module.exports=e(t,require("fizzy-ui-utils")):(t.Flickity=t.Flickity||{},t.Flickity.animatePrototype=e(t,t.fizzyUIUtils))}(window,function(t,e){var i={};return i.startAnimation=function(){this.isAnimating||(this.isAnimating=!0,this.restingFrames=0,this.animate())},i.animate=function(){this.applyDragForce(),this.applySelectedAttraction();var t=this.x;if(this.integratePhysics(),this.positionSlider(),this.settle(t),this.isAnimating){var e=this;requestAnimationFrame(function(){e.animate()})}},i.positionSlider=function(){var t=this.x;this.options.wrapAround&&this.cells.length>1&&(t=e.modulo(t,this.slideableWidth),t-=this.slideableWidth,this.shiftWrapCells(t)),t+=this.cursorPosition,t=this.opti
ons.rightToLeft?-t:t;var i=this.getPositionValue(t);this.slider.style.transform=this.isAnimating?"translate3d("+i+",0,0)":"translateX("+i+")";var n=this.slides[0];if(n){var s=-this.x-n.target,o=s/this.slidesWidth;this.dispatchEvent("scroll",null,[o,s])}},i.positionSliderAtSelected=function(){this.cells.length&&(this.x=-this.selectedSlide.target,this.velocity=0,this.positionSlider())},i.getPositionValue=function(t){return this.options.percentPosition?.01*Math.round(t/this.size.innerWidth*1e4)+"%":Math.round(t)+"px"},i.settle=function(t){this.isPointerDown||Math.round(100*this.x)!=Math.round(100*t)||this.restingFrames++,this.restingFrames>2&&(this.isAnimating=!1,delete this.isFreeScrolling,this.positionSlider(),this.dispatchEvent("settle",null,[this.selectedIndex]))},i.shiftWrapCells=function(t){var e=this.cursorPosition+t;this._shiftCells(this.beforeShiftCells,e,-1);var i=this.size.innerWidth-(t+this.slideableWidth+this.cursorPosition);this._shiftCells(this.afterShiftCells,i,1)},i._s
hiftCells=function(t,e,i){for(var n=0;n<t.length;n++){var s=t[n],o=e>0?i:0;s.wrapShift(o),e-=s.size.outerWidth}},i._unshiftCells=function(t){if(t&&t.length)for(var e=0;e<t.length;e++)t[e].wrapShift(0)},i.integratePhysics=function(){this.x+=this.velocity,this.velocity*=this.getFrictionFactor()},i.applyForce=function(t){this.velocity+=t},i.getFrictionFactor=function(){return 1-this.options[this.isFreeScrolling?"freeScrollFriction":"friction"]},i.getRestingPosition=function(){return this.x+this.velocity/(1-this.getFrictionFactor())},i.applyDragForce=function(){if(this.isDraggable&&this.isPointerDown){var t=this.dragX-this.x,e=t-this.velocity;this.applyForce(e)}},i.applySelectedAttraction=function(){var t=this.isDraggable&&this.isPointerDown;if(!t&&!this.isFreeScrolling&&this.slides.length){var e=this.selectedSlide.target*-1-this.x,i=e*this.options.selectedAttraction;this.applyForce(i)}},i}),function(t,e){if("function"==typeof define&&define.amd)define("flickity/js/flickity",["ev-emitte
r/ev-emitter","get-size/get-size","fizzy-ui-utils/utils","./cell","./slide","./animate"],function(i,n,s,o,r,a){return e(t,i,n,s,o,r,a)});else if("object"==typeof module&&module.exports)module.exports=e(t,require("ev-emitter"),require("get-size"),require("fizzy-ui-utils"),require("./cell"),require("./slide"),require("./animate"));else{var i=t.Flickity;t.Flickity=e(t,t.EvEmitter,t.getSize,t.fizzyUIUtils,i.Cell,i.Slide,i.animatePrototype)}}(window,function(t,e,i,n,s,o,r){function a(t,e){for(t=n.makeArray(t);t.length;)e.appendChild(t.shift())}function l(t,e){var i=n.getQueryElement(t);if(!i)return void(d&&d.error("Bad element for Flickity: "+(i||t)));if(this.element=i,this.element.flickityGUID){var s=f[this.element.flickityGUID];return s.option(e),s}h&&(this.$element=h(this.element)),this.options=n.extend({},this.constructor.defaults),this.option(e),this._create()}var h=t.jQuery,c=t.getComputedStyle,d=t.console,u=0,f={};l.defaults={accessibility:!0,cellAlign:"center",freeScrollFriction:
.075,friction:.28,namespaceJQueryEvents:!0,percentPosition:!0,resize:!0,selectedAttraction:.025,setGallerySize:!0},l.createMethods=[];var p=l.prototype;n.extend(p,e.prototype),p._create=function(){var e=this.guid=++u;this.element.flickityGUID=e,f[e]=this,this.selectedIndex=0,this.restingFrames=0,this.x=0,this.velocity=0,this.originSide=this.options.rightToLeft?"right":"left",this.viewport=document.createElement("div"),this.viewport.className="flickity-viewport",this._createSlider(),(this.options.resize||this.options.watchCSS)&&t.addEventListener("resize",this);for(var i in this.options.on){var n=this.options.on[i];this.on(i,n)}l.createMethods.forEach(function(t){this[t]()},this),this.options.watchCSS?this.watchCSS():this.activate()},p.option=function(t){n.extend(this.options,t)},p.activate=function(){if(!this.isActive){this.isActive=!0,this.element.classList.add("flickity-enabled"),this.options.rightToLeft&&this.element.classList.add("flickity-rtl"),this.getSize();var t=this._filter
FindCellElements(this.element.children);a(t,this.slider),this.viewport.appendChild(this.slider),this.element.appendChild(this.viewport),this.reloadCells(),this.options.accessibility&&(this.element.tabIndex=0,this.element.addEventListener("keydown",this)),this.emitEvent("activate");var e,i=this.options.initialIndex;e=this.isInitActivated?this.selectedIndex:void 0!==i&&this.cells[i]?i:0,this.select(e,!1,!0),this.isInitActivated=!0,this.dispatchEvent("ready")}},p._createSlider=function(){var t=document.createElement("div");t.className="flickity-slider",t.style[this.originSide]=0,this.slider=t},p._filterFindCellElements=function(t){return n.filterFindElements(t,this.options.cellSelector)},p.reloadCells=function(){this.cells=this._makeCells(this.slider.children),this.positionCells(),this._getWrapShiftCells(),this.setGallerySize()},p._makeCells=function(t){var e=this._filterFindCellElements(t),i=e.map(function(t){return new s(t,this)},this);return i},p.getLastCell=function(){return this.c
ells[this.cells.length-1]},p.getLastSlide=function(){return this.slides[this.slides.length-1]},p.positionCells=function(){this._sizeCells(this.cells),this._positionCells(0)},p._positionCells=function(t){t=t||0,this.maxCellHeight=t?this.maxCellHeight||0:0;var e=0;if(t>0){var i=this.cells[t-1];e=i.x+i.size.outerWidth}for(var n=this.cells.length,s=t;s<n;s++){var o=this.cells[s];o.setPosition(e),e+=o.size.outerWidth,this.maxCellHeight=Math.max(o.size.outerHeight,this.maxCellHeight)}this.slideableWidth=e,this.updateSlides(),this._containSlides(),this.slidesWidth=n?this.getLastSlide().target-this.slides[0].target:0},p._sizeCells=function(t){t.forEach(function(t){t.getSize()})},p.updateSlides=function(){if(this.slides=[],this.cells.length){var t=new o(this);this.slides.push(t);var e="left"==this.originSide,i=e?"marginRight":"marginLeft",n=this._getCanCellFit();this.cells.forEach(function(e,s){if(!t.cells.length)return void t.addCell(e);var r=t.outerWidth-t.firstMargin+(e.size.outerWidth-e.
size[i]);n.call(this,s,r)?t.addCell(e):(t.updateTarget(),t=new o(this),this.slides.push(t),t.addCell(e))},this),t.updateTarget(),this.updateSelectedSlide()}},p._getCanCellFit=function(){var t=this.options.groupCells;if(!t)return function(){return!1};if("number"==typeof t){var e=parseInt(t,10);return function(t){return t%e!==0}}var i="string"==typeof t&&t.match(/^(\d+)%$/),n=i?parseInt(i[1],10)/100:1;return function(t,e){return e<=(this.size.innerWidth+1)*n}},p._init=p.reposition=function(){this.positionCells(),this.positionSliderAtSelected()},p.getSize=function(){this.size=i(this.element),this.setCellAlign(),this.cursorPosition=this.size.innerWidth*this.cellAlign};var g={center:{left:.5,right:.5},left:{left:0,right:1},right:{right:0,left:1}};return p.setCellAlign=function(){var t=g[this.options.cellAlign];this.cellAlign=t?t[this.originSide]:this.options.cellAlign},p.setGallerySize=function(){if(this.options.setGallerySize){var t=this.options.adaptiveHeight&&this.selectedSlide?this.s
electedSlide.height:this.maxCellHeight;this.viewport.style.height=t+"px"}},p._getWrapShiftCells=function(){if(this.options.wrapAround){this._unshiftCells(this.beforeShiftCells),this._unshiftCells(this.afterShiftCells);var t=this.cursorPosition,e=this.cells.length-1;this.beforeShiftCells=this._getGapCells(t,e,-1),t=this.size.innerWidth-this.cursorPosition,this.afterShiftCells=this._getGapCells(t,0,1)}},p._getGapCells=function(t,e,i){for(var n=[];t>0;){var s=this.cells[e];if(!s)break;n.push(s),e+=i,t-=s.size.outerWidth}return n},p._containSlides=function(){if(this.options.contain&&!this.options.wrapAround&&this.cells.length){var t=this.options.rightToLeft,e=t?"marginRight":"marginLeft",i=t?"marginLeft":"marginRight",n=this.slideableWidth-this.getLastCell().size[i],s=n<this.size.innerWidth,o=this.cursorPosition+this.cells[0].size[e],r=n-this.size.innerWidth*(1-this.cellAlign);this.slides.forEach(function(t){s?t.target=n*this.cellAlign:(t.target=Math.max(t.target,o),t.target=Math.min(t.
target,r))},this)}},p.dispatchEvent=function(t,e,i){var n=e?[e].concat(i):i;if(this.emitEvent(t,n),h&&this.$element){t+=this.options.namespaceJQueryEvents?".flickity":"";var s=t;if(e){var o=h.Event(e);o.type=t,s=o}this.$element.trigger(s,i)}},p.select=function(t,e,i){if(this.isActive&&(t=parseInt(t,10),this._wrapSelect(t),(this.options.wrapAround||e)&&(t=n.modulo(t,this.slides.length)),this.slides[t])){var s=this.selectedIndex;this.selectedIndex=t,this.updateSelectedSlide(),i?this.positionSliderAtSelected():this.startAnimation(),this.options.adaptiveHeight&&this.setGallerySize(),this.dispatchEvent("select",null,[t]),t!=s&&this.dispatchEvent("change",null,[t]),this.dispatchEvent("cellSelect")}},p._wrapSelect=function(t){var e=this.slides.length,i=this.options.wrapAround&&e>1;if(!i)return t;var s=n.modulo(t,e),o=Math.abs(s-this.selectedIndex),r=Math.abs(s+e-this.selectedIndex),a=Math.abs(s-e-this.selectedIndex);!this.isDragSelect&&r<o?t+=e:!this.isDragSelect&&a<o&&(t-=e),t<0?this.x-=t
his.slideableWidth:t>=e&&(this.x+=this.slideableWidth)},p.previous=function(t,e){this.select(this.selectedIndex-1,t,e)},p.next=function(t,e){this.select(this.selectedIndex+1,t,e)},p.updateSelectedSlide=function(){var t=this.slides[this.selectedIndex];t&&(this.unselectSelectedSlide(),this.selectedSlide=t,t.select(),this.selectedCells=t.cells,this.selectedElements=t.getCellElements(),this.selectedCell=t.cells[0],this.selectedElement=this.selectedElements[0])},p.unselectSelectedSlide=function(){this.selectedSlide&&this.selectedSlide.unselect()},p.selectCell=function(t,e,i){var n=this.queryCell(t);if(n){var s=this.getCellSlideIndex(n);this.select(s,e,i)}},p.getCellSlideIndex=function(t){for(var e=0;e<this.slides.length;e++){var i=this.slides[e],n=i.cells.indexOf(t);if(n!=-1)return e}},p.getCell=function(t){for(var e=0;e<this.cells.length;e++){var i=this.cells[e];if(i.element==t)return i}},p.getCells=function(t){t=n.makeArray(t);var e=[];return t.forEach(function(t){var i=this.getCell(t)
;i&&e.push(i)},this),e},p.getCellElements=function(){return this.cells.map(function(t){return t.element})},p.getParentCell=function(t){var e=this.getCell(t);return e?e:(t=n.getParent(t,".flickity-slider > *"),this.getCell(t))},p.getAdjacentCellElements=function(t,e){if(!t)return this.selectedSlide.getCellElements();e=void 0===e?this.selectedIndex:e;var i=this.slides.length;if(1+2*t>=i)return this.getCellElements();for(var s=[],o=e-t;o<=e+t;o++){var r=this.options.wrapAround?n.modulo(o,i):o,a=this.slides[r];a&&(s=s.concat(a.getCellElements()))}return s},p.queryCell=function(t){return"number"==typeof t?this.cells[t]:("string"==typeof t&&(t=this.element.querySelector(t)),this.getCell(t))},p.uiChange=function(){this.emitEvent("uiChange")},p.childUIPointerDown=function(t){this.emitEvent("childUIPointerDown",[t])},p.onresize=function(){this.watchCSS(),this.resize()},n.debounceMethod(l,"onresize",150),p.resize=function(){if(this.isActive){this.getSize(),this.options.wrapAround&&(this.x=n.m
odulo(this.x,this.slideableWidth)),this.positionCells(),this._getWrapShiftCells(),this.setGallerySize(),this.emitEvent("resize");var t=this.selectedElements&&this.selectedElements[0];this.selectCell(t,!1,!0)}},p.watchCSS=function(){var t=this.options.watchCSS;if(t){var e=c(this.element,":after").content;e.indexOf("flickity")!=-1?this.activate():this.deactivate()}},p.onkeydown=function(t){var e=document.activeElement&&document.activeElement!=this.element;if(this.options.accessibility&&!e){var i=l.keyboardHandlers[t.keyCode];i&&i.call(this)}},l.keyboardHandlers={37:function(){var t=this.options.rightToLeft?"next":"previous";this.uiChange(),this[t]()},39:function(){var t=this.options.rightToLeft?"previous":"next";this.uiChange(),this[t]()}},p.focus=function(){var e=t.pageYOffset;this.element.focus({preventScroll:!0}),t.pageYOffset!=e&&t.scrollTo(t.pageXOffset,e)},p.deactivate=function(){this.isActive&&(this.element.classList.remove("flickity-enabled"),this.element.classList.remove("fli
ckity-rtl"),this.unselectSelectedSlide(),this.cells.forEach(function(t){t.destroy()}),this.element.removeChild(this.viewport),a(this.slider.children,this.element),this.options.accessibility&&(this.element.removeAttribute("tabIndex"),this.element.removeEventListener("keydown",this)),this.isActive=!1,this.emitEvent("deactivate"))},p.destroy=function(){this.deactivate(),t.removeEventListener("resize",this),this.emitEvent("destroy"),h&&this.$element&&h.removeData(this.element,"flickity"),delete this.element.flickityGUID,delete f[this.guid]},n.extend(p,r),l.data=function(t){t=n.getQueryElement(t);var e=t&&t.flickityGUID;return e&&f[e]},n.htmlInit(l,"flickity"),h&&h.bridget&&h.bridget("flickity",l),l.setJQuery=function(t){h=t},l.Cell=s,l}),function(t,e){"function"==typeof define&&define.amd?define("unipointer/unipointer",["ev-emitter/ev-emitter"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?module.exports=e(t,require("ev-emitter")):t.Unipointer=e(t,t.EvEmitter)}(win
dow,function(t,e){function i(){}function n(){}var s=n.prototype=Object.create(e.prototype);s.bindStartEvent=function(t){this._bindStartEvent(t,!0)},s.unbindStartEvent=function(t){this._bindStartEvent(t,!1)},s._bindStartEvent=function(e,i){i=void 0===i||i;var n=i?"addEventListener":"removeEventListener",s="mousedown";t.PointerEvent?s="pointerdown":"ontouchstart"in t&&(s="touchstart"),e[n](s,this)},s.handleEvent=function(t){var e="on"+t.type;this[e]&&this[e](t)},s.getTouch=function(t){for(var e=0;e<t.length;e++){var i=t[e];if(i.identifier==this.pointerIdentifier)return i}},s.onmousedown=function(t){var e=t.button;e&&0!==e&&1!==e||this._pointerDown(t,t)},s.ontouchstart=function(t){this._pointerDown(t,t.changedTouches[0])},s.onpointerdown=function(t){this._pointerDown(t,t)},s._pointerDown=function(t,e){t.button||this.isPointerDown||(this.isPointerDown=!0,this.pointerIdentifier=void 0!==e.pointerId?e.pointerId:e.identifier,this.pointerDown(t,e))},s.pointerDown=function(t,e){this._bindPos
tStartEvents(t),this.emitEvent("pointerDown",[t,e])};var o={mousedown:["mousemove","mouseup"],touchstart:["touchmove","touchend","touchcancel"],pointerdown:["pointermove","pointerup","pointercancel"]};return s._bindPostStartEvents=function(e){if(e){var i=o[e.type];i.forEach(function(e){t.addEventListener(e,this)},this),this._boundPointerEvents=i}},s._unbindPostStartEvents=function(){this._boundPointerEvents&&(this._boundPointerEvents.forEach(function(e){t.removeEventListener(e,this)},this),delete this._boundPointerEvents)},s.onmousemove=function(t){this._pointerMove(t,t)},s.onpointermove=function(t){t.pointerId==this.pointerIdentifier&&this._pointerMove(t,t)},s.ontouchmove=function(t){var e=this.getTouch(t.changedTouches);e&&this._pointerMove(t,e)},s._pointerMove=function(t,e){this.pointerMove(t,e)},s.pointerMove=function(t,e){this.emitEvent("pointerMove",[t,e])},s.onmouseup=function(t){this._pointerUp(t,t)},s.onpointerup=function(t){t.pointerId==this.pointerIdentifier&&this._pointe
rUp(t,t)},s.ontouchend=function(t){var e=this.getTouch(t.changedTouches);e&&this._pointerUp(t,e)},s._pointerUp=function(t,e){this._pointerDone(),this.pointerUp(t,e)},s.pointerUp=function(t,e){this.emitEvent("pointerUp",[t,e])},s._pointerDone=function(){this._pointerReset(),this._unbindPostStartEvents(),this.pointerDone()},s._pointerReset=function(){this.isPointerDown=!1,delete this.pointerIdentifier},s.pointerDone=i,s.onpointercancel=function(t){t.pointerId==this.pointerIdentifier&&this._pointerCancel(t,t)},s.ontouchcancel=function(t){var e=this.getTouch(t.changedTouches);e&&this._pointerCancel(t,e)},s._pointerCancel=function(t,e){this._pointerDone(),this.pointerCancel(t,e)},s.pointerCancel=function(t,e){this.emitEvent("pointerCancel",[t,e])},n.getPointerPoint=function(t){return{x:t.pageX,y:t.pageY}},n}),function(t,e){"function"==typeof define&&define.amd?define("unidragger/unidragger",["unipointer/unipointer"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?modu
le.exports=e(t,require("unipointer")):t.Unidragger=e(t,t.Unipointer)}(window,function(t,e){function i(){}var n=i.prototype=Object.create(e.prototype);n.bindHandles=function(){this._bindHandles(!0)},n.unbindHandles=function(){this._bindHandles(!1)},n._bindHandles=function(e){e=void 0===e||e;for(var i=e?"addEventListener":"removeEventListener",n=e?this._touchActionValue:"",s=0;s<this.handles.length;s++){var o=this.handles[s];this._bindStartEvent(o,e),o[i]("click",this),t.PointerEvent&&(o.style.touchAction=n)}},n._touchActionValue="none",n.pointerDown=function(t,e){var i=this.okayPointerDown(t);i&&(this.pointerDownPointer=e,t.preventDefault(),this.pointerDownBlur(),this._bindPostStartEvents(t),this.emitEvent("pointerDown",[t,e]))};var s={TEXTAREA:!0,INPUT:!0,SELECT:!0,OPTION:!0},o={radio:!0,checkbox:!0,button:!0,submit:!0,image:!0,file:!0};return n.okayPointerDown=function(t){var e=s[t.target.nodeName],i=o[t.target.type],n=!e||i;return n||this._pointerReset(),n},n.pointerDownBlur=funct
ion(){var t=document.activeElement,e=t&&t.blur&&t!=document.body;e&&t.blur()},n.pointerMove=function(t,e){var i=this._dragPointerMove(t,e);this.emitEvent("pointerMove",[t,e,i]),this._dragMove(t,e,i)},n._dragPointerMove=function(t,e){var i={x:e.pageX-this.pointerDownPointer.pageX,y:e.pageY-this.pointerDownPointer.pageY};return!this.isDragging&&this.hasDragStarted(i)&&this._dragStart(t,e),i},n.hasDragStarted=function(t){return Math.abs(t.x)>3||Math.abs(t.y)>3},n.pointerUp=function(t,e){this.emitEvent("pointerUp",[t,e]),this._dragPointerUp(t,e)},n._dragPointerUp=function(t,e){this.isDragging?this._dragEnd(t,e):this._staticClick(t,e)},n._dragStart=function(t,e){this.isDragging=!0,this.isPreventingClicks=!0,this.dragStart(t,e)},n.dragStart=function(t,e){this.emitEvent("dragStart",[t,e])},n._dragMove=function(t,e,i){this.isDragging&&this.dragMove(t,e,i)},n.dragMove=function(t,e,i){t.preventDefault(),this.emitEvent("dragMove",[t,e,i])},n._dragEnd=function(t,e){this.isDragging=!1,setTimeout
(function(){delete this.isPreventingClicks}.bind(this)),this.dragEnd(t,e)},n.dragEnd=function(t,e){this.emitEvent("dragEnd",[t,e])},n.onclick=function(t){this.isPreventingClicks&&t.preventDefault()},n._staticClick=function(t,e){this.isIgnoringMouseUp&&"mouseup"==t.type||(this.staticClick(t,e),"mouseup"!=t.type&&(this.isIgnoringMouseUp=!0,setTimeout(function(){delete this.isIgnoringMouseUp}.bind(this),400)))},n.staticClick=function(t,e){this.emitEvent("staticClick",[t,e])},i.getPointerPoint=e.getPointerPoint,i}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/drag",["./flickity","unidragger/unidragger","fizzy-ui-utils/utils"],function(i,n,s){return e(t,i,n,s)}):"object"==typeof module&&module.exports?module.exports=e(t,require("./flickity"),require("unidragger"),require("fizzy-ui-utils")):t.Flickity=e(t,t.Flickity,t.Unidragger,t.fizzyUIUtils)}(window,function(t,e,i,n){function s(){return{x:t.pageXOffset,y:t.pageYOffset}}n.extend(e.defaults,{draggable:">1",drag
Threshold:3}),e.createMethods.push("_createDrag");var o=e.prototype;n.extend(o,i.prototype),o._touchActionValue="pan-y";var r="createTouch"in document,a=!1;o._createDrag=function(){this.on("activate",this.onActivateDrag),this.on("uiChange",this._uiChangeDrag),this.on("childUIPointerDown",this._childUIPointerDownDrag),this.on("deactivate",this.onDeactivateDrag),this.on("cellChange",this.updateDraggable),r&&!a&&(t.addEventListener("touchmove",function(){}),a=!0)},o.onActivateDrag=function(){this.handles=[this.viewport],this.bindHandles(),this.updateDraggable()},o.onDeactivateDrag=function(){this.unbindHandles(),this.element.classList.remove("is-draggable")},o.updateDraggable=function(){">1"==this.options.draggable?this.isDraggable=this.slides.length>1:this.isDraggable=this.options.draggable,this.isDraggable?this.element.classList.add("is-draggable"):this.element.classList.remove("is-draggable")},o.bindDrag=function(){this.options.draggable=!0,this.updateDraggable()},o.unbindDrag=funct
ion(){this.options.draggable=!1,this.updateDraggable()},o._uiChangeDrag=function(){delete this.isFreeScrolling},o._childUIPointerDownDrag=function(t){t.preventDefault(),this.pointerDownFocus(t)},o.pointerDown=function(e,i){if(!this.isDraggable)return void this._pointerDownDefault(e,i);var n=this.okayPointerDown(e);n&&(this._pointerDownPreventDefault(e),this.pointerDownFocus(e),document.activeElement!=this.element&&this.pointerDownBlur(),this.dragX=this.x,this.viewport.classList.add("is-pointer-down"),this.pointerDownScroll=s(),t.addEventListener("scroll",this),this._pointerDownDefault(e,i))},o._pointerDownDefault=function(t,e){this.pointerDownPointer=e,this._bindPostStartEvents(t),this.dispatchEvent("pointerDown",t,[e])};var l={INPUT:!0,TEXTAREA:!0,SELECT:!0};return o.pointerDownFocus=function(t){var e=l[t.target.nodeName];e||this.focus()},o._pointerDownPreventDefault=function(t){var e="touchstart"==t.type,i="touch"==t.pointerType,n=l[t.target.nodeName];e||i||n||t.preventDefault()},
o.hasDragStarted=function(t){return Math.abs(t.x)>this.options.dragThreshold},o.pointerUp=function(t,e){delete this.isTouchScrolling,this.viewport.classList.remove("is-pointer-down"),this.dispatchEvent("pointerUp",t,[e]),this._dragPointerUp(t,e)},o.pointerDone=function(){t.removeEventListener("scroll",this),delete this.pointerDownScroll},o.dragStart=function(e,i){this.isDraggable&&(this.dragStartPosition=this.x,this.startAnimation(),t.removeEventListener("scroll",this),this.dispatchEvent("dragStart",e,[i]))},o.pointerMove=function(t,e){var i=this._dragPointerMove(t,e);this.dispatchEvent("pointerMove",t,[e,i]),this._dragMove(t,e,i)},o.dragMove=function(t,e,i){if(this.isDraggable){t.preventDefault(),this.previousDragX=this.dragX;var n=this.options.rightToLeft?-1:1;this.options.wrapAround&&(i.x=i.x%this.slideableWidth);var s=this.dragStartPosition+i.x*n;if(!this.options.wrapAround&&this.slides.length){var o=Math.max(-this.slides[0].target,this.dragStartPosition);s=s>o?.5*(s+o):s;var r=
Math.min(-this.getLastSlide().target,this.dragStartPosition);s=s<r?.5*(s+r):s}this.dragX=s,this.dragMoveTime=new Date,
-this.dispatchEvent("dragMove",t,[e,i])}},o.dragEnd=function(t,e){if(this.isDraggable){this.options.freeScroll&&(this.isFreeScrolling=!0);var i=this.dragEndRestingSelect();if(this.options.freeScroll&&!this.options.wrapAround){var n=this.getRestingPosition();this.isFreeScrolling=-n>this.slides[0].target&&-n<this.getLastSlide().target}else this.options.freeScroll||i!=this.selectedIndex||(i+=this.dragEndBoostSelect());delete this.previousDragX,this.isDragSelect=this.options.wrapAround,this.select(i),delete this.isDragSelect,this.dispatchEvent("dragEnd",t,[e])}},o.dragEndRestingSelect=function(){var t=this.getRestingPosition(),e=Math.abs(this.getSlideDistance(-t,this.selectedIndex)),i=this._getClosestResting(t,e,1),n=this._getClosestResting(t,e,-1),s=i.distance<n.distance?i.index:n.index;return s},o._getClosestResting=function(t,e,i){for(var n=this.selectedIndex,s=1/0,o=this.options.contain&&!this.options.wrapAround?function(t,e){return t<=e}:function(t,e){return t<e};o(e,s)&&(n+=i,s=e,e
=this.getSlideDistance(-t,n),null!==e);)e=Math.abs(e);return{distance:s,index:n-i}},o.getSlideDistance=function(t,e){var i=this.slides.length,s=this.options.wrapAround&&i>1,o=s?n.modulo(e,i):e,r=this.slides[o];if(!r)return null;var a=s?this.slideableWidth*Math.floor(e/i):0;return t-(r.target+a)},o.dragEndBoostSelect=function(){if(void 0===this.previousDragX||!this.dragMoveTime||new Date-this.dragMoveTime>100)return 0;var t=this.getSlideDistance(-this.dragX,this.selectedIndex),e=this.previousDragX-this.dragX;return t>0&&e>0?1:t<0&&e<0?-1:0},o.staticClick=function(t,e){var i=this.getParentCell(t.target),n=i&&i.element,s=i&&this.cells.indexOf(i);this.dispatchEvent("staticClick",t,[e,n,s])},o.onscroll=function(){var t=s(),e=this.pointerDownScroll.x-t.x,i=this.pointerDownScroll.y-t.y;(Math.abs(e)>3||Math.abs(i)>3)&&this._pointerDone()},e}),function(t,e){"function"==typeof define&&define.amd?define("tap-listener/tap-listener",["unipointer/unipointer"],function(i){return e(t,i)}):"object"=
=typeof module&&module.exports?module.exports=e(t,require("unipointer")):t.TapListener=e(t,t.Unipointer)}(window,function(t,e){function i(t){this.bindTap(t)}var n=i.prototype=Object.create(e.prototype);return n.bindTap=function(t){t&&(this.unbindTap(),this.tapElement=t,this._bindStartEvent(t,!0))},n.unbindTap=function(){this.tapElement&&(this._bindStartEvent(this.tapElement,!0),delete this.tapElement)},n.pointerUp=function(i,n){if(!this.isIgnoringMouseUp||"mouseup"!=i.type){var s=e.getPointerPoint(n),o=this.tapElement.getBoundingClientRect(),r=t.pageXOffset,a=t.pageYOffset,l=s.x>=o.left+r&&s.x<=o.right+r&&s.y>=o.top+a&&s.y<=o.bottom+a;if(l&&this.emitEvent("tap",[i,n]),"mouseup"!=i.type){this.isIgnoringMouseUp=!0;var h=this;setTimeout(function(){delete h.isIgnoringMouseUp},400)}}},n.destroy=function(){this.pointerDone(),this.unbindTap()},i}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/prev-next-button",["./flickity","tap-listener/tap-listener","fizzy-ui-ut
ils/utils"],function(i,n,s){return e(t,i,n,s)}):"object"==typeof module&&module.exports?module.exports=e(t,require("./flickity"),require("tap-listener"),require("fizzy-ui-utils")):e(t,t.Flickity,t.TapListener,t.fizzyUIUtils)}(window,function(t,e,i,n){"use strict";function s(t,e){this.direction=t,this.parent=e,this._create()}function o(t){return"string"==typeof t?t:"M "+t.x0+",50 L "+t.x1+","+(t.y1+50)+" L "+t.x2+","+(t.y2+50)+" L "+t.x3+",50 L "+t.x2+","+(50-t.y2)+" L "+t.x1+","+(50-t.y1)+" Z"}var r="http://www.w3.org/2000/svg";s.prototype=Object.create(i.prototype),s.prototype._create=function(){this.isEnabled=!0,this.isPrevious=this.direction==-1;var t=this.parent.options.rightToLeft?1:-1;this.isLeft=this.direction==t;var e=this.element=document.createElement("button");e.className="flickity-button flickity-prev-next-button",e.className+=this.isPrevious?" previous":" next",e.setAttribute("type","button"),this.disable(),e.setAttribute("aria-label",this.isPrevious?"Previous":"Next")
;var i=this.createSVG();e.appendChild(i),this.on("tap",this.onTap),this.parent.on("select",this.update.bind(this)),this.on("pointerDown",this.parent.childUIPointerDown.bind(this.parent))},s.prototype.activate=function(){this.bindTap(this.element),this.element.addEventListener("click",this),this.parent.element.appendChild(this.element)},s.prototype.deactivate=function(){this.parent.element.removeChild(this.element),i.prototype.destroy.call(this),this.element.removeEventListener("click",this)},s.prototype.createSVG=function(){var t=document.createElementNS(r,"svg");t.setAttribute("class","flickity-button-icon"),t.setAttribute("viewBox","0 0 100 100");var e=document.createElementNS(r,"path"),i=o(this.parent.options.arrowShape);return e.setAttribute("d",i),e.setAttribute("class","arrow"),this.isLeft||e.setAttribute("transform","translate(100, 100) rotate(180) "),t.appendChild(e),t},s.prototype.onTap=function(){if(this.isEnabled){this.parent.uiChange();var t=this.isPrevious?"previous":"n
ext";this.parent[t]()}},s.prototype.handleEvent=n.handleEvent,s.prototype.onclick=function(t){var e=document.activeElement;e&&e==this.element&&this.onTap(t,t)},s.prototype.enable=function(){this.isEnabled||(this.element.disabled=!1,this.isEnabled=!0)},s.prototype.disable=function(){this.isEnabled&&(this.element.disabled=!0,this.isEnabled=!1)},s.prototype.update=function(){var t=this.parent.slides;if(this.parent.options.wrapAround&&t.length>1)return void this.enable();var e=t.length?t.length-1:0,i=this.isPrevious?0:e,n=this.parent.selectedIndex==i?"disable":"enable";this[n]()},s.prototype.destroy=function(){this.deactivate()},n.extend(e.defaults,{prevNextButtons:!0,arrowShape:{x0:10,x1:60,y1:50,x2:70,y2:40,x3:30}}),e.createMethods.push("_createPrevNextButtons");var a=e.prototype;return a._createPrevNextButtons=function(){this.options.prevNextButtons&&(this.prevButton=new s((-1),this),this.nextButton=new s(1,this),this.on("activate",this.activatePrevNextButtons))},a.activatePrevNextBu
ttons=function(){this.prevButton.activate(),this.nextButton.activate(),this.on("deactivate",this.deactivatePrevNextButtons)},a.deactivatePrevNextButtons=function(){this.prevButton.deactivate(),this.nextButton.deactivate(),this.off("deactivate",this.deactivatePrevNextButtons)},e.PrevNextButton=s,e}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/page-dots",["./flickity","tap-listener/tap-listener","fizzy-ui-utils/utils"],function(i,n,s){return e(t,i,n,s)}):"object"==typeof module&&module.exports?module.exports=e(t,require("./flickity"),require("tap-listener"),require("fizzy-ui-utils")):e(t,t.Flickity,t.TapListener,t.fizzyUIUtils)}(window,function(t,e,i,n){function s(t){this.parent=t,this._create()}s.prototype=new i,s.prototype._create=function(){this.holder=document.createElement("ol"),this.holder.className="flickity-page-dots",this.dots=[],this.on("tap",this.onTap),this.on("pointerDown",this.parent.childUIPointerDown.bind(this.parent))},s.prototype.activate=
function(){this.setDots(),this.bindTap(this.holder),this.parent.element.appendChild(this.holder)},s.prototype.deactivate=function(){this.parent.element.removeChild(this.holder),i.prototype.destroy.call(this)},s.prototype.setDots=function(){var t=this.parent.slides.length-this.dots.length;t>0?this.addDots(t):t<0&&this.removeDots(-t)},s.prototype.addDots=function(t){for(var e=document.createDocumentFragment(),i=[],n=this.dots.length,s=n+t,o=n;o<s;o++){var r=document.createElement("li");r.className="dot",r.setAttribute("aria-label","Page dot "+(o+1)),e.appendChild(r),i.push(r)}this.holder.appendChild(e),this.dots=this.dots.concat(i)},s.prototype.removeDots=function(t){var e=this.dots.splice(this.dots.length-t,t);e.forEach(function(t){this.holder.removeChild(t)},this)},s.prototype.updateSelected=function(){this.selectedDot&&(this.selectedDot.className="dot",this.selectedDot.removeAttribute("aria-current")),this.dots.length&&(this.selectedDot=this.dots[this.parent.selectedIndex],this.sel
ectedDot.className="dot is-selected",this.selectedDot.setAttribute("aria-current","step"))},s.prototype.onTap=function(t){var e=t.target;if("LI"==e.nodeName){this.parent.uiChange();var i=this.dots.indexOf(e);this.parent.select(i)}},s.prototype.destroy=function(){this.deactivate()},e.PageDots=s,n.extend(e.defaults,{pageDots:!0}),e.createMethods.push("_createPageDots");var o=e.prototype;return o._createPageDots=function(){this.options.pageDots&&(this.pageDots=new s(this),this.on("activate",this.activatePageDots),this.on("select",this.updateSelectedPageDots),this.on("cellChange",this.updatePageDots),this.on("resize",this.updatePageDots),this.on("deactivate",this.deactivatePageDots))},o.activatePageDots=function(){this.pageDots.activate()},o.updateSelectedPageDots=function(){this.pageDots.updateSelected()},o.updatePageDots=function(){this.pageDots.setDots()},o.deactivatePageDots=function(){this.pageDots.deactivate()},e.PageDots=s,e}),function(t,e){"function"==typeof define&&define.amd?d
efine("flickity/js/player",["ev-emitter/ev-emitter","fizzy-ui-utils/utils","./flickity"],function(t,i,n){return e(t,i,n)}):"object"==typeof module&&module.exports?module.exports=e(require("ev-emitter"),require("fizzy-ui-utils"),require("./flickity")):e(t.EvEmitter,t.fizzyUIUtils,t.Flickity)}(window,function(t,e,i){function n(t){this.parent=t,this.state="stopped",this.onVisibilityChange=this.visibilityChange.bind(this),this.onVisibilityPlay=this.visibilityPlay.bind(this)}n.prototype=Object.create(t.prototype),n.prototype.play=function(){if("playing"!=this.state){var t=document.hidden;if(t)return void document.addEventListener("visibilitychange",this.onVisibilityPlay);this.state="playing",document.addEventListener("visibilitychange",this.onVisibilityChange),this.tick()}},n.prototype.tick=function(){if("playing"==this.state){var t=this.parent.options.autoPlay;t="number"==typeof t?t:3e3;var e=this;this.clear(),this.timeout=setTimeout(function(){e.parent.next(!0),e.tick()},t)}},n.prototy
pe.stop=function(){this.state="stopped",this.clear(),document.removeEventListener("visibilitychange",this.onVisibilityChange)},n.prototype.clear=function(){clearTimeout(this.timeout)},n.prototype.pause=function(){"playing"==this.state&&(this.state="paused",this.clear())},n.prototype.unpause=function(){"paused"==this.state&&this.play()},n.prototype.visibilityChange=function(){var t=document.hidden;this[t?"pause":"unpause"]()},n.prototype.visibilityPlay=function(){this.play(),document.removeEventListener("visibilitychange",this.onVisibilityPlay)},e.extend(i.defaults,{pauseAutoPlayOnHover:!0}),i.createMethods.push("_createPlayer");var s=i.prototype;return s._createPlayer=function(){this.player=new n(this),this.on("activate",this.activatePlayer),this.on("uiChange",this.stopPlayer),this.on("pointerDown",this.stopPlayer),this.on("deactivate",this.deactivatePlayer)},s.activatePlayer=function(){this.options.autoPlay&&(this.player.play(),this.element.addEventListener("mouseenter",this))},s.p
layPlayer=function(){this.player.play()},s.stopPlayer=function(){this.player.stop()},s.pausePlayer=function(){this.player.pause()},s.unpausePlayer=function(){this.player.unpause()},s.deactivatePlayer=function(){this.player.stop(),this.element.removeEventListener("mouseenter",this)},s.onmouseenter=function(){this.options.pauseAutoPlayOnHover&&(this.player.pause(),this.element.addEventListener("mouseleave",this))},s.onmouseleave=function(){this.player.unpause(),this.element.removeEventListener("mouseleave",this)},i.Player=n,i}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/add-remove-cell",["./flickity","fizzy-ui-utils/utils"],function(i,n){return e(t,i,n)}):"object"==typeof module&&module.exports?module.exports=e(t,require("./flickity"),require("fizzy-ui-utils")):e(t,t.Flickity,t.fizzyUIUtils)}(window,function(t,e,i){function n(t){var e=document.createDocumentFragment();return t.forEach(function(t){e.appendChild(t.element)}),e}var s=e.prototype;return s.inse
rt=function(t,e){var i=this._makeCells(t);if(i&&i.length){var s=this.cells.length;e=void 0===e?s:e;var o=n(i),r=e==s;if(r)this.slider.appendChild(o);else{var a=this.cells[e].element;this.slider.insertBefore(o,a)}if(0===e)this.cells=i.concat(this.cells);else if(r)this.cells=this.cells.concat(i);else{var l=this.cells.splice(e,s-e);this.cells=this.cells.concat(i).concat(l)}this._sizeCells(i),this.cellChange(e,!0)}},s.append=function(t){this.insert(t,this.cells.length)},s.prepend=function(t){this.insert(t,0)},s.remove=function(t){var e=this.getCells(t);if(e&&e.length){var n=this.cells.length-1;e.forEach(function(t){t.remove();var e=this.cells.indexOf(t);n=Math.min(e,n),i.removeFrom(this.cells,t)},this),this.cellChange(n,!0)}},s.cellSizeChange=function(t){var e=this.getCell(t);if(e){e.getSize();var i=this.cells.indexOf(e);this.cellChange(i)}},s.cellChange=function(t,e){var i=this.selectedElement;this._positionCells(t),this._getWrapShiftCells(),this.setGallerySize();var n=this.getCell(i);
n&&(this.selectedIndex=this.getCellSlideIndex(n)),this.selectedIndex=Math.min(this.slides.length-1,this.selectedIndex),this.emitEvent("cellChange",[t]),this.select(this.selectedIndex),e&&this.positionSliderAtSelected()},e}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/lazyload",["./flickity","fizzy-ui-utils/utils"],function(i,n){return e(t,i,n)}):"object"==typeof module&&module.exports?module.exports=e(t,require("./flickity"),require("fizzy-ui-utils")):e(t,t.Flickity,t.fizzyUIUtils)}(window,function(t,e,i){"use strict";function n(t){if("IMG"==t.nodeName){var e=t.getAttribute("data-flickity-lazyload"),n=t.getAttribute("data-flickity-lazyload-src"),s=t.getAttribute("data-flickity-lazyload-srcset");if(e||n||s)return[t]}var o="img[data-flickity-lazyload], img[data-flickity-lazyload-src], img[data-flickity-lazyload-srcset]",r=t.querySelectorAll(o);return i.makeArray(r)}function s(t,e){this.img=t,this.flickity=e,this.load()}e.createMethods.push("_createLazyload"
);var o=e.prototype;return o._createLazyload=function(){this.on("select",this.lazyLoad)},o.lazyLoad=function(){var t=this.options.lazyLoad;if(t){var e="number"==typeof t?t:0,i=this.getAdjacentCellElements(e),o=[];i.forEach(function(t){var e=n(t);o=o.concat(e)}),o.forEach(function(t){new s(t,this)},this)}},s.prototype.handleEvent=i.handleEvent,s.prototype.load=function(){this.img.addEventListener("load",this),this.img.addEventListener("error",this);var t=this.img.getAttribute("data-flickity-lazyload")||this.img.getAttribute("data-flickity-lazyload-src"),e=this.img.getAttribute("data-flickity-lazyload-srcset");this.img.src=t,e&&this.img.setAttribute("srcset",e),this.img.removeAttribute("data-flickity-lazyload"),this.img.removeAttribute("data-flickity-lazyload-src"),this.img.removeAttribute("data-flickity-lazyload-srcset")},s.prototype.onload=function(t){this.complete(t,"flickity-lazyloaded")},s.prototype.onerror=function(t){this.complete(t,"flickity-lazyerror")},s.prototype.complete=f
unction(t,e){this.img.removeEventListener("load",this),this.img.removeEventListener("error",this);var i=this.flickity.getParentCell(this.img),n=i&&i.element;this.flickity.cellSizeChange(n),this.img.classList.add(e),this.flickity.dispatchEvent("lazyLoad",t,n)},e.LazyLoader=s,e}),function(t,e){"function"==typeof define&&define.amd?define("flickity/js/index",["./flickity","./drag","./prev-next-button","./page-dots","./player","./add-remove-cell","./lazyload"],e):"object"==typeof module&&module.exports&&(module.exports=e(require("./flickity"),require("./drag"),require("./prev-next-button"),require("./page-dots"),require("./player"),require("./add-remove-cell"),require("./lazyload")))}(window,function(t){return t}),function(t,e){"function"==typeof define&&define.amd?define("flickity-as-nav-for/as-nav-for",["flickity/js/index","fizzy-ui-utils/utils"],e):"object"==typeof module&&module.exports?module.exports=e(require("flickity"),require("fizzy-ui-utils")):t.Flickity=e(t.Flickity,t.fizzyUI
Utils)}(window,function(t,e){function i(t,e,i){return(e-t)*i+t}t.createMethods.push("_createAsNavFor");var n=t.prototype;return n._createAsNavFor=function(){this.on("activate",this.activateAsNavFor),this.on("deactivate",this.deactivateAsNavFor),this.on("destroy",this.destroyAsNavFor);var t=this.options.asNavFor;if(t){var e=this;setTimeout(function(){e.setNavCompanion(t)})}},n.setNavCompanion=function(i){i=e.getQueryElement(i);var n=t.data(i);if(n&&n!=this){this.navCompanion=n;var s=this;this.onNavCompanionSelect=function(){s.navCompanionSelect()},n.on("select",this.onNavCompanionSelect),this.on("staticClick",this.onNavStaticClick),this.navCompanionSelect(!0)}},n.navCompanionSelect=function(t){if(this.navCompanion){var e=this.navCompanion.selectedCells[0],n=this.navCompanion.cells.indexOf(e),s=n+this.navCompanion.selectedCells.length-1,o=Math.floor(i(n,s,this.navCompanion.cellAlign));if(this.selectCell(o,!1,t),this.removeNavSelectedElements(),!(o>=this.cells.length)){var r=this.cells
.slice(n,s+1);this.navSelectedElements=r.map(function(t){return t.element}),this.changeNavSelectedClass("add")}}},n.changeNavSelectedClass=function(t){this.navSelectedElements.forEach(function(e){e.classList[t]("is-nav-selected")})},n.activateAsNavFor=function(){this.navCompanionSelect(!0)},n.removeNavSelectedElements=function(){this.navSelectedElements&&(this.changeNavSelectedClass("remove"),delete this.navSelectedElements)},n.onNavStaticClick=function(t,e,i,n){"number"==typeof n&&this.navCompanion.selectCell(n)},n.deactivateAsNavFor=function(){this.removeNavSelectedElements()},n.destroyAsNavFor=function(){this.navCompanion&&(this.navCompanion.off("select",this.onNavCompanionSelect),this.off("staticClick",this.onNavStaticClick),delete this.navCompanion)},t}),function(t,e){"use strict";"function"==typeof define&&define.amd?define("imagesloaded/imagesloaded",["ev-emitter/ev-emitter"],function(i){return e(t,i)}):"object"==typeof module&&module.exports?module.exports=e(t,require("ev-em
itter")):t.imagesLoaded=e(t,t.EvEmitter)}("undefined"!=typeof window?window:this,function(t,e){function i(t,e){for(var i in e)t[i]=e[i];return t}function n(t){if(Array.isArray(t))return t;var e="object"==typeof t&&"number"==typeof t.length;return e?h.call(t):[t]}function s(t,e,o){if(!(this instanceof s))return new s(t,e,o);var r=t;return"string"==typeof t&&(r=document.querySelectorAll(t)),r?(this.elements=n(r),this.options=i({},this.options),"function"==typeof e?o=e:i(this.options,e),o&&this.on("always",o),this.getImages(),a&&(this.jqDeferred=new a.Deferred),void setTimeout(this.check.bind(this))):void l.error("Bad element for imagesLoaded "+(r||t))}function o(t){this.img=t}function r(t,e){this.url=t,this.element=e,this.img=new Image}var a=t.jQuery,l=t.console,h=Array.prototype.slice;s.prototype=Object.create(e.prototype),s.prototype.options={},s.prototype.getImages=function(){this.images=[],this.elements.forEach(this.addElementImages,this)},s.prototype.addElementImages=function(t){
"IMG"==t.nodeName&&this.addImage(t),this.options.background===!0&&this.addElementBackgroundImages(t);var e=t.nodeType;if(e&&c[e]){for(var i=t.querySelectorAll("img"),n=0;n<i.length;n++){var s=i[n];this.addImage(s)}if("string"==typeof this.options.background){var o=t.querySelectorAll(this.options.background);for(n=0;n<o.length;n++){var r=o[n];this.addElementBackgroundImages(r)}}}};var c={1:!0,9:!0,11:!0};return s.prototype.addElementBackgroundImages=function(t){var e=getComputedStyle(t);if(e)for(var i=/url\((['"])?(.*?)\1\)/gi,n=i.exec(e.backgroundImage);null!==n;){var s=n&&n[2];s&&this.addBackground(s,t),n=i.exec(e.backgroundImage)}},s.prototype.addImage=function(t){var e=new o(t);this.images.push(e)},s.prototype.addBackground=function(t,e){var i=new r(t,e);this.images.push(i)},s.prototype.check=function(){function t(t,i,n){setTimeout(function(){e.progress(t,i,n)})}var e=this;return this.progressedCount=0,this.hasAnyBroken=!1,this.images.length?void this.images.forEach(function(e){e
.once("progress",t),e.check()}):void this.complete()},s.prototype.progress=function(t,e,i){this.progressedCount++,this.hasAnyBroken=this.hasAnyBroken||!t.isLoaded,this.emitEvent("progress",[this,t,e]),this.jqDeferred&&this.jqDeferred.notify&&this.jqDeferred.notify(this,t),this.progressedCount==this.images.length&&this.complete(),this.options.debug&&l&&l.log("progress: "+i,t,e)},s.prototype.complete=function(){var t=this.hasAnyBroken?"fail":"done";if(this.isComplete=!0,this.emitEvent(t,[this]),this.emitEvent("always",[this]),this.jqDeferred){var e=this.hasAnyBroken?"reject":"resolve";this.jqDeferred[e](this)}},o.prototype=Object.create(e.prototype),o.prototype.check=function(){var t=this.getIsImageComplete();return t?void this.confirm(0!==this.img.naturalWidth,"naturalWidth"):(this.proxyImage=new Image,this.proxyImage.addEventListener("load",this),this.proxyImage.addEventListener("error",this),this.img.addEventListener("load",this),this.img.addEventListener("error",this),void(this.pr
oxyImage.src=this.img.src))},o.prototype.getIsImageComplete=function(){return this.img.complete&&this.img.naturalWidth},o.prototype.confirm=function(t,e){this.isLoaded=t,this.emitEvent("progress",[this,this.img,e])},o.prototype.handleEvent=function(t){var e="on"+t.type;this[e]&&this[e](t)},o.prototype.onload=function(){this.confirm(!0,"onload"),this.unbindEvents()},o.prototype.onerror=function(){this.confirm(!1,"onerror"),this.unbindEvents()},o.prototype.unbindEvents=function(){this.proxyImage.removeEventListener("load",this),this.proxyImage.removeEventListener("error",this),this.img.removeEventListener("load",this),this.img.removeEventListener("error",this)},r.prototype=Object.create(o.prototype),r.prototype.check=function(){this.img.addEventListener("load",this),this.img.addEventListener("error",this),this.img.src=this.url;var t=this.getIsImageComplete();t&&(this.confirm(0!==this.img.naturalWidth,"naturalWidth"),this.unbindEvents())},r.prototype.unbindEvents=function(){this.img.re
moveEventListener("load",this),this.img.removeEventListener("error",this)},r.prototype.confirm=function(t,e){this.isLoaded=t,this.emitEvent("progress",[this,this.element,e])},s.makeJQueryPlugin=function(e){e=e||t.jQuery,e&&(a=e,a.fn.imagesLoaded=function(t,e){var i=new s(this,t,e);return i.jqDeferred.promise(a(this))})},s.makeJQueryPlugin(),s}),function(t,e){"function"==typeof define&&define.amd?define(["flickity/js/index","imagesloaded/imagesloaded"],function(i,n){return e(t,i,n)}):"object"==typeof module&&module.exports?module.exports=e(t,require("flickity"),require("imagesloaded")):t.Flickity=e(t,t.Flickity,t.imagesLoaded)}(window,function(t,e,i){"use strict";e.createMethods.push("_createImagesLoaded");var n=e.prototype;return n._createImagesLoaded=function(){this.on("activate",this.imagesLoaded)},n.imagesLoaded=function(){function t(t,i){var n=e.getParentCell(i.img);e.cellSizeChange(n&&n.element),e.options.freeScroll||e.positionSliderAtSelected()}if(this.options.imagesLoaded){va
r e=this;i(this.slider).on("progress",t)}},e});
\ No newline at end of file
[09/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/19ededcc
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/19ededcc
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/19ededcc
Branch: refs/heads/master
Commit: 19ededccf562bc100be7fedd4fbc6a30c13e9cbe
Parents: b729f29 ba0b262
Author: Jagadish <jv...@linkedin.com>
Authored: Thu Oct 18 18:34:32 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Thu Oct 18 18:34:32 2018 -0700
----------------------------------------------------------------------
.../documentation/versioned/connectors/kafka.md | 2 +-
.../samza/application/SamzaApplication.java | 24 +-
.../samza/application/StreamApplication.java | 67 +-
.../samza/application/TaskApplication.java | 70 +--
.../descriptors/ApplicationDescriptor.java | 61 +-
.../StreamApplicationDescriptor.java | 11 +-
.../descriptors/TaskApplicationDescriptor.java | 34 +-
.../context/ApplicationContainerContext.java | 36 +-
.../ApplicationContainerContextFactory.java | 25 +-
.../samza/context/ApplicationTaskContext.java | 32 +-
.../context/ApplicationTaskContextFactory.java | 30 +-
.../apache/samza/context/ContainerContext.java | 21 +-
.../java/org/apache/samza/context/Context.java | 56 +-
.../org/apache/samza/context/JobContext.java | 20 +-
.../org/apache/samza/context/TaskContext.java | 53 +-
.../descriptors/GenericInputDescriptor.java | 14 +-
.../descriptors/GenericOutputDescriptor.java | 14 +-
.../descriptors/GenericSystemDescriptor.java | 13 +-
.../system/descriptors/InputDescriptor.java | 8 +-
.../system/descriptors/OutputDescriptor.java | 8 +-
.../system/descriptors/StreamDescriptor.java | 9 +-
.../system/descriptors/SystemDescriptor.java | 9 +-
.../main/java/org/apache/samza/table/Table.java | 21 +-
.../samza/table/TableDescriptorsProvider.java | 98 ---
.../table/descriptors/TableDescriptor.java | 37 +-
.../java/org/apache/samza/task/TaskFactory.java | 4 +-
.../descriptors/EventHubsInputDescriptor.java | 7 +-
.../descriptors/EventHubsOutputDescriptor.java | 8 +-
.../descriptors/EventHubsSystemDescriptor.java | 7 +-
.../application/LegacyTaskApplication.java | 4 +-
.../TaskApplicationDescriptorImpl.java | 12 +-
.../org/apache/samza/execution/JobPlanner.java | 18 +-
.../apache/samza/processor/StreamProcessor.java | 14 +-
.../application/MockStreamApplication.java | 2 +-
.../samza/application/TestApplicationUtil.java | 2 +-
.../TestTaskApplicationDescriptorImpl.java | 10 +-
.../execution/ExecutionPlannerTestBase.java | 12 +-
.../samza/execution/TestExecutionPlanner.java | 2 +-
.../TestJobNodeConfigurationGenerator.java | 4 +-
.../kafka/descriptors/KafkaInputDescriptor.java | 8 +-
.../descriptors/KafkaOutputDescriptor.java | 8 +-
.../descriptors/KafkaSystemDescriptor.java | 7 +-
.../system/kafka_deprecated/BrokerProxy.scala | 332 ++++++++++
.../DefaultFetchSimpleConsumer.scala | 66 ++
.../system/kafka_deprecated/GetOffset.scala | 116 ++++
.../kafka_deprecated/KafkaSystemAdmin.scala | 609 +++++++++++++++++++
.../kafka_deprecated/KafkaSystemConsumer.scala | 309 ++++++++++
.../KafkaSystemConsumerMetrics.scala | 100 +++
.../kafka_deprecated/KafkaSystemFactory.scala | 178 ++++++
.../kafka_deprecated/KafkaSystemProducer.scala | 235 +++++++
.../KafkaSystemProducerMetrics.scala | 42 ++
.../system/kafka_deprecated/MessageSink.scala | 35 ++
.../kafka_deprecated/TopicMetadataCache.scala | 78 +++
.../samza/system/kafka_deprecated/Toss.scala | 28 +
.../samza/sql/interfaces/DslConverter.java | 1 +
.../interfaces/SamzaSqlJavaTypeFactoryImpl.java | 8 +-
.../samza/sql/runner/SamzaSqlApplication.java | 14 +-
.../example/AppWithGlobalConfigExample.java | 8 +-
.../apache/samza/example/BroadcastExample.java | 10 +-
.../samza/example/KeyValueStoreExample.java | 8 +-
.../org/apache/samza/example/MergeExample.java | 6 +-
.../samza/example/OrderShipmentJoinExample.java | 8 +-
.../samza/example/PageViewCounterExample.java | 6 +-
.../samza/example/RepartitionExample.java | 8 +-
.../samza/example/TaskApplicationExample.java | 12 +-
.../org/apache/samza/example/WindowExample.java | 6 +-
.../TestStandaloneIntegrationApplication.java | 6 +-
.../EndOfStreamIntegrationTest.java | 4 +-
.../WatermarkIntegrationTest.java | 4 +-
.../test/framework/BroadcastAssertApp.java | 6 +-
.../test/framework/FaultInjectionTest.java | 9 +-
.../StreamApplicationIntegrationTest.java | 18 +-
.../framework/StreamTaskIntegrationTest.java | 15 +-
.../samza/test/framework/TestSchedulingApp.java | 4 +-
.../test/operator/RepartitionJoinWindowApp.java | 8 +-
.../test/operator/RepartitionWindowApp.java | 6 +-
.../samza/test/operator/SessionWindowApp.java | 6 +-
.../samza/test/operator/TumblingWindowApp.java | 6 +-
.../test/processor/TestStreamApplication.java | 6 +-
.../apache/samza/test/table/TestLocalTable.java | 9 +-
.../table/TestLocalTableWithSideInputs.java | 8 +-
.../table/TestTableDescriptorsProvider.java | 172 ------
82 files changed, 2711 insertions(+), 701 deletions(-)
----------------------------------------------------------------------
[06/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/080f5e0d
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/080f5e0d
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/080f5e0d
Branch: refs/heads/master
Commit: 080f5e0da94a18569901c28919a2e7a81ba1e718
Parents: 75a53db 62857f6
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 15 16:55:07 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 15 16:55:07 2018 -0700
----------------------------------------------------------------------
docs/_meetups/december-2018.md | 32 --------------------------------
docs/_meetups/july-2018.md | 2 --
docs/_meetups/oct-2018.md | 29 +++++++++++++++++++++++++++++
docs/_meetups/sep-2015.md | 2 +-
docs/css/main.new.css | 2 ++
docs/meetups/index.html | 5 +----
6 files changed, 33 insertions(+), 39 deletions(-)
----------------------------------------------------------------------
[35/50] samza git commit: Cleanup the EventHubs connector section.
Use System/Stream Descriptors
Posted by ja...@apache.org.
Cleanup the EventHubs connector section. Use System/Stream Descriptors
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/d315cae6
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/d315cae6
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/d315cae6
Branch: refs/heads/master
Commit: d315cae63bf514a15152c23b8b7e8fde786f8bc5
Parents: 3273e96 affc312
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 29 16:29:39 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 29 16:29:39 2018 -0700
----------------------------------------------------------------------
docs/learn/documentation/versioned/connectors/kafka.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/d315cae6/docs/learn/documentation/versioned/connectors/kafka.md
----------------------------------------------------------------------
diff --cc docs/learn/documentation/versioned/connectors/kafka.md
index 5b15b91,447bfdc..b71c736
--- a/docs/learn/documentation/versioned/connectors/kafka.md
+++ b/docs/learn/documentation/versioned/connectors/kafka.md
@@@ -136,14 -137,14 +136,14 @@@ In this section, we walk through a comp
{% endhighlight %}
- -Lines 1-3 create a KafkaSystemDescriptor defining the coordinates of our Kafka cluster
+ - Lines 1-3 create a KafkaSystemDescriptor defining the coordinates of our Kafka cluster
- -Lines 4-6 define a KafkaInputDescriptor for our input topic - `page-views`
+ - Lines 4-6 defines a KafkaInputDescriptor for our input topic - `page-views`
- -Lines 7-9 define a KafkaOutputDescriptor for our output topic - `filtered-page-views`
+ - Lines 7-9 defines a KafkaOutputDescriptor for our output topic - `filtered-page-views`
- -Line 9 creates a MessageStream for the input topic so that you can chain operations on it later
+ - Line 9 creates a MessageStream for the input topic so that you can chain operations on it later
- -Line 10 creates an OuputStream for the output topic
+ - Line 10 creates an OuputStream for the output topic
- -Lines 11\-13 define a simple pipeline that reads from the input stream and writes filtered results to the output stream
-- Lines 11-13 define a simple pipeline that reads from the input stream and writes filtered results to the output stream
++- Lines 11-13 define a simple pipeline that reads from the input stream and writes filtered results to the output stream
[07/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/88a4a7cc
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/88a4a7cc
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/88a4a7cc
Branch: refs/heads/master
Commit: 88a4a7ccacdabf38edf1fb7ad3e57c5dd0571f0f
Parents: 080f5e0 a8a8dc7
Author: Jagadish <jv...@linkedin.com>
Authored: Wed Oct 17 19:20:51 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Wed Oct 17 19:20:51 2018 -0700
----------------------------------------------------------------------
docs/learn/documentation/versioned/index.html | 2 +-
.../versioned/jobs/configuration.md | 56 ++++----
.../versioned/jobs/samza-configurations.md | 7 +-
.../descriptors/GenericInputDescriptor.java | 2 +-
.../descriptors/GenericOutputDescriptor.java | 2 +-
.../descriptors/GenericSystemDescriptor.java | 2 +-
.../system/descriptors/InputDescriptor.java | 2 +-
.../system/descriptors/OutputDescriptor.java | 2 +-
.../system/descriptors/StreamDescriptor.java | 2 +-
.../system/descriptors/SystemDescriptor.java | 2 +-
.../descriptors/EventHubsInputDescriptor.java | 17 +--
.../descriptors/EventHubsOutputDescriptor.java | 15 ++-
.../descriptors/EventHubsSystemDescriptor.java | 31 ++---
.../TestEventHubsInputDescriptor.java | 26 +++-
.../TestEventHubsOutputDescriptor.java | 26 +++-
.../TestEventHubsSystemDescriptor.java | 26 ++--
.../ClusterBasedJobCoordinator.java | 2 +-
.../samza/config/ClusterManagerConfig.java | 29 +++--
.../JobNodeConfigurationGenerator.java | 129 +++++++------------
.../org/apache/samza/execution/JobPlanner.java | 66 ++++++----
.../apache/samza/execution/LocalJobPlanner.java | 6 +-
.../samza/execution/RemoteJobPlanner.java | 2 +-
.../samza/util/EmbeddedTaggedRateLimiter.java | 9 +-
.../org/apache/samza/config/JobConfig.scala | 6 +-
.../apache/samza/container/SamzaContainer.scala | 10 +-
.../samza/job/local/ThreadJobFactory.scala | 11 +-
.../samza/execution/TestExecutionPlanner.java | 3 +-
.../TestJobNodeConfigurationGenerator.java | 52 +-------
.../descriptors/TestRemoteTableDescriptor.java | 26 +++-
.../util/TestEmbeddedTaggedRateLimiter.java | 18 ++-
.../samza/system/kafka/KafkaSystemAdmin.java | 90 +++++++------
.../samza/system/kafka/KafkaSystemConsumer.java | 17 +--
.../kafka/descriptors/KafkaInputDescriptor.java | 2 +-
.../descriptors/KafkaOutputDescriptor.java | 2 +-
.../descriptors/KafkaSystemDescriptor.java | 2 +-
.../samza/config/RegExTopicGenerator.scala | 3 +-
.../kafka/DefaultFetchSimpleConsumer.scala | 66 ----------
.../apache/samza/system/kafka/GetOffset.scala | 116 -----------------
.../samza/system/kafka/TestGetOffset.scala | 110 ----------------
.../samza/sql/runner/SamzaSqlApplication.java | 31 ++++-
.../sql/runner/SamzaSqlApplicationContext.java | 11 +-
.../samza/sql/translator/FilterTranslator.java | 13 +-
.../samza/sql/translator/ModifyTranslator.java | 23 +++-
.../samza/sql/translator/ProjectTranslator.java | 12 +-
.../samza/sql/translator/QueryTranslator.java | 88 +++++++------
.../samza/sql/translator/ScanTranslator.java | 20 ++-
.../samza/sql/translator/TranslatorContext.java | 10 +-
.../samza/sql/testutil/SamzaSqlTestConfig.java | 6 +
.../sql/translator/TestFilterTranslator.java | 8 +-
.../sql/translator/TestProjectTranslator.java | 14 +-
.../sql/translator/TestQueryTranslator.java | 91 ++++---------
.../apache/samza/test/framework/TestRunner.java | 19 +--
.../descriptors/InMemoryInputDescriptor.java | 4 +-
.../descriptors/InMemorySystemDescriptor.java | 18 +--
.../test/samzasql/TestSamzaSqlEndToEnd.java | 69 +++++++++-
55 files changed, 615 insertions(+), 819 deletions(-)
----------------------------------------------------------------------
[30/50] samza git commit: Cleanup the deployment section for Samza 1.0
Posted by ja...@apache.org.
Cleanup the deployment section for Samza 1.0
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/1befcad8
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/1befcad8
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/1befcad8
Branch: refs/heads/master
Commit: 1befcad8fb729996da7597984b814380204ac53c
Parents: 48f5f28
Author: Jagadish <jv...@linkedin.com>
Authored: Sat Oct 27 11:12:38 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sat Oct 27 11:12:38 2018 -0700
----------------------------------------------------------------------
.../yarn/coordinator-internals.png | Bin 30163 -> 39061 bytes
.../documentation/versioned/deployment/yarn.md | 225 ++++++++-----------
2 files changed, 93 insertions(+), 132 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/1befcad8/docs/img/versioned/learn/documentation/yarn/coordinator-internals.png
----------------------------------------------------------------------
diff --git a/docs/img/versioned/learn/documentation/yarn/coordinator-internals.png b/docs/img/versioned/learn/documentation/yarn/coordinator-internals.png
index 7f4f161..9a19552 100755
Binary files a/docs/img/versioned/learn/documentation/yarn/coordinator-internals.png and b/docs/img/versioned/learn/documentation/yarn/coordinator-internals.png differ
http://git-wip-us.apache.org/repos/asf/samza/blob/1befcad8/docs/learn/documentation/versioned/deployment/yarn.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/deployment/yarn.md b/docs/learn/documentation/versioned/deployment/yarn.md
index c30346b..b32ba68 100644
--- a/docs/learn/documentation/versioned/deployment/yarn.md
+++ b/docs/learn/documentation/versioned/deployment/yarn.md
@@ -1,6 +1,6 @@
---
layout: page
-title: Run on YARN.
+title: Run on YARN
---
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -45,44 +45,42 @@ title: Run on YARN.
- [Coordinator Internals](#coordinator-internals)
-# Introduction
+## Introduction
-YARN (Yet Another Resource Negotiator) is part of the Hadoop project and provides the ability to run distributed applications on a cluster. A YARN cluster minimally consists of a Resource Manager (RM) and multiple Node Managers (NM). The RM is responsible for coordinating allocations and tracks resources available on the NMs. The NM is an agent that executes on each node in the cluster and is responsible for running containers (user processes), monitoring their resource usage and reporting the same to the ResourceManager. Applications are run on the cluster by implementing a coordinator called an ApplicationMaster (AM). The AM is responsible for requesting resources (CPU, Memory etc) from the Resource Manager (RM) on behalf of the application. The RM allocates the requested resources on one or more NMs that can accomodate the request made.
+Apache YARN is part of the Hadoop project and provides the ability to run distributed applications on a cluster. A YARN cluster minimally consists of a Resource Manager (RM) and multiple Node Managers (NM). The RM is responsible for managing the resources in the cluster and allocating them to applications. Every node in the cluster has an NM (Node Manager), which is responsible for managing containers on that node - starting them, monitoring their resource usage and reporting the same to the RM.
-Samza provides an implementation of the AM in order to run a jobs alongside other application deployed on YARN. The AM makes decisions such as requesting allocation of containers, which machines a Samza job’s containers should run on, what to do when a container fails etc.
+Applications are run on the cluster by implementing a coordinator called an ApplicationMaster (AM). The AM is responsible for requesting resources including CPU, memory from the Resource Manager (RM) on behalf of the application. Samza provides its own implementation of the AM for each job.
+## Running on YARN: Quickstart
-# Starting your application on YARN
+We will demonstrate running a Samza application on YARN by using the `hello-samza` example. Lets first checkout our repository.
-## Setting up a single node YARN cluster (optional)
-
-If you already have a YARN cluster setup to deploy jobs, please jump to [Submitting the application to YARN](#submitting-the-application-to-yarn). If not the following section will help set up a single node cluster to test a Samza job deploy.
+```bash
+git clone https://github.com/apache/samza-hello-samza.git
+cd samza-hello-samza
+git checkout latest
+```
-We can use the `grid` script which is part of the [hello-samza](https://github.com/apache/samza-hello-samza/) repository to setup a single node YARN cluster (and optionally a Zookeeper and Kafka cluster as well).
+### Set up a single node YARN cluster
-Run the following to setup a single node YARN cluster:
+You can use the `grid` script included as part of the [hello-samza](https://github.com/apache/samza-hello-samza/) repository to setup a single-node cluster. The script also starts Zookeeper and Kafka locally.
-```bash
-./grid install yarn
-./grid start yarn
+```
+./bin/grid bootstrap
```
-## Submitting the application to YARN
+### Submitting the application to YARN
-Assuming you have a YARN cluster setup, let us take a look at building your application and deploying it to YARN. Samza provides shell scripts as part of the `samza-shell` module that help in submitting the application to YARN and you should include it as part of your dependencies jobs dependencies.
+Now that we have a YARN cluster ready, lets build our application. The below command does a maven-build and generates an archive in the `./target` folder.
-```xml
-<dependency>
- <groupId>org.apache.samza</groupId>
- <artifactId>samza-shell</artifactId>
- <version>${samza.version}</version>
-</dependency>
+```bash
+./bin/build-package.sh
```
-Samza jobs are usually deployed in a tarball and archive should contain the following as top-level directories.
+You can inspect the structure of the generated archive. To run on YARN, Samza jobs should be packaged with the following structure.
```bash
-samza-job-artifact-folder
+samza-job-name-folder
├── bin
│ ├── run-app.sh
│ ├── run-class.sh
@@ -96,94 +94,81 @@ samza-job-artifact-folder
├── samza-yarn_2.11-0.14.0.jar
└── ...
```
-The scripts in the `samza-shell` module make the assumption that the built artifact (tarball) has the exact directory structure as seen above. The scripts in the samza-shell module should be copied to a bin directory and all jars need to be part of lib as seen above. The hello-samza project is a good example on setting the structure of your application’s build.
-Once the job is built, the `run-app.sh` script can be used to submit the application to the Resource Manager. The script takes 2 CLI parameters - the config factory and the config file for the application. It can be invoked as follows:
+Once the archive is built, the `run-app.sh` script can be used to submit the application to YARN's resource manager. The script takes 2 CLI parameters - the config factory and the config file for the application. As an example, lets run our [FilterExample](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) on YARN as follows:
```bash
-$ /path/to/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://path/to/config/application.properties
+$ ./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path ./deploy/samza/config/filter-example.properties
```
-Make sure that the following configurations are set in your configs.
-
-```properties
-job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
-yarn.package.path=https://url/to/artifact/artifact-version-dist.tar.gz
-```
+Congratulations, you've successfully submitted your first job to YARN! You can view the YARN Web UI to view its status.
-# Application Master UI
-The AM implementation in Samza exposes metadata about the job via both a JSON REST interface and a Web UI.
-This Web UI can be accessed by clicking the Tracking UI (*ApplicationMaster*) link on the YARN RM dashboard.
+## Application Master UI
+The YARN RM provides a Web UI to view the status of applications in the cluster, their containers and logs. By default, it can be accessed from `localhost:8088` on the RM host.
-The Application Master UI provides you the ability to view:
+In addition to YARN's UI, Samza also offers a REST end-point and a web interface for its ApplicationMaster. To access it, simply click on the Tracking UI link corresponding to your application.
+Samza's Application Master UI provides you the ability to view:
- - Job level runtime metadata
+ - Job-level runtime metadata - eg: JMX endpoints, running JVM version
- - Container information
+ - Information about individual containers eg: their uptime, status and logs
- - Job model (SystemStreamPartition to Task and Container mapping)
+ - Task Groups eg: Information on individual tasks, where they run and which partitions are consumed from what host
-- Runtime configs
+ - Runtime configs for your application
-# Viewing logs
+### Configurations
-Each container produces logs and they can be easily accessed via the Container information page in ApplicationMaster UI described in the previous section. Clicking on the container name under the Running or Failed container section will take you to the logs page that corresponds to that specific container.
+In this section, we'll look at configuring your jobs when running on YARN.
-If there is a need to analyze logs across containers, it is recommended to set up a centralized logging system like ELK (Elasticsearch, Logstash and Kibana). Samza provides a StreamAppender that supports emitting your logs to Kafka for this purpose. The logs written to the stream can then be ingested by service like Logstash and indexed in Elasticsearch.
+#### Configuring parallelism
+[Recall](/learn/documentation/{{site.version}}/architecture/architecture-overview.html#container) that Samza scales your applications by breaking them into multiple tasks. On YARN, these tasks are executed on one or more containers, each of which is a Java process. You can control the number of containers allocated to your application by configuring `cluster-manager.container.count`. For example, if we are consuming from an input topic with 5 partitions, Samza will create 5 tasks, each of which process one partition. Tasks are equally distributed among available containers. The number of containers can be utmost the number of tasks - since, we cannot have idle containers without any tasks assigned to them.
-# Configuration
+#### Configuring resources
-In the following section let's take a look at the different tunables that exist as part of Samza to control your deployment on YARN
+Samza jobs on YARN run on a multi-tenant cluster and should be isolated from each other. YARN implements isolation by enforcing limits on memory and CPU each application can use.
-## Configuring parallelism
+##### Memory
-As a refresher, Samza scales your applications by breaking them into multiple tasks. On YARN, these tasks are executed on one or more containers, each of which is a Java process. You can control the number of containers allocated to your job by configuring `cluster-manager.container.count`. For example If we had 2 input topics with 10 partitions each processor would consist of 10 Tasks, each processing 2 partitions each. Setting `cluster-manager.container.count` to 1 would run all 10 tasks in one JVM process, setting it to 2 will distribute the tasks equally among 2 JVM processes and so on.
+You can configure the memory-limit per-container using `cluster-manager.container.memory.mb` and memory-limit for the AM using `yarn.am.container.memory.mb`. If your container process exceeds its configured memory-limits, it is automatically killed by YARN.
-Please note that it is not possible to distribute 10 tasks across more than 10 containers, therefore the upper bound for `cluster-manager.container.count` is less than or equal to the number of Tasks in your job (or more generally the max of number of partitions among all input streams).
+##### CPU
-## Configuring resources
+Similar to configuring memory-limits, you can configure the maximum number of vCores (virtual cores) each container can use by setting `cluster-manager.container.cpu.cores`. A _vCore_ is YARN's abstraction over a physical core on a NodeManager which allows for over-provisioning. YARN supports [isolation]((http://riccomini.name/posts/hadoop/2013-06-14-yarn-with-cgroups/)) of cpu cores using Linux CGroups.
-When running Samza jobs in a shared environment, the stream processors can have an impact on each other’s performance. YARN prevents these issues by providing isolation when running applications on the cluster by enforcing strict limits on resources that each application is allowed to use. YARN (2.7.*) currently supports resource management for memory and CPU.
-### Memory
+#### Configuring retries
-All containers requests by the Application Master will have a max-memory size defined when they’re created. Samza supports configuring these memory limits using `cluster-manager.container.memory.mb` and `yarn.am.container.memory.mb`. If your container exceeds the configured memory-limits, it is automatically killed by YARN. Keep in mind that this is the maximum memory YARN will allow a Samza Container or ApplicationMaster to have and you will still need to configure your heap settings appropriately using `task.opts`, when using the JVM.
+Failures are common when running any distributed system and should be handled gracefully. The Samza AM automatically restarts containers during a failure. The following properties govern this behavior.
-As a cluster administrator if you are running other processes on the same box as the Node Managers (eg: samza-rest) you will want to reserve appropriate amount of memory by configuring `yarn.nodemanager.resource.system-reserved-memory-mb`. Another behaviour to keep in mind is that the Resource Manager allocates resource on the cluster in increments of `yarn.scheduler.minimum-allocation-mb` and `yarn.scheduler.minimum-allocation-vcores`, therefore requesting allocations that are not multiples of the above configs can lead to resource fragmentation.
+`cluster-manager.container.retry.count`: This property determines the maximum number of times Samza will attempt to restart a failed container within a time window. If this property is set to 0, any failed container immediately causes the whole job to fail. If it is set to a negative number, there is no limit on the number of retries.
-### CPU
-Similar to memory configurations all containers also are CPU bound to a max number of vCores (Virtual cores) on a NM that they are configured to use. YARN has the concept of a virtual core which is generally set to the number of physical cores on the NMs, but can be bump to a higher number if you want to over-provision the NMs with respect to the CPU. Samza supports configuring the vCore of each container by setting `cluster-manager.container.cpu.cores`.
+`cluster-manager.container.retry.window.ms`: This property determines how frequently a container is allowed to fail before we give up and fail the job. If the same container has failed more than cluster-manager.container.retry.count times and the time between failures is less than this property, then Samza terminates the job. There is no limit to the number of times we restart a container, if the time between failures is greater than cluster-manager.container.retry.window.ms.
-Unlike memory, which YARN can enforce limits by itself (by looking at the /proc folder), YARN can’t enforce CPU isolation, since this must be done at the Linux kernel level. One of YARN’s features is its support for Linux CGroups (used to control process utilization at the kernel level in Linux). If YARN is setup to use CGroups, then it will guarantee that a container will get at least the amount of CPU that it requires. Currently, by default YARN will give you more CPU to the container, if it’s available. If enforcing “at most” CPU usage for more predictable performance by your container at the cost of underutilization you can set `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage` to `true`, see [this article](https://hortonworks.com/blog/apache-hadoop-yarn-in-hdp-2-2-isolation-of-cpu-resources-in-your-hadoop-yarn-clusters/) for more details. For an indepth look at using YARN with CGroups take a look at [this blog post](http://riccomini.name/posts/h
adoop/2013-06-14-yarn-with-cgroups/).
-## Configuring retries
+## YARN - Operations Best practices
-Failures are common when running a distributed system and the AM is used to handle Samza Container failures gracefully by automatically restarting containers on failure.
-It should also be noted that if a Samza Container keeps failing constantly it could indicate a deeper problem and we should kill the job rather than having the AM restart it indefinitely. `cluster-manager.container.retry.count` can be used to set the maximum number of times a failed container will be restarted within a time window (configured with `cluster-manager.container.retry.window.ms`), before shutting down the job.
-YARN also provides us a way to automatically restart the job if the AM process fails due to external issues (network partitions, hardware failures etc). By configuring the value of `yarn.resourcemanager.am.max-attempts` YARN will automatically restart the AM process for a fixed number of times before requiring manual intervention to start the job again.
+Although this section is not Samza specific, it describes some best practices for running a YARN cluster in production.
-## Configuring RM high-availability and NM work-preserving recovery
-
-Although this section is not Samza specific, it talks about some of the best practices for running a YARN cluster in production specifically around running a highly-available Resource Manager and NodeManager work preserving recovery.
### Resource Manager high-availability
-The Resource Manager (RM) component of a YARN cluster is the source of truth regarding resource utilization and resource scheduling in the cluster. Losing the host running the RM process would kill every single application running on the cluster - making it a single point of failure. The High Availability feature introduced in Hadoop 2.4 adds redundancy in the form of standby Resource Managers to remove this single point of failure.
+The Resource Manager (RM) provides services like scheduling, heartbeats, liveness monitoring to all applications running in the YARN cluster. Losing the host running the RM would kill every application running on the cluster - making it a single point of failure. The High Availability feature introduced in Hadoop 2.4 adds redundancy by allowing multiple stand-by RMs.
-In order to configure YARN to run the highly available Resource Manager process set your yarn-site.xml file with the following configs:
+To configure YARN's ResourceManager to be highly available Resource Manager, set your yarn-site.xml file with the following configs:
```xml
<property>
@@ -220,108 +205,84 @@ In order to configure YARN to run the highly available Resource Manager process
</property>
```
-### NodeManager work-preserving recovery
-
-Turning on work-preserving recovery for the NM gives you the ability to perform maintenance on the cluster (kill NM process for a short duration) without having the containers that run on the node also get killed. You can turn on this feature by setting `yarn.nodemanager.recovery.enabled` to `true` in `yarn-site.xml`
-
-It is also recommended that you change the value of `yarn.nodemanager.recovery.dir` as by default this directory is set to `${hadoop.tmp.dir}/yarn-nm-recovery` where `hadoop.tmp.dir` is set to `/tmp/hadoop-${user.name}` and usually the contents of the `/tmp` directory are not preserved across a reboots.
-
+### Reserving memory for other services
-## Configuring host-affinity
-
-When a stateful Samza job is deployed in YARN, the state stores for the tasks are co-located in the current working directory of YARN’s application attempt.
-
-```properties
-container_working_dir=${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}/container_${contid}/
-# Data Stores
-ls ${container_working_dir}/state/${store-name}/${task_name}/
-```
+Often, other services including monitoring daemons like Samza-REST run on the same nodes in the YARN cluster. You can configure `yarn.nodemanager.resource.system-reserved-memory-mb` to control the amount of physical memory reserved for non-YARN processes.
-This allows the Node Manager’s (NM) DeletionService to clean-up the working directory once the application completes or fails. In order to re-use local state store, the state store needs to be persisted outside the scope of NM’s deletion service. The cluster administrator should set this location as an environment variable in YARN ( `LOGGED_STORE_BASE_DIR`).
+Another behaviour to keep in mind is that the Resource Manager allocates memory and cpu on the cluster in increments of `yarn.scheduler.minimum-allocation-mb` and `yarn.scheduler.minimum-allocation-vcores`. Hence, requesting allocations that are not multiples of the above configs will cause internal fragmentation.
-Since we store the state stores outside of the container’s working directory it is necessary to periodically clean-up unused or orphaned state stores on the machines to manage disk-space. This can be done by running a clean up periodically from the samza-rest service (*LocalStoreMonitor*) that is meant to be deployed on all Node Manager hosts.
+### NodeManager work-preserving recovery
-## Configuring security
-
-You can run a Samza job on a secure YARN cluster. YARN uses Kerberos as its authentication and authorization mechanism. Take a look at the official YARN [documentation](https://hadoop.apache.org/docs/r2.7.4/hadoop-project-dist/hadoop-common/SecureMode.html) page for more details.
-
-### Delegation token management strategy
-
-One of the challenges for long-lived applications running on a secure YARN cluster is its token renewal strategy. Samza handles this by having the AM periodically re-authenticate itself with the given principal and keytab. It periodically creates new delegation tokens and stores them in a job specific staging directory on HDFS accessible only by the AM and its Containers. In this process each running container will get new delegation tokens from the credentials file on HDFS before the current ones expire. The AM and Containers don’t need to communicate with each other in this process and each side proceeds independently by accessing the tokens on HDFS.
+Often, NMs have to be bounced in the cluster for upgrades or maintenance reasons. By default, bouncing a Node Manager kills all containers running on its host. Work-preserving NM Restart enables NodeManagers to be restarted without losing active containers running on the node. You can turn on this feature by setting `yarn.nodemanager.recovery.enabled` to `true` in `yarn-site.xml`. You should also set `yarn.nodemanager.recovery.dir` to a directory where the NM should store its state needed for recovery.
-By default, any HDFS delegation token has a maximum life of 7 days (configured by `dfs.namenode.delegation.token.max-lifetime` in `hdfs-site.xml`) and the token is normally renewed every 24 hours (configured by `dfs.namenode.delegation.token.renew-interval` in `hdfs-site.xml`). What if the Application Master dies and needs restarts after 7 days? The original HDFS delegation token stored in the launcher context will be invalid no matter what. Luckily, Samza can rely on Resource Manager to handle this scenario. See the Configuration section below for details.
+### Configuring state-store directories
-### Security Components
+When a stateful Samza job is deployed in YARN, the state stores for the tasks are located in the current working directory of YARN’s attempt. YARN's DeletionService cleans up the working directories after an application exits. To ensure durability of Samza's state, its stores need to be persisted outside the scope of YARN's DeletionService. You can set this location by configuring an environment variable named `LOGGED_STORE_BASE_DIR` across the cluster.
-#### SecurityManager
+To manage disk space and clean-up state stores that are no longer necessary, Samza-REST supports periodic, long-running tasks named [monitors](/learn/documentation/{{site.version}}/rest/monitors.html).
-When ApplicationMaster starts, it spawns `SamzaAppMasterSecurityManager`, which runs on its separate thread. The `SamzaAppMasterSecurityManager` is responsible for periodically logging in through the given Kerberos keytab and regenerates the HDFS delegation tokens regularly. After each run, it writes new tokens on a pre-defined job specific directory on HDFS. The frequency of this process is determined by `yarn.token.renewal.interval.seconds`.
+### Configuring security
-Each container, upon start, runs a `SamzaContainerSecurityManager`. It reads from the credentials file on HDFS and refreshes its delegation tokens at the same interval.
+You can run Samza jobs on a secure YARN cluster. YARN uses Kerberos as its authentication and authorization mechanism. See [this article](https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cdh_sg_yarn_security.html) for details on operating Hadoop in secure mode.
-### Security configuration
-For the Samza job, the following job configurations are required on a YARN cluster with security enabled.
+#### Management of Kerberos tokens
-#### Job
+One challenge for long-running applications on YARN is how they periodically renew their Kerberos tokens. Samza handles this by having the AM periodically create tokens and refresh them in a staging directory on HDFS. This directory is accessible only by the containers of your job. You can set your Kerberos principal and kerberos keytab file as follows:
```properties
+# Use the SamzaYarnSecurityManagerFactory, which fetches and renews the Kerberos delegation tokens when the job is running in a secure environment.
job.security.manager.factory=org.apache.samza.job.yarn.SamzaYarnSecurityManagerFactory
-```
-#### YARN
+# Kerberos principal
+yarn.kerberos.principal=your-principal-name
-```properties
-yarn.kerberos.principal=user/localhost
-yarn.kerberos.keytab=/etc/krb5.keytab.user
-yarn.token.renewal.interval.seconds=86400
+# Path of the keytab file (local path)
+yarn.kerberos.keytab=/tmp/keytab
```
-Configure the Hadoop cluster to enable Resource Manager to recreate and renew the delegation token on behalf of the application user. This will address the following 2 scenarios.
-
-- When Application Master dies unexpectedly and needs a restart after 7 days (the default maximum lifespan a delegation token can be renewed).
-- When the Samza job terminates and log aggregation is turned on for the job. Node managers need to be able to upload all the local application logs to HDFS.
-Enable the resource manager as a privileged user in yarn-site.xml.
+By default, Kerberos tokens on YARN have a maximum life-time of 7 days, beyond which they auto-expire. Often streaming applications are long-running and don't terminate within this life-time. To get around this, you can configure YARN's Resource Manager to automatically re-create tokens on your behalf by setting these configs in your `yarn-site.xml` file.
```xml
<property>
- <name>yarn.resourcemanager.proxy-user-privileges.enabled</name>
- <value>true</value>
+<name>hadoop.proxyuser.yarn.hosts</name>
+<value>*</value>
</property>
-```
-
-Make `yarn` as a proxy user, in `core-site.xml`
-```xml
-<property>
- <name>hadoop.proxyuser.yarn.hosts</name>
- <value>*</value>
-</property>
<property>
- <name>hadoop.proxyuser.yarn.groups</name>
- <value>*</value>
+<name>hadoop.proxyuser.yarn.groups</name>
+<value>*</value>
</property>
```
+# Samza Coordinator Internals
-# Coordinator Internals
+In this section, we will discuss some of implementation internals of the Samza ApplicationMaster (AM).
-The `ClusterBasedJobCoordinator` is used as the control hub for a running Samza job in a cluster like YARN. Among other things it is responsible for bootstrapping configs from the Coordinator Stream on job startup, constructing the JobModel, managing container allocations and handling callbacks from the cluster manager (in YARN’s case the Resource Manager). Just like most other components in the framework, Samza has a plugable interface for managing container allocations and is configured using the key `samza.cluster-manager.factory`.
+The Samza AM is the control-hub for a Samza application running on a YARN cluster. It is responsible for coordinating work assignment across individual containers. It includes the following componeents:
+- YARNClusterResourceManager, which handles interactions with YARN and provides APIs for requesting resources and starting containers.
+- ContainerProcessManager, which uses the above APIs to manage Samza containers - including restarting them on failure, ensuring they stay in a healthy state.
-The `ClusterBasedJobCoordinator` contains a component called the `ContainerProcessManager` to handle metadata regarding container allocations. It uses the information (eg: host affinity) obtained from configs and the `CoordinatorStream` in order to make container allocation requests to the cluster manager (RM). In the case of YARN the config for `samza.cluster-manager.factory` which encapsulates the Application Master, is configured to `org.apache.samza.job.yarn.YarnResourceManagerFactory` and the `ContainerProcessManager` uses `YarnResourceManager` to interact with the RM.
+Here's a life-cycle of a Samza job submitted to YARN:
+
+- The `run-app.sh` script is started providing the location of your application's binaries and its config file. The script instantiates an ApplicationRunner, which is the main entry-point responsible for running your application.
+
+- The ApplicationRunner parses your configs and writes them to a special Kafka topic named - the coordinator stream for distributing them. It proceeds to submit a request to YARN to launch your application.
+
+- The first step in launching any YARN application is starting its Application Master (AM).
+
+- The ResourceManager allocates an available host and starts the Samza AM.
+
+- The Samza AM is then responsible for managing the overall application. It reads configs from the Coordinator Stream and computes work-assignments for individual containers.
+
+- It also determines the hosts each container should run on taking data-locality into account. It proceeds to request resources on those nodes using the `YARNClusterResourceManager` APIs.
-The following is a walkthrough of the different actions taken when the `run-job.sh` script is run:
-- When the job is submitted using `run-app.sh` the JobRunner invoked as part of this script first writes all the configs to the coordinator stream.
-- The JobRunner then uses the configured StreamJob (YarnJob) to submit the request to start the AM to the RM.
-- The ResourceManager allocates the AM on an available NM and starts the ClusterBasedJobCoordinator.
-- The ClusterBasedJobCoordinator bootstraps the configs written to the Coordinator Stream in step (1) and constructs the JobModel, check for host-affinity if configured and instantiates the ClusterResourceManager (YarnClusterResourceManager).
-- The YarnClusterResourceManager is then used to make requests to the RM to start job.container.count number of containers. The RM then issues callbacks to the process when the containers are allocated.
-- When the containers are returned by the RM, the YarnClusterResourceManager allocates a SamzaContainer ID to the YARN containers to indicate which subset of tasks in the JobModel the YARN container should process on startup.
-- When the containers start up, they read the configs and the JobModel from the configs and use their own SamzaContainer ID and the JobModel to pick specific tasks and start message processing.
+- Once resources have been allocated, it proceeds to start the containers on the allocated hosts.
+- When it is started, each container first queries the Samza AM to determine its work-assignments and configs. It then proceeds to execute its assigned tasks.
-During the course of processing message all container failures will result in a callback from the RM to the YarnClusterResourceManager. These callbacks can then be used to request for a new container and restart processing from the last checkpoint, thus making YARN deployments resilient to container failures.
+- The Samza AM periodically monitors each container using heartbeats and ensure they stay alive.
\ No newline at end of file
[20/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/04a12767
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/04a12767
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/04a12767
Branch: refs/heads/master
Commit: 04a12767b9e7b5246bce51b3f00594e9fd5d7a7d
Parents: b027b43 282f834
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:10:51 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:10:51 2018 -0700
----------------------------------------------------------------------
.../documentation/versioned/api/samza-sql.md | 2 +-
docs/learn/tutorials/versioned/samza-sql.md | 2 +-
docs/learn/tutorials/versioned/samza-tools.md | 6 +-
.../kafka_deprecated/TestBrokerProxy.scala | 434 +++++++++++++
.../system/kafka_deprecated/TestGetOffset.scala | 110 ++++
.../kafka_deprecated/TestKafkaSystemAdmin.scala | 351 +++++++++++
.../TestKafkaSystemConsumer.scala | 191 ++++++
.../TestKafkaSystemFactory.scala | 98 +++
.../TestKafkaSystemProducer.scala | 604 +++++++++++++++++++
.../TestTopicMetadataCache.scala | 139 +++++
.../TestStandaloneIntegrationApplication.java | 4 +-
11 files changed, 1932 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
[22/50] samza git commit: Revert "clean-up the quickstart page"
Posted by ja...@apache.org.
Revert "clean-up the quickstart page"
This reverts commit cd34d98bf638d84ec652f006844e05d343278d7a.
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/2615ad56
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/2615ad56
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/2615ad56
Branch: refs/heads/master
Commit: 2615ad5642b3cbcf9de1f79523c531c4204e6b09
Parents: cd34d98
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:18:01 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:18:01 2018 -0700
----------------------------------------------------------------------
docs/_docs/replace-versioned.sh | 5 +--
docs/_menu/index.html | 2 +-
docs/startup/code-examples/versioned/index.md | 49 ----------------------
docs/startup/quick-start/versioned/index.md | 26 +++++-------
4 files changed, 12 insertions(+), 70 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/_docs/replace-versioned.sh
----------------------------------------------------------------------
diff --git a/docs/_docs/replace-versioned.sh b/docs/_docs/replace-versioned.sh
index c454cac..24bf7ae 100755
--- a/docs/_docs/replace-versioned.sh
+++ b/docs/_docs/replace-versioned.sh
@@ -44,7 +44,4 @@ echo "replaced startup/hello-samza/versioned to startup/hello-samza/"$version
mv -f $DIR/_site/startup/hello-samza/versioned $DIR/_site/startup/hello-samza/$version
echo "replaced startup/quick-start/versioned to startup/quick-start/"$version
-mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
-
-echo "replaced startup/code-examples/versioned to startup/code-examples/"$version
-mv -f $DIR/_site/startup/code-examples/versioned $DIR/_site/startup/code-examples/$version
+mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/_menu/index.html
----------------------------------------------------------------------
diff --git a/docs/_menu/index.html b/docs/_menu/index.html
index a363bae..0d1750f 100644
--- a/docs/_menu/index.html
+++ b/docs/_menu/index.html
@@ -5,7 +5,7 @@ items:
- menu_title: QuickStart
url: /startup/quick-start/version/
- menu_title: Code Examples
- url: /startup/code-examples/version/
+ url: /learn/tutorials/version/
- menu_title: Documentation
has_sub: true
has_sub_subs: true
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/startup/code-examples/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/code-examples/versioned/index.md b/docs/startup/code-examples/versioned/index.md
deleted file mode 100644
index ba1cc3e..0000000
--- a/docs/startup/code-examples/versioned/index.md
+++ /dev/null
@@ -1,49 +0,0 @@
----
-layout: page
-title:
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-
-### Checking out our examples
-
-The [hello-samza](https://github.com/apache/samza-hello-samza) project contains several examples to help you create your Samza applications. To checkout the hello-samza project:
-
-{% highlight bash %}
-> git clone https://git.apache.org/samza-hello-samza.git hello-samza
-{% endhighlight %}
-
-#### High-level API examples
-[The Samza Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook) contains various recipes using the Samza high-level API.
-These include:
-
-- The [Filter example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) demonstrates how to perform stateless operations on a stream.
-
-- The [Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/JoinExample.java]) demonstrates how you can join a Kafka stream of page-views with a stream of ad-clicks
-
-- The [Stream-Table Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/RemoteTableJoinExample.java) demonstrates how the Samza Table API. It joins a Kafka stream with a remote dataset accessed through a REST service.
-
-- The [SessionWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/SessionWindowExample.java) and [TumblingWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/TumblingWindowExample.java) examples illustrate Samza's rich windowing and triggering capabilities.
-
-
-In addition to the cookbook, you can also consult these:
-
-- [Wikipedia Parser](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): An advanced example that builds a streaming pipeline consuming a live-feed of wikipedia edits, parsing each message and generating statistics from them.
-
-
-- [Amazon Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis) and [Azure Eventhubs](https://github.com/apache/samza-hello-samza/tree/latest/src/main/java/samza/examples/azure) examples that cover how to consume input data from the respective systems.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/startup/quick-start/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/quick-start/versioned/index.md b/docs/startup/quick-start/versioned/index.md
index da20db2..a046ee7 100644
--- a/docs/startup/quick-start/versioned/index.md
+++ b/docs/startup/quick-start/versioned/index.md
@@ -65,7 +65,7 @@ public class WordCount implements StreamApplication {
The StreamApplication interface provides an API method named describe() for you to specify your streaming pipeline. Using [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html), you can describe your entire data processing task from data inputs, operations and outputs.
-### Describe your inputs and outputs
+### Input data source using Kafka
In this example, we are going to use Kafka as the input data source and consume the text for word count line by line. We start by defining a KafkaSystemDescriptor, which specifies the properties to establishing the connection to the local Kafka cluster. Then we create a `KafkaInputDescriptor`/`KafkaOutputDescriptor` to set up the topic, Serializer and Deserializer. Finally we use this input in the [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html) so we can consume from this topic. The code is in the following:
@@ -106,7 +106,7 @@ The resulting [MessageStream](/learn/documentation/{{site.version}}/api/javadocs
First we are going to extract the value from lines. This is a one-to-one transform and we can use the Samza map operator as following:
{% highlight java %}
-lines.map(kv -> kv.value)
+lines .map(kv -> kv.value)
{% endhighlight %}
Then we will split the line into words by using the flatmap operator:
@@ -152,29 +152,23 @@ lines
In this section we will configure the word count example to run locally in a single JVM. Please add a file named “word-count.properties” under the config folder. We will add the job configs in this file.
-In this section, we will configure our word count example to run locally in a single JVM. Let us add a file named “word-count.properties” under the config folder.
+Since there is only a single Samza processor, there is no coordination required. We use the PassthroughJobCoordinator for the example. We also group all Samza tasks into this single processor. As for the Kafka topic, we will consume from the beginning. Here is the full config needed for the job:
{% highlight jproperties %}
job.name=word-count
-# Use a PassthroughJobCoordinator since there is no coordination needed
job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory
job.coordination.utils.factory=org.apache.samza.standalone.PassthroughCoordinationUtilsFactory
-
job.changelog.system=kafka
-
-# Use a single container to process all of the data
task.name.grouper.factory=org.apache.samza.container.grouper.task.SingleContainerGrouperFactory
processor.id=0
-
-# Read from the beginning of the topic
systems.kafka.default.stream.samza.offset.default=oldest
{% endhighlight %}
-For more details on Samza's configs, feel free to check out the latest [configuration reference](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
+For more details about Samza config, feel free to check out the latest config [here](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
### Run your application
-Let’s now add a `main()` function to the `WordCount` class. The function reads the config file and factory from the args, and creates a `LocalApplicationRunner` that run the application locally.
+Let’s add a `main()` function to `WordCount` class first. The function reads the config file and factory from the args, and create a `LocalApplicationRunner` to run the application locally. Here is the function details:
{% highlight java %}
public static void main(String[] args) {
@@ -195,7 +189,7 @@ apply plugin:'application'
mainClassName = "samzaapp.WordCount"
{% endhighlight %}
-Before running `main()`, we will create our input Kafka topic and populate it with sample data. You can download the scripts to interact with Kafka along with the sample data from [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
+Before running `main()`, we need to create the input Kafka topic with some sample data. Let’s start a local kafka broker first. Samza examples provides a script named “grid” which you can use to start zookeeper, kafka broker and yarn. Your can download it [here](https://github.com/apache/samza-hello-samza/blob/master/bin/grid) and put it under scripts/ folder, then issue the following command:
{% highlight bash %}
> ./scripts/grid install zookeeper && ./scripts/grid start zookeeper
@@ -251,10 +245,10 @@ The [hello-samza](https://github.com/apache/samza-hello-samza) project contains
There are four main categories of examples in this project, including:
-1. [Wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
+1. [wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
-2. [Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
+2. [cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
-3. [Azure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): This example shows how to build an application that consumes input streams from Azure EventHubs.
+3. [asure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): this example shows how to run your application on Microsoft Asure.
-4. [Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): This example shows how to consume from Kinesis streams.
\ No newline at end of file
+4. [kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): this example shows how to consume from Kinesis streams
\ No newline at end of file
[11/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/d36d6b6c
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/d36d6b6c
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/d36d6b6c
Branch: refs/heads/master
Commit: d36d6b6c29e598b968c10b3311ed5a4b8cf23b98
Parents: e91223d 5ba543f
Author: Jagadish <jv...@linkedin.com>
Authored: Fri Oct 19 14:13:19 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Fri Oct 19 14:13:19 2018 -0700
----------------------------------------------------------------------
...ncing-the-release-of-apache-samza--0.13.1.md | 3 +-
...ncing-the-release-of-apache-samza--0.14.0.md | 5 +-
docs/_blog/2018-09-26-stangeloop-recap.md | 3 +-
docs/_meetups/aug-2016.md | 2 +
docs/_meetups/aug-2017.md | 1 +
docs/_meetups/dec-2017.md | 2 +
docs/_meetups/feb-2015.md | 4 +-
docs/_meetups/feb-2016.md | 1 +
docs/_meetups/feb-2017.md | 6 +-
docs/_meetups/july-2018.md | 3 +-
docs/_meetups/jun-2016.md | 2 +
docs/_meetups/june-2018.md | 1 +
docs/_meetups/mar-2018.md | 3 +-
docs/_meetups/may-2017.md | 5 +-
docs/_meetups/nov-2016.md | 5 +-
docs/_meetups/nov-2017.md | 1 +
docs/_meetups/sep-2017.md | 3 +
.../azure/eventhub_send_methods.png | Bin 52427 -> 0 bytes
.../documentation/versioned/azure/eventhubs.md | 201 -------------------
.../versioned/connectors/eventhubs.md | 121 ++++++++++-
docs/learn/tutorials/versioned/index.md | 4 +-
.../versioned/samza-event-hubs-standalone.md | 127 ++++++++++++
.../consumer/ITestEventHubSystemConsumer.java | 2 +-
.../producer/ITestEventHubSystemProducer.java | 2 +-
.../descriptors/RemoteTableDescriptor.java | 15 +-
.../descriptors/TestRemoteTableDescriptor.java | 1 +
.../TestInMemoryTableDescriptor.java | 7 +
.../descriptors/TestRocksDbTableDescriptor.java | 6 +-
28 files changed, 300 insertions(+), 236 deletions(-)
----------------------------------------------------------------------
[26/50] samza git commit: fix deployment models
Posted by ja...@apache.org.
fix deployment models
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/424b042d
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/424b042d
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/424b042d
Branch: refs/heads/master
Commit: 424b042dd4e394c4a80ddfdfa4905519a87b3038
Parents: 456e697
Author: Jagadish <jv...@linkedin.com>
Authored: Wed Oct 24 14:34:03 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Wed Oct 24 14:34:03 2018 -0700
----------------------------------------------------------------------
.../versioned/deployment/deployment-model.md | 36 +++++++-------------
docs/learn/documentation/versioned/index.html | 2 +-
.../task/SingleContainerGrouperFactory.java | 13 +++++--
.../standalone/PassthroughJobCoordinator.java | 9 ++++-
.../EndOfStreamIntegrationTest.java | 2 +-
.../WatermarkIntegrationTest.java | 2 +-
6 files changed, 35 insertions(+), 29 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/424b042d/docs/learn/documentation/versioned/deployment/deployment-model.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/deployment/deployment-model.md b/docs/learn/documentation/versioned/deployment/deployment-model.md
index 9192278..81de2eb 100644
--- a/docs/learn/documentation/versioned/deployment/deployment-model.md
+++ b/docs/learn/documentation/versioned/deployment/deployment-model.md
@@ -19,40 +19,30 @@ title: Deployment model
limitations under the License.
-->
-# Overview
-One unique thing about Samza is that it provides multiple ways to deploy an application. Each deployment model comes with its own benefits, so you have flexibility in being able to choose which model best fits your needs. Samza supports “write once, run anywhere”, so application logic is the same regardless of the deployment model that you choose.
+### Overview
+One unique thing about Samza is that it provides multiple ways to deploy an application. Each deployment model comes with its own benefits, so you have flexibility in choosing which model best fits your needs. Samza supports “write once, run anywhere”, so application logic is the same regardless of where you choose to deploy your application.
-## YARN
-Apache YARN is a technology that manages resources, deploys applications, and monitors applications for a cluster of machines. Samza submits an application to YARN, and YARN assigns resources from across the cluster to that application. Multiple applications can run on a single YARN cluster.
+### Running Samza on YARN
+Samza integrates with Apache YARN for running stream-processing as a managed service. Samza leverages YARN for multi-tenancy, resource-management, isolation and deployment for your applications. In this mode, you write your Samza application and submit it to be scheduled on a YARN cluster. You also specify its resource requirements - the number of containers needed, number of cores and memory per-container. Samza then works with YARN to provision resources for your application and run it across a cluster of machines. It also handles failures of individual instances and restarts them.
-* Provides central cluster management
-* Each application has an associated application master in YARN to coordinate processing containers
-* Enforces CPU and memory limits
-* Supports multi-tenancy for applications
-* A Samza application is run directly as its own set of processes
-* Automatically restarts containers that have failed
-* Provides centrally managed tools and dashboards
+When multiple applications share the same YARN cluster, they need to be isolated from each other. For this purpose, Samza works with YARN to enforce cpu and memory limits. Any application that uses more than its requested share of memory or cpu is terminated - thereby, enabling multi-tenancy. Just like you would for any YARN-based application, you can use YARN's web UI to manage your Samza jobs, view their logs etc.
-## Standalone
+### Running Samza in standalone mode
-In standalone mode, a Samza application is a library embedded into another application (similar to Kafka Streams). This means that an application owner can control the full lifecycle of the application. Samza will do the coordination between processing containers to ensure that processing is balanced and failures are handled.
+Often you want to embed Samza as a component in a larger application. To enable this, Samza supports a standalone mode of operation. In this mode, Samza can be used like a library within your application. This is very similar to Kafka Streams and offers greater control over the application life-cycle. You can increase capacity by spinning up multiple instances. The instances will dynamically coordinate among themselves to distribute work. If any instance fails, the tasks running on it will be re-assigned to the remaining ones. By default, Samza uses Zookeeper for coordination across instances. The coordination logic by itself is pluggable.
-* Application owner is free to control cluster management, CPU and memory limits, and multi-tenancy
-* Container coordination is done by Zookeeper out of the box, and container coordination can be extended to be done by a technology other than Zookeeper
-* If containers fail, then partitions will be rebalanced across remaining containers
-* Samza logic can run within the same process as non-Samza logic
-* Application owner can run tools and dashboards wherever the application is deployed
+This mode allows you to run Samza with any cluster-manager of your choice - including Kubernetes, Marathon or on any hosting environment. You are free to control memory-limits, multi-tenancy for your application on your own - since Samza now acts as a light-weight library used by your application.
-# Choosing a deployment model
+### Choosing a deployment model
-Here are some guidelines when choosing your deployment model.
+A common question that we get asked is - "Where should I run my Samza application?". Here are some guidelines when choosing your deployment model. Since your application logic does not change, it is easy to port from one deployment model to the other.
-* Would you like your Samza application to be embedded as a component of a larger application?
+* Would you like Samza to be embedded as a component of a larger application?
* If so, then you should use standalone.
* Would you like to have out-of-the-box resource management (e.g. CPU/memory limits, restarts on failures)?
* If so, then you should use YARN.
-* Would you like to have the freedom to deploy and run your application anywhere?
+* Would you like to run your application on any other cluster manager - eg: Kubernetes?
* If so, then you should use standalone.
* Would you like to run centrally-managed tools and dashboards?
* If so, then you should use YARN.
- * Note: You can still have tools and dashboards when using standalone, but you will need to run them yourself wherever you have actually deployed your application.
+ * Note: You can still have tools and dashboards when using standalone, but you will need to run them yourself wherever your application is deployed.
http://git-wip-us.apache.org/repos/asf/samza/blob/424b042d/docs/learn/documentation/versioned/index.html
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/index.html b/docs/learn/documentation/versioned/index.html
index 50bfd2d..893e428 100644
--- a/docs/learn/documentation/versioned/index.html
+++ b/docs/learn/documentation/versioned/index.html
@@ -21,7 +21,7 @@ title: Documentation
<h4><a href="core-concepts/core-concepts.html">CORE CONCEPTS</a></h4>
<h4><a href="architecture/architecture-overview.html">ARCHITECTURE</a></h4>
-<h4><a href="jobs/configuration.html">CONFIGURATIONS</a></h4>
+<h4><a href="jobs/samza-configurations.html">CONFIGURATIONS</a></h4>
<h4>API</h4>
http://git-wip-us.apache.org/repos/asf/samza/blob/424b042d/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java b/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
index ee962d5..90c031a 100644
--- a/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
+++ b/samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java
@@ -19,6 +19,8 @@
package org.apache.samza.container.grouper.task;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
import org.apache.samza.config.Config;
import org.apache.samza.config.JobConfig;
import org.apache.samza.container.TaskName;
@@ -27,6 +29,7 @@ import org.apache.samza.job.model.TaskModel;
import java.util.Collections;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -41,16 +44,22 @@ class SingleContainerGrouper implements TaskNameGrouper {
private final String containerId;
SingleContainerGrouper(String containerId) {
- this.containerId = containerId;
+ this.containerId = null;
}
@Override
public Set<ContainerModel> group(Set<TaskModel> taskModels) {
+ return group(taskModels, ImmutableList.of(this.containerId));
+ }
+
+ @Override
+ public Set<ContainerModel> group(Set<TaskModel> taskModels, List<String> containersIds) {
+ Preconditions.checkState(containersIds.size() == 1);
Map<TaskName, TaskModel> taskNameTaskModelMap = new HashMap<>();
for (TaskModel taskModel: taskModels) {
taskNameTaskModelMap.put(taskModel.getTaskName(), taskModel);
}
- ContainerModel containerModel = new ContainerModel(containerId, taskNameTaskModelMap);
+ ContainerModel containerModel = new ContainerModel(containersIds.get(0), taskNameTaskModelMap);
return Collections.singleton(containerModel);
}
}
http://git-wip-us.apache.org/repos/asf/samza/blob/424b042d/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
index 737ac3e..bcf2085 100644
--- a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
+++ b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
@@ -18,11 +18,14 @@
*/
package org.apache.samza.standalone;
+import com.google.common.collect.ImmutableMap;
import org.apache.samza.checkpoint.CheckpointManager;
import org.apache.samza.config.ApplicationConfig;
import org.apache.samza.config.Config;
import org.apache.samza.config.ConfigException;
import org.apache.samza.config.JobConfig;
+import org.apache.samza.config.MapConfig;
+import org.apache.samza.config.TaskConfig;
import org.apache.samza.config.TaskConfigJava;
import org.apache.samza.coordinator.JobCoordinator;
import org.apache.samza.coordinator.JobModelManager;
@@ -69,7 +72,7 @@ public class PassthroughJobCoordinator implements JobCoordinator {
public PassthroughJobCoordinator(Config config) {
this.processorId = createProcessorId(config);
- this.config = config;
+ this.config = new MapConfig(config, ImmutableMap.of(TaskConfig.GROUPER_FACTORY(), "org.apache.samza.container.grouper.task.SingleContainerGrouperFactory"));
}
@Override
@@ -119,7 +122,11 @@ public class PassthroughJobCoordinator implements JobCoordinator {
SystemAdmins systemAdmins = new SystemAdmins(config);
StreamMetadataCache streamMetadataCache = new StreamMetadataCache(systemAdmins, 5000, SystemClock.instance());
systemAdmins.start();
+<<<<<<< Updated upstream
String containerId = Integer.toString(config.getInt(JobConfig.PROCESSOR_ID()));
+=======
+ String containerId = this.processorId;
+>>>>>>> Stashed changes
/** TODO:
Locality Manager seems to be required in JC for reading locality info and grouping tasks intelligently and also,
http://git-wip-us.apache.org/repos/asf/samza/blob/424b042d/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
----------------------------------------------------------------------
diff --git a/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java b/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
index 6f381e2..f967672 100644
--- a/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
+++ b/samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
@@ -78,7 +78,7 @@ public class EndOfStreamIntegrationTest extends AbstractIntegrationTestHarness {
configs.put("streams.PageView.partitionCount", String.valueOf(partitionCount));
configs.put(JobConfig.JOB_NAME(), "test-eos-job");
- configs.put(JobConfig.PROCESSOR_ID(), "1");
+ //configs.put(JobConfig.PROCESSOR_ID(), "1");
configs.put(JobCoordinatorConfig.JOB_COORDINATOR_FACTORY, PassthroughJobCoordinatorFactory.class.getName());
configs.put(TaskConfig.GROUPER_FACTORY(), SingleContainerGrouperFactory.class.getName());
http://git-wip-us.apache.org/repos/asf/samza/blob/424b042d/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
----------------------------------------------------------------------
diff --git a/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java b/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
index 74c32b4..088278d 100644
--- a/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
+++ b/samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
@@ -131,7 +131,7 @@ public class WatermarkIntegrationTest extends AbstractIntegrationTestHarness {
configs.put("streams.PageView.partitionCount", String.valueOf(PARTITION_COUNT));
configs.put(JobConfig.JOB_NAME(), "test-watermark-job");
- configs.put(JobConfig.PROCESSOR_ID(), "1");
+ //configs.put(JobConfig.PROCESSOR_ID(), "hello");
configs.put(JobCoordinatorConfig.JOB_COORDINATOR_FACTORY, PassthroughJobCoordinatorFactory.class.getName());
configs.put(TaskConfig.GROUPER_FACTORY(), SingleContainerGrouperFactory.class.getName());
[46/50] samza git commit: Clean up open-source documentation for
Samza SQL
Posted by ja...@apache.org.
Clean up open-source documentation for Samza SQL
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/072cff60
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/072cff60
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/072cff60
Branch: refs/heads/master
Commit: 072cff600b41aaac564aefebdea2b541c9f59277
Parents: 65f58bf
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Nov 26 14:18:01 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Nov 26 14:18:01 2018 -0800
----------------------------------------------------------------------
.../documentation/versioned/api/samza-sql.md | 103 +++++--------------
1 file changed, 23 insertions(+), 80 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/072cff60/docs/learn/documentation/versioned/api/samza-sql.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/api/samza-sql.md b/docs/learn/documentation/versioned/api/samza-sql.md
index 7412f6c..0b488b3 100644
--- a/docs/learn/documentation/versioned/api/samza-sql.md
+++ b/docs/learn/documentation/versioned/api/samza-sql.md
@@ -20,127 +20,66 @@ title: Samza SQL
-->
-# Overview
-Samza SQL allows users to write Stream processing application by just writing a SQL query. SQL query is translated to a Samza job using the high level API, which can then be run on all the environments (e.g. Yarn, standalone, etc..) that Samza currently supports.
+### Overview
+Samza SQL allows you to define your stream processing logic declaratively as a a SQL query.
+This allows you to create streaming pipelines without Java code or configuration unless you require user-defined functions (UDFs). Support for SQL internally uses Apache Calcite, which provides SQL parsing and query planning. The query is automatically translated to Samza's high level API and runs on Samza's execution engine.
-Samza SQL was created with following principles:
+You can run Samza SQL locally on your machine or on a YARN cluster.
-1. Users of Samza SQL should be able to create stream processing apps without needing to write Java code unless they require UDFs.
-2. No configs needed by users to develop and execute a Samza SQL job.
-3. Samza SQL should support all the runtimes and systems that Samza supports.
+### Running Samza SQL on your local machine
+The [Samza SQL console](https://samza.apache.org/learn/tutorials/0.14/samza-tools.html) allows you to experiment with Samza SQL locally on your machine.
-Samza SQL uses Apache Calcite to provide the SQL interface. Apache Calcite is a popular SQL engine used by wide variety of big data processing systems (Beam, Flink, Storm etc..)
+#### Setup Kafka
+Follow the instructions from the [Kafka quickstart](http://kafka.apache.org/quickstart) to start the zookeeper and Kafka server.
-# How to use Samza SQL
-There are couple of ways to use Samza SQL:
-
-* Run Samza SQL on your local machine.
-* Run Samza SQL on YARN.
-
-# Running Samza SQL on your local machine
-Samza SQL console tool documented [here](https://samza.apache.org/learn/tutorials/0.14/samza-tools.html) uses Samza standalone to run Samza SQL on your local machine. This is the quickest way to play with Samza SQL. Please follow the instructions [here](https://samza.apache.org/learn/tutorials/0.14/samza-tools.html) to get access to the Samza tools on your machine.
-
-## Start the Kafka server
-Please follow the instructions from the [Kafka quickstart](http://kafka.apache.org/quickstart) to start the zookeeper and Kafka server.
-
-## Create ProfileChangeStream Kafka topic
-The below sql statements require a topic named ProfileChangeStream to be created on the Kafka broker. You can follow the instructions in the Kafka quick start guide to create a topic named “ProfileChangeStream”.
+Let us create a Kafka topic named “ProfileChangeStream” for this demo.
```bash
>./deploy/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic ProfileChangeStream
```
-## Generate events into ProfileChangeStream topic
-Use generate-kafka-events from Samza tools to generate events into the ProfileChangeStream
+Download the Samza tools package from [here](https://samza.apache.org/learn/tutorials/0.14/samza-tools.html) and use the `generate-kafka-events` script populate the stream with sample data.
```bash
> cd samza-tools-<version>
> ./scripts/generate-kafka-events.sh -t ProfileChangeStream -e ProfileChange
```
-## Using Samza SQL Console to run Samza sql on your local machine
+#### Using the Samza SQL Console
-Below are some of the sql queries that you can execute using the samza-sql-console tool from Samza tools package.
-This command just prints out all the events in the Kafka topic ProfileChangeStream into console output as a json serialized payload.
+The simplest SQL query is to read all events from a Kafka topic `ProfileChangeStream` and print them to the console.
```bash
> ./scripts/samza-sql-console.sh --sql "insert into log.consoleoutput select * from kafka.ProfileChangeStream"
```
-This command prints out the fields that are selected into the console output as a json serialized payload.
+Next, let us project a few fields from the input stream.
```bash
> ./scripts/samza-sql-console.sh --sql "insert into log.consoleoutput select Name, OldCompany, NewCompany from kafka.ProfileChangeStream"
```
-
-This command showcases the RegexMatch udf and filtering capabilities.
+You can also filter messages in the input stream based on some predicate. In this example, we filter profiles currently working at LinkedIn, whose previous employer matches the regex `.*soft`. The function `RegexMatch(regex, company)` is an example of
+a UDF that defines a predicate.
```bash
> ./scripts/samza-sql-console.sh --sql "insert into log.consoleoutput select Name as __key__, Name, NewCompany, RegexMatch('.*soft', OldCompany) from kafka.ProfileChangeStream where NewCompany = 'LinkedIn'"
```
-Note: Samza sql console right now doesn’t support queries that need state, for e.g. Stream-Table join, GroupBy and Stream-Stream joins.
-
-
-
-# Running Samza SQL on YARN
-The [hello-samza](https://github.com/apache/samza-hello-samza) project is an example project designed to help you run your first Samza application. It has examples of applications using the Low Level Task API, High Level Streams API as well as Samza SQL.
+### Running Samza SQL on YARN
+The [hello-samza](https://github.com/apache/samza-hello-samza) project has examples to get started with Samza on YARN. You can define your SQL query in a [configuration file](https://github.com/apache/samza-hello-samza/blob/master/src/main/config/pageview-filter-sql.properties) and submit it to a YARN cluster.
-This tutorial demonstrates a simple Samza application that uses SQL to perform stream processing.
-
-## Get the hello-samza code and start the grid
-Please follow the instructions from hello-samza-high-level-yarn on how to build the hello-samza repository and start the yarn grid.
-
-## Create the topic and generate Kafka events
-Please follow the steps in the section “Create ProfileChangeStream Kafka topic” and “Generate events into ProfileChangeStream topic” above.
-
-Build a Samza Application package
-Before you can run a Samza application, you need to build a package for it. Please follow the instructions from hello-samza-high-level-yarn on how to build the hello-samza application package.
-
-## Run a Samza Application
-After you’ve built your Samza package, you can start the app on the grid using the run-app.sh script.
```bash
> ./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties
```
-The app executes the following SQL command :
-
-
-```sql
-insert into kafka.NewLinkedInEmployees select Name from ProfileChangeStream where NewCompany = 'LinkedIn'
-```
-
-
-This SQL performs the following:
-
-* Consumes the Kafka topic ProfileChangeStream which contains the avro serialized ProfileChangeEvent(s)
-* Deserializes the events and filters out only the profile change events where NewCompany = ‘LinkedIn’ i.e. Members who have moved to LinkedIn.
-* Writes the Avro serialized event that contains the Id and Name of those profiles to Kafka topic NewLinkedInEmployees.
-
-Give the job a minute to startup, and then tail the Kafka topic:
-```bash
-> ./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic NewLinkedInEmployees
-```
-
-Congratulations! You’ve now setup a local grid that includes YARN, Kafka, and ZooKeeper, and run a Samza SQL application on it.
-## Shutdown and cleanup
-To shutdown the app, use the same run-app.sh script with an extra –operation=kill argument
-
-```bash
-> ./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties --operation=kill
-```
-
-Please follow the instructions from Hello Samza High Level API - YARN Deployment on how to shutdown and cleanup the app.
-
-
-# SQL Grammar
-Following BNF grammar is based on Apache Calicite’s SQL parser. It is the subset of the capabilities that Calcite supports.
+### SQL Grammar
+Samza SQL's grammar is a subset of capabilities supported by Calcite's SQL parser.
```
statement:
@@ -188,3 +127,7 @@ values:
VALUES expression [, expression ]*
```
+### Known Limitations
+
+Samza SQL only supports simple stateless queries including selections and projections. We are actively working on supporting stateful operations such as aggregations, windows and joins.
+
[34/50] samza git commit: Cleanup the EventHubs connector section.
Use System/Stream Descriptors
Posted by ja...@apache.org.
Cleanup the EventHubs connector section. Use System/Stream Descriptors
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/3273e960
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/3273e960
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/3273e960
Branch: refs/heads/master
Commit: 3273e960ed35671ab04f51a89c826714b12fecea
Parents: c9596b2
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 29 16:28:14 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 29 16:28:14 2018 -0700
----------------------------------------------------------------------
.../versioned/connectors/eventhubs.md | 209 +++++++------------
.../documentation/versioned/connectors/kafka.md | 17 +-
.../documentation/versioned/jobs/logging.md | 2 +
3 files changed, 80 insertions(+), 148 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/3273e960/docs/learn/documentation/versioned/connectors/eventhubs.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/eventhubs.md b/docs/learn/documentation/versioned/connectors/eventhubs.md
index 16120f3..0be288b 100644
--- a/docs/learn/documentation/versioned/connectors/eventhubs.md
+++ b/docs/learn/documentation/versioned/connectors/eventhubs.md
@@ -19,135 +19,59 @@ title: Event Hubs Connector
limitations under the License.
-->
-## Overview
+## EventHubs I/O: QuickStart
-The Samza Event Hubs connector provides access to [Azure Event Hubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features), Microsoft’s data streaming service on Azure. An event hub is similar to a Kafka topic and can have multiple partitions with producers and consumers. Each message produced or consumed from an event hub is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data).
+The Samza EventHubs connector provides access to [Azure EventHubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features), Microsoft’s data streaming service on Azure. An eventhub is similar to a Kafka topic and can have multiple partitions with producers and consumers. Each message produced or consumed from an event hub is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data).
-You may find an [example](../../../tutorials/versioned/samza-event-hubs-standalone.md) using this connector in the [hello-samza](https://github.com/apache/samza-hello-samza) project.
+The [hello-samza](https://github.com/apache/samza-hello-samza) project includes an [example](../../../tutorials/versioned/samza-event-hubs-standalone.md) of reading and writing to EventHubs.
-## Consuming from Event Hubs
+### Concepts
-Samza’s [EventHubSystemConsumer](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubSystemConsumer.java) wraps the EventData into an [EventHubIncomingMessageEnvelope](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubIncomingMessageEnvelope.java). Samza's Event Hubs consumer wraps each message from Event Hubs into an EventHubMessageEnvelope. The envelope has two fields of interest - the key, which is set to the event's String partition key and the message, which is set to the actual data in the event.
-
-You can describe your Samza jobs to process data from Azure Event Hubs. To set Samza to consume from Event Hubs streams:
+####EventHubsSystemDescriptor
+Samza refers to any IO source (eg: Kafka) it interacts with as a _system_, whose properties are set using a corresponding `SystemDescriptor`. The `EventHubsSystemDescriptor` allows you to configure various properties for the `EventHubsClient` used by Samza.
{% highlight java %}
- 1 public void describe(StreamApplicationDescriptor appDescriptor) {
- 2 // Define your system here
- 3 EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs");
- 4
- 5 // Choose your serializer/deserializer for the EventData payload
- 6 StringSerde serde = new StringSerde();
- 7
- 8 // Define the input descriptors with respective descriptors
- 9 EventHubsInputDescriptor<KV<String, String>> inputDescriptor =
-10 systemDescriptor.getInputDescriptor(INPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_INPUT_ENTITY, serde)
-11 .withSasKeyName(EVENTHUBS_SAS_KEY_NAME)
-12 .withSasKey(EVENTHUBS_SAS_KEY_TOKEN);
-13
-14 // Define the input streams with descriptors
-15 MessageStream<KV<String, String>> eventhubInput = appDescriptor.getInputStream(inputDescriptor);
-16
-17 //...
-18 }
-{% endhighlight %}
-
-In the code snippet above, we create the input and output streams that can consume and produce from the configured Event Hubs entities.
-
-1. Line 3: A `EventHubsSystemDescriptor` is created with the name "eventhubs". You may set different system descriptors here.
-2. Line 6: Event Hubs messages are consumed as key value pairs. The [serde](../../documentation/versioned/container/serialization.html) is defined for the value of the incoming payload of the Event Hubs' EventData. You may use any of the serdes that samza ships with out of the box or define your own.
-The serde for the key is not set since it will always the String from the EventData [partitionKey](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data._system_properties.getpartitionkey?view=azure-java-stable#com_microsoft_azure_eventhubs__event_data__system_properties_getPartitionKey__).
-3. Line 8-12: An `EventHubsInputDescriptor` is created with the required descriptors to gain access of the Event Hubs entity (`STREAM_ID`, `EVENTHUBS_NAMESPACE`, `EVENTHUBS_ENTITY`, `EVENTHUBS_SAS_KEY_NAME`, `EVENTHUBS_SAS_KEY_TOKEN`).
-These must be set to the credentials of the entities you wish to connect to.
-4. Line 15: creates an `InputStream` with the previously defined `EventHubsInputDescriptor`.
-
-Alternatively, you can set these properties in the `.properties` file of the application.
-Note: the keys set in the `.properties` file will override the ones set in code with descriptors.
-Refer to the [Event Hubs configuration reference](../../documentation/versioned/jobs/samza-configurations.html#eventhubs) for the complete list of configurations.
-
-{% highlight jproperties %}
-# define an event hub system factory with your identifier. eg: eh-system
-systems.eh-system.samza.factory=org.apache.samza.system.eventhub.EventHubSystemFactory
-
-# define your streams
-systems.eh-system.stream.list=eh-input-stream
-streams.eh-stream.samza.system=eh-system
-
-# define required properties for your streams
-streams.eh-input-stream.eventhubs.namespace=YOUR-STREAM-NAMESPACE
-streams.eh-input-stream.eventhubs.entitypath=YOUR-ENTITY-NAME
-streams.eh-input-stream.eventhubs.sas.keyname=YOUR-SAS-KEY-NAME
-streams.eh-input-stream.eventhubs.sas.token=YOUR-SAS-KEY-TOKEN
+ 1 EventHubsSystemDescriptor eventHubsSystemDescriptor = new EventHubsSystemDescriptor("eventhubs").withNumClientThreads(5);
{% endhighlight %}
-It is required to provide values for YOUR-STREAM-NAMESPACE, YOUR-ENTITY-NAME, YOUR-SAS-KEY-NAME, YOUR-SAS-KEY-TOKEN to read or write to the stream.
-
-## Producing to Event Hubs
+####EventHubsInputDescriptor
-Each [OutgoingMessageEnvelope](https://samza.apache.org/learn/documentation/latest/api/javadocs/org/apache/samza/system/OutgoingMessageEnvelope.html) from Samza is converted into an [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data) instance whose body is set to the message in the envelope. Additionally, the key and the produce timestamp are set as properties in the EventData before sending it to Event Hubs.
-Similarly, you can also configure your Samza job to write to Event Hubs. Follow the same descriptors defined in the Consuming from Event Hubs section to write to Event Hubs:
+The EventHubsInputDescriptor allows you to specify the properties of each EventHubs stream your application should read from. For each of your input streams, you should create a corresponding instance of EventHubsInputDescriptor by providing a topic-name and a serializer.
{% highlight java %}
- 1 public void describe(StreamApplicationDescriptor appDescriptor) {
- 2 // Define your system here
- 3 EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs");
- 4
- 5 // Choose your serializer/deserializer for the EventData payload
- 6 StringSerde serde = new StringSerde();
- 7
- 8 // Define the input and output descriptors with respective descriptors
- 9 EventHubsOutputDescriptor<KV<String, String>> outputDescriptor =
-10 systemDescriptor.getOutputDescriptor(OUTPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_OUTPUT_ENTITY, serde)
-11 .withSasKeyName(EVENTHUBS_SAS_KEY_NAME)
-12 .withSasKey(EVENTHUBS_SAS_KEY_TOKEN);
-13
-14 // Define the output streams with descriptors
-15 OutputStream<KV<String, String>> eventhubOutput = appDescriptor.getOutputStream(outputDescriptor);
-16
-17 //...
-18 }
+ EventHubsInputDescriptor<KV<String, String>> inputDescriptor =
+ systemDescriptor.getInputDescriptor(streamId, "eventhubs-namespace", "eventhubs-name", new StringSerde())
+ .withSasKeyName("secretkey")
+ .withSasKey("sasToken-123")
+ .withConsumerGroup("$notdefault");
{% endhighlight %}
-In the code snippet above, we create the input and output streams that can consume and produce from the configured Event Hubs entities.
+By default, messages are sent and received as byte arrays. Samza then de-serializes them to typed objects using your provided Serde. For example, the above uses a `StringSerde` to de-serialize messages.
-1. Line 3: A `EventHubsSystemDescriptor` is created with the name "eventhubs". You may set different system descriptors here.
-2. Line 6: Event Hubs messages are produced as key value pairs. The [serde](../../documentation/versioned/container/serialization.html) is defined for the value of the payload of the outgoing Event Hubs' EventData. You may use any of the serdes that samza ships with out of the box or define your own.
-The serde for the key is not set since it will always the String from the EventData [partitionKey](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data._system_properties.getpartitionkey?view=azure-java-stable#com_microsoft_azure_eventhubs__event_data__system_properties_getPartitionKey__).
-3. Line 9-12: An `EventHubsOutputDescriptor` is created with the required descriptors to gain access of the Event Hubs entity (`STREAM_ID`, `EVENTHUBS_NAMESPACE`, `EVENTHUBS_ENTITY`, `EVENTHUBS_SAS_KEY_NAME`, `EVENTHUBS_SAS_KEY_TOKEN`).
-These must be set to the credentials of the entities you wish to connect to.
-4. Line 15: creates an `OutputStream` with the previously defined `EventHubsOutputDescriptor`.
-Alternatively, you can set these properties in the `.properties` file of the application.
-Note: the keys set in the `.properties` file will override the ones set in code with descriptors.
-Refer to the [Event Hubs configuration reference](../../documentation/versioned/jobs/samza-configurations.html#eventhubs) for the complete list of configurations.
+####EventHubsOutputDescriptor
-{% highlight jproperties %}
-# define an event hub system factory with your identifier. eg: eh-system
-systems.eh-system.samza.factory=org.apache.samza.system.eventhub.EventHubSystemFactory
+Similarly, the `EventHubsOutputDescriptor` allows you to specify the output streams for your application. For each output stream you write to in EventHubs, you should create an instance of `EventHubsOutputDescriptor`.
-# define your streams
-systems.eh-system.stream.list=eh-output-stream
-streams.eh-stream.samza.system=eh-system
-
-streams.eh-output-stream.eventhubs.namespace=YOUR-STREAM-NAMESPACE
-streams.eh-output-stream.eventhubs.entitypath=YOUR-ENTITY-NAME
-streams.eh-output-stream.eventhubs.sas.keyname=YOUR-SAS-KEY-NAME
-streams.eh-output-stream.eventhubs.sas.token=YOUR-SAS-KEY-TOKEN
+{% highlight java %}
+ EventHubsOutputDescriptor<KV<String, String>> outputDescriptor =
+ systemDescriptor.getOutputDescriptor(OUTPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_OUTPUT_ENTITY, new StringSerde();)
+ .withSasKeyName(..)
+ .withSasKey(..);
{% endhighlight %}
-Then you can consume and produce a message to Event Hubs in your code as below:
+####Security Model
+Each EventHubs stream is scoped to a container called a _namespace_, which uniquely identifies an EventHubs in a region. EventHubs's [security model](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-authentication-and-security-model-overview) is based on Shared Access Signatures(SAS).
+Hence, you should also provide your SAS keys and tokens to access the stream. You can generate your SAS tokens using the
-{% highlight java %}
-MessageStream<KV<String, String>> eventhubInput = appDescriptor.getInputStream(inputDescriptor);
-OutputStream<KV<String, String>> eventhubOutput = appDescriptor.getOutputStream(outputDescriptor);
-eventhubInput.sendTo(eventhubOutput)
-{% endhighlight %}
+####Data Model
+Each event produced and consumed from an EventHubs stream is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data), which wraps a byte-array payload. When producing to EventHubs, Samza serializes your object into an `EventData` payload before sending it over the wire. Likewise, when consuming messages from EventHubs, messages are de-serialized into typed objects using the provided Serde.
-## Advanced configuration
+## Configuration
###Producer partitioning
-The partition.method property determines how outgoing messages are partitioned. Valid values for this config are EVENT\_HUB\_HASHING, PARTITION\_KEY\_AS_PARTITION or ROUND\_ROBIN.
+You can use `#withPartitioningMethod` to control how outgoing messages are partitioned. The following partitioning schemes are supported:
1. EVENT\_HUB\_HASHING: By default, Samza computes the partition for an outgoing message based on the hash of its partition-key. This ensures that events with the same key are sent to the same partition. If this option is chosen, the partition key should be a string. If the partition key is not set, the key in the message is used for partitioning.
@@ -155,22 +79,16 @@ The partition.method property determines how outgoing messages are partitioned.
3. ROUND\_ROBIN: In this method, outgoing messages are distributed in a round-robin across all partitions. The key and the partition key in the message are ignored.
-##### Using descriptors
{% highlight java %}
EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs")
.withPartitioningMethod(PartitioningMethod.EVENT_HUB_HASHING);
{% endhighlight %}
-##### Using config properties
-{% highlight jproperties %}
-systems.eh-system.partition.method = EVENT_HUB_HASHING
-{% endhighlight %}
### Consumer groups
-Event Hubs supports the notion of [consumer groups](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups) which enable multiple applications to have their own view of the event stream. Each partition is exclusively consumed by one consumer in the consumer group. Each event hub stream has a pre-defined consumer group named $Default. You can define your own consumer group for your job by configuring an Event Hubs.consumer.group
+Event Hubs supports the notion of [consumer groups](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups) which enable multiple applications to have their own view of the event stream. Each partition is exclusively consumed by one consumer in the group. Each event hub stream has a pre-defined consumer group named $Default. You can define your own consumer group for your job using `withConsumerGroup`.
-##### Using descriptors
{% highlight java %}
EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs");
EventHubsInputDescriptor<KV<String, String>> inputDescriptor =
@@ -178,41 +96,54 @@ EventHubsInputDescriptor<KV<String, String>> inputDescriptor =
.withConsumerGroup("my-group");
{% endhighlight %}
-##### Using config properties
-{% highlight jproperties %}
-streams.eh-input-stream.eventhubs.consumer.group = my-group
-{% endhighlight %}
-### Serde
+### Consumer buffer size
-By default, the messages from Event Hubs are sent and received as byte arrays. You can configure a serializer and deserializer for your message by setting a value for msg.serde for your stream.
+When the consumer reads a message from EventHubs, it appends them to a shared producer-consumer queue corresponding to its partition. This config determines the per-partition queue size. Setting a higher value for this config typically achieves a higher throughput at the expense of increased on-heap memory.
-##### Using descriptors
{% highlight java %}
-JsonSerde inputSerde = new JsonSerde();
-EventHubsInputDescriptor<KV<String, String>> inputDescriptor =
- systemDescriptor.getInputDescriptor(INPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_INPUT_ENTITY, inputSerde);
-JsonSerde outputSerde = new JsonSerde();
-EventHubsOutputDescriptor<KV<String, String>> outputDescriptor =
- systemDescriptor.getOutputDescriptor(OUTPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_OUTPUT_ENTITY, outputSerde);
+ EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs")
+ .withReceiveQueueSize(10);
{% endhighlight %}
-##### Using config properties
-{% highlight jproperties %}
-streams.input0.samza.msg.serde = json
-streams.output0.samza.msg.serde = json
-{% endhighlight %}
+### Code walkthrough
-### Consumer buffer size
+In this section, we will walk through a simple pipeline that reads from one EventHubs stream and copies each message to another output stream.
-When the consumer reads a message from event hubs, it appends them to a shared producer-consumer queue corresponding to its partition. This config determines the per-partition queue size. Setting a higher value for this config typically achieves a higher throughput at the expense of increased on-heap memory.
-##### Using descriptors
{% highlight java %}
- EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs")
- .withReceiveQueueSize(10);
+1 EventHubsSystemDescriptor systemDescriptor = new EventHubsSystemDescriptor("eventhubs").withNumClientThreads(5);
+
+2 EventHubsInputDescriptor<KV<String, String>> inputDescriptor =
+ systemDescriptor.getInputDescriptor(INPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_INPUT_ENTITY, new StringSerde())
+ .withSasKeyName(..)
+ .withSasKey(..));
+
+3 EventHubsOutputDescriptor<KV<String, String>> outputDescriptor =
+ systemDescriptor.getOutputDescriptor(OUTPUT_STREAM_ID, EVENTHUBS_NAMESPACE, EVENTHUBS_OUTPUT_ENTITY, serde)
+ .withSasKeyName(..))
+ .withSasKey(..));
+
+4 MessageStream<KV<String, String>> eventhubInput = appDescriptor.getInputStream(inputDescriptor);
+5 OutputStream<KV<String, String>> eventhubOutput = appDescriptor.getOutputStream(outputDescriptor);
+
+ // Define the execution flow with the high-level API
+6 eventhubInput
+7 .map((message) -> {
+8 System.out.println("Received Key: " + message.getKey());
+9 System.out.println("Received Message: " + message.getValue());
+10 return message;
+11 })
+12 .sendTo(eventhubOutput);
{% endhighlight %}
-##### Using config properties
-{% highlight jproperties %}
-systems.eh-system.eventhubs.receive.queue.size = 10
-{% endhighlight %}
\ No newline at end of file
+-Line 1 instantiates an `EventHubsSystemDescriptor` configuring an EventHubsClient with 5 threads. To consume from other input sources like Kafka, you can define their corresponding descriptors.
+
+-Line 2 creates an `EventHubsInputDescriptor` with a String serde for its values. Recall that Samza follows a KV data-model for input messages. In the case of EventHubs, the key is a string which is set to the [partitionKey](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data._system_properties.getpartitionkey?view=azure-java-stable#com_microsoft_azure_eventhubs__event_data__system_properties_getPartitionKey__) in the message. Hence, no separate key serde is required.
+
+-Line 3 creates an `EventHubsOutputDescriptor` to write to an EventHubs stream with the given credentials.
+
+-Line 4 obtains a `MessageStream` from the input descriptor that you can later chain operations on.
+
+-Line 5 creates an `OutputStream` with the previously defined `EventHubsOutputDescriptor` that you can send messages to.
+
+-Line 7-12 define a simple pipeline that copies message from one EventHubs stream to another
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/3273e960/docs/learn/documentation/versioned/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/kafka.md b/docs/learn/documentation/versioned/connectors/kafka.md
index 447bfdc..5b15b91 100644
--- a/docs/learn/documentation/versioned/connectors/kafka.md
+++ b/docs/learn/documentation/versioned/connectors/kafka.md
@@ -105,11 +105,10 @@ The above example configures Samza to ignore checkpointed offsets for `page-view
-### Code walkthrough
+### Code walkthrough: High-level API
-In this section, we walk through a complete example.
+In this section, we walk through a complete example that reads from a Kafka topic, filters a few messages and writes them to another topic.
-#### High-level API
{% highlight java %}
// Define coordinates of the Kafka cluster using the KafkaSystemDescriptor
1 KafkaSystemDescriptor kafkaSystemDescriptor = new KafkaSystemDescriptor("kafka")
@@ -137,14 +136,14 @@ In this section, we walk through a complete example.
{% endhighlight %}
-- Lines 1-3 create a KafkaSystemDescriptor defining the coordinates of our Kafka cluster
+-Lines 1-3 create a KafkaSystemDescriptor defining the coordinates of our Kafka cluster
-- Lines 4-6 defines a KafkaInputDescriptor for our input topic - `page-views`
+-Lines 4-6 define a KafkaInputDescriptor for our input topic - `page-views`
-- Lines 7-9 defines a KafkaOutputDescriptor for our output topic - `filtered-page-views`
+-Lines 7-9 define a KafkaOutputDescriptor for our output topic - `filtered-page-views`
-- Line 9 creates a MessageStream for the input topic so that you can chain operations on it later
+-Line 9 creates a MessageStream for the input topic so that you can chain operations on it later
-- Line 10 creates an OuputStream for the output topic
+-Line 10 creates an OuputStream for the output topic
-- Lines 11-13 define a simple pipeline that reads from the input stream and writes filtered results to the output stream
\ No newline at end of file
+-Lines 11\-13 define a simple pipeline that reads from the input stream and writes filtered results to the output stream
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/3273e960/docs/learn/documentation/versioned/jobs/logging.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/jobs/logging.md b/docs/learn/documentation/versioned/jobs/logging.md
index 8717c28..9dde172 100644
--- a/docs/learn/documentation/versioned/jobs/logging.md
+++ b/docs/learn/documentation/versioned/jobs/logging.md
@@ -163,7 +163,9 @@ Rest all of the system properties will be set exactly like in the case of log4j,
If you are already using log4j and want to upgrade to using log4j2, following are the changes you will need to make in your job:
- Clean your lib directory. This will be rebuilt with new dependency JARs and xml files.
+
- Replace log4j’s dependencies with log4j2’s in your pom.xml/build.gradle as mentioned above. Please ensure that none of log4j’s dependencies remain in pom.xml/build.gradle
+
- Create a log4j2.xml to match your existing log4j.xml file.
- Rebuild your application
[03/50] samza git commit: More edits to the LinkedIn case-study. fix
rendering
Posted by ja...@apache.org.
More edits to the LinkedIn case-study. fix rendering
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/f569a5ce
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/f569a5ce
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/f569a5ce
Branch: refs/heads/master
Commit: f569a5ce67b61955aba03a6c1f0973d9866e84d9
Parents: 7942dbe
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 15 16:32:56 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 15 16:32:56 2018 -0700
----------------------------------------------------------------------
docs/_case-studies/linkedin.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/f569a5ce/docs/_case-studies/linkedin.md
----------------------------------------------------------------------
diff --git a/docs/_case-studies/linkedin.md b/docs/_case-studies/linkedin.md
index 1ecb440..66e4108 100644
--- a/docs/_case-studies/linkedin.md
+++ b/docs/_case-studies/linkedin.md
@@ -44,9 +44,9 @@ The _ATC pipeline_ processors aggregate the output from the _Relevance_ and the
ATC uses several of Samza features:
-**Stateful processing**: The ML models in the relevance module are stored locally in RocksDb and are updated realtime time based on user feedback. <br/>
-**Async APIs and Multi-threading**: Samza’s multi-threading and Async APIs allow ATC to perform remote calls with high throughput. This helps bring down the 90th percentile end-to-end latency for push notifications. <br/>
-**Host affinity**: Samza's incremental checkpointing and host-affinity enable ATC to achieve zero downtime during upgrades and instant recovery during failures. <br/>
+**1.Stateful processing**: The ML models in the relevance module are stored locally in RocksDb and are updated realtime time based on user feedback. <br/><br/>
+**2.Async APIs and Multi-threading**: Samza’s multi-threading and Async APIs allow ATC to perform remote calls with high throughput. This helps bring down the 90th percentile end-to-end latency for push notifications. <br/><br/>
+**3.Host affinity**: Samza's incremental checkpointing and host-affinity enable ATC to achieve zero downtime during upgrades and instant recovery during failures. <br/><br/>
Key Samza Features: *Stateful processing*, *Async API*, *Host affinity*
[32/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/7d4c2ed6
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/7d4c2ed6
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/7d4c2ed6
Branch: refs/heads/master
Commit: 7d4c2ed651d0d1e0325c3539748b7373f954a480
Parents: 2ffedc9 45f2558
Author: Jagadish <jv...@linkedin.com>
Authored: Sat Oct 27 11:18:52 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sat Oct 27 11:18:52 2018 -0700
----------------------------------------------------------------------
----------------------------------------------------------------------
[16/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/50fa9747
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/50fa9747
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/50fa9747
Branch: refs/heads/master
Commit: 50fa974714370f3ae024d38e966ddcbc908a526c
Parents: 937d1de 40aa6a5
Author: Jagadish <jv...@linkedin.com>
Authored: Sun Oct 21 11:49:01 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sun Oct 21 11:49:01 2018 -0700
----------------------------------------------------------------------
----------------------------------------------------------------------
[25/50] samza git commit: Revert changes to PassthroughJC
Posted by ja...@apache.org.
Revert changes to PassthroughJC
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/456e697d
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/456e697d
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/456e697d
Branch: refs/heads/master
Commit: 456e697d8034099f032361268f3768b085395696
Parents: d8336be
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:38:13 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:38:13 2018 -0700
----------------------------------------------------------------------
.../org/apache/samza/standalone/PassthroughJobCoordinator.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/456e697d/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
index 7522106..737ac3e 100644
--- a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
+++ b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
@@ -119,7 +119,7 @@ public class PassthroughJobCoordinator implements JobCoordinator {
SystemAdmins systemAdmins = new SystemAdmins(config);
StreamMetadataCache streamMetadataCache = new StreamMetadataCache(systemAdmins, 5000, SystemClock.instance());
systemAdmins.start();
- String containerId = config.get(JobConfig.PROCESSOR_ID());
+ String containerId = Integer.toString(config.getInt(JobConfig.PROCESSOR_ID()));
/** TODO:
Locality Manager seems to be required in JC for reading locality info and grouping tasks intelligently and also,
[14/50] samza git commit: Cleanup website docs for 1.0 release
Posted by ja...@apache.org.
Cleanup website docs for 1.0 release
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/ce52acf0
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/ce52acf0
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/ce52acf0
Branch: refs/heads/master
Commit: ce52acf0736349e4ebac66a804471d9a16bdbf9d
Parents: 28da01c
Author: Jagadish <jv...@linkedin.com>
Authored: Sun Oct 21 10:50:31 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sun Oct 21 10:50:31 2018 -0700
----------------------------------------------------------------------
docs/_blog/2018-07-19-stream_processing-meetup-summary.md | 8 ++++----
docs/css/main.new.css | 1 -
2 files changed, 4 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/ce52acf0/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
----------------------------------------------------------------------
diff --git a/docs/_blog/2018-07-19-stream_processing-meetup-summary.md b/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
index de3fa06..52c8a54 100644
--- a/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
+++ b/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
@@ -25,14 +25,14 @@ excerpt_separator: <!--more-->
limitations under the License.
-->
-On July 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies.
-The event featured technical deep dives by engineers from LinkedIn and Uber on the latest in the stream processing space...
+On July 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies.
+The event featured technical deep dives by engineers from LinkedIn and Uber on the latest in the stream processing space...
<!--more-->
-July 2018 - On July 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies.
+July 2018 - On July 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies.
The event featured technical deep dives by engineers from LinkedIn and Uber on the latest in the stream processing space. Here is a brief summary
-of the talks that were presented.
+of the talks that were presented.
### [Beam me up Samza: How we built a Samza Runner for Apache Beam](https://youtu.be/o5GaifLoZho)
http://git-wip-us.apache.org/repos/asf/samza/blob/ce52acf0/docs/css/main.new.css
----------------------------------------------------------------------
diff --git a/docs/css/main.new.css b/docs/css/main.new.css
index 558dd29..468e10e 100644
--- a/docs/css/main.new.css
+++ b/docs/css/main.new.css
@@ -90,7 +90,6 @@ a.side-navigation__group-title::after {
border-bottom: none;
}
-
.content a:hover::after,
.section a:hover::after,
.content a.active::after,
[47/50] samza git commit: Clean up standalone docs
Posted by ja...@apache.org.
Clean up standalone docs
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/d6b89882
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/d6b89882
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/d6b89882
Branch: refs/heads/master
Commit: d6b8988215f959348c4b740b9d40064c3599fdbc
Parents: 072cff6
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Nov 27 00:26:06 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Nov 27 00:26:06 2018 -0800
----------------------------------------------------------------------
.../versioned/deployment/standalone.md | 177 ++++---------------
1 file changed, 35 insertions(+), 142 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/d6b89882/docs/learn/documentation/versioned/deployment/standalone.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/deployment/standalone.md b/docs/learn/documentation/versioned/deployment/standalone.md
index 146f9cb..d8bd541 100644
--- a/docs/learn/documentation/versioned/deployment/standalone.md
+++ b/docs/learn/documentation/versioned/deployment/standalone.md
@@ -1,6 +1,6 @@
---
layout: page
-title: Run as embedded library.
+title: Run as embedded library
---
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -20,145 +20,46 @@ title: Run as embedded library.
-->
- [Introduction](#introduction)
-- [User guide](#user-guide)
- - [Setup dependencies](#setup-dependencies)
- - [Configuration](#configuration)
- - [Code sample](#code-sample)
-- [Quick start guide](#quick-start-guide)
- - [Setup zookeeper](#setup-zookeeper)
- - [Setup kafka](#setup-kafka)
- - [Build binaries](#build-binaries)
- - [Deploy binaries](#deploy-binaries)
- - [Inspect results](#inspect-results)
+- [Quick start](#quick-start-guide)
+ - [Installing Zookeeper and Kafka](#setup-zookeeper)
+ - [Building binaries](#build-binaries)
+ - [Running the application](#deploy-binaries)
+ - [Inspecting results](#inspect-results)
- [Coordinator internals](#coordinator-internals)
-#
-# Introduction
+### Introduction
-With Samza 0.13.0, the deployment model of samza jobs has been simplified and decoupled from YARN. _Standalone_ model provides the stream processing capabilities of samza packaged in the form of a library with pluggable coordination. This library model offers an easier integration path and promotes a flexible deployment model for an application. Using the standalone mode, you can leverage Samza processors directly in your application and deploy Samza applications to self-managed clusters.
+Often you want to embed and integrate Samza as a component within a larger application. To enable this, Samza supports a standalone mode of deployment allowing greater control over your application’s life-cycle. In this model, Samza can be used just like any library you import within your Java application. This is identical to using a high-level Kafka consumer to process your streams.
-A standalone application typically is comprised of multiple _stream processors_. A _stream processor_ encapsulates a user defined processing function and is responsible for processing a subset of input topic partitions. A stream processor of a standalone application is uniquely identified by a _processorId_.
+You can increase your application’s capacity by spinning up multiple instances. These instances will then dynamically coordinate with each other and distribute work among themselves. If an instance fails for some reason, the tasks running on it will be re-assigned to the remaining ones. By default, Samza uses Zookeeper for coordination across individual instances. The coordination logic by itself is pluggable and hence, can integrate with other frameworks.
-Samza provides pluggable job _coordinator_ layer to perform leader election and assign work to the stream processors. Standalone supports Zookeeper coordination out of the box and uses it for distributed coordination between the stream processors of standalone application. A processor can become part of a standalone application by setting its app.name(Ex: app.name=group\_1) and joining the group.
+This mode allows you to bring any cluster-manager or hosting-environment of your choice(eg: Kubernetes, Marathon) to run your application. You are also free to control memory-limits, multi-tenancy on your own - since Samza is used as a light-weight library.
-In samza standalone, the input topic partitions are distributed between the available processors dynamically at runtime. In each standalone application, one stream processor will be chosen as a leader initially to mediate the assignment of input topic partitions to the stream processors. If the number of available processors changes(for example, if a processors is shutdown or added), then the leader processor will regenerate the partition assignments and re-distribute it to all the processors.
-On processor group change, the act of re-assigning input topic partitions to the remaining live processors in the group is known as rebalancing the group. On failure of the leader processor of a standalone application, an another stream processor of the standalone application will be chosen as leader.
+### Quick start
-## User guide
-
-Samza standalone is designed to help you to have more control over the deployment of the application. So it is your responsibility to configure and deploy the processors. In case of ZooKeeper coordination, you have to configure the URL for an instance of ZooKeeper.
-
-A stream processor is identified by a unique processorID which is generated by the pluggable ProcessorIdGenerator abstraction. ProcessorId of the stream processor is used with the coordination service. Samza supports UUID based ProcessorIdGenerator out of the box.
-
-The diagram below shows a input topic with three partitions and an standalone application with three processors consuming messages from it.
-
-<img src="/img/versioned/learn/documentation/standalone/standalone-application.jpg" alt="Standalone application" height="550px" width="700px" align="middle">
-
-When a group is first initialized, each stream processor typically starts processing messages from either the earliest or latest offset of the input topic partition. The messages in each partition are sequentially delivered to the user defined processing function. As the stream processor makes progress, it commits the offsets of the messages it has successfully processed. For example, in the figure above, the stream processor position is at offset 7 and its last committed offset is at offset 3.
-
-When a input partition is reassigned to another processor in the group, the initial position is set to the last committed offset. If the processor-1 in the example above suddenly crashed, then the live processor taking over the partition would begin consumption from offset 3. In that case, it would not have to reprocess the messages up to the crashed processor's position of 3.
-
-### Setup dependencies
-
-Add the following samza-standalone maven dependencies to your project.
-
-```xml
-<dependency>
- <groupId>org.apache.samza</groupId>
- <artifactId>samza-kafka_2.11</artifactId>
- <version>1.0</version>
-</dependency>
-<dependency>
- <groupId>org.apache.samza</groupId>
- <artifactId>samza-core_2.11</artifactId>
- <version>1.0</version>
-</dependency>
-<dependency>
- <groupId>org.apache.samza</groupId>
- <artifactId>samza-api</artifactId>
- <version>1.0</version>
-</dependency>
-```
-
-### Configuration
-
-A samza standalone application requires you to define the following mandatory configurations:
-
-```bash
-job.coordinator.factory=org.apache.samza.zk.ZkJobCoordinatorFactory
-job.coordinator.zk.connect=your_zk_connection(for local zookeeper, use localhost:2181)
-task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerIdsFactory
-```
-
-You have to configure the stream processor with the kafka brokers as defined in the following sample(we have assumed that the broker is running on localhost):
-
-```bash
-systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
-systems.kafka.samza.msg.serde=json
-systems.kafka.consumer.zookeeper.connect=localhost:2181
-systems.kafka.producer.bootstrap.servers=localhost:9092
-```
-
-### Code sample
-
-Here's a sample standalone application with app.name set to sample-test. Running this class would launch a stream processor.
-
-```java
-public class PageViewEventExample implements StreamApplication {
-
- public static void main(String[] args) {
- CommandLine cmdLine = new CommandLine();
- OptionSet options = cmdLine.parser().parse(args);
- Config config = cmdLine.loadConfig(options);
-
- ApplicationRunner runner = ApplicationRunners.getApplicationRunner(ApplicationClassUtils.fromConfig(config), config);
- runner.run();
- runner.waitForFinish();
- }
-
- @Override
- public void describe(StreamAppDescriptor appDesc) {
- MessageStream<PageViewEvent> pageViewEvents = null;
- pageViewEvents = appDesc.getInputStream("inputStream", new JsonSerdeV2<>(PageViewEvent.class));
- OutputStream<KV<String, PageViewCount>> pageViewEventPerMemberStream =
- appDesc.getOutputStream("outputStream", new JsonSerdeV2<>(PageViewEvent.class));
- pageViewEvents.sendTo(pageViewEventPerMemberStream);
- }
-}
-```
-
-## Quick start guide
-
-The [Hello-samza](https://github.com/apache/samza-hello-samza/) project contains sample Samza standalone applications. Here are step by step instruction guide to install, build and run a standalone application binaries using the local zookeeper cluster for coordination. Check out the hello-samza project by running the following commands:
+The [Hello-samza](https://github.com/apache/samza-hello-samza/) project includes multiple examples of Samza standalone applications. Let us first check out the repository.
```bash
git clone https://git.apache.org/samza-hello-samza.git hello-samza
cd hello-samza
```
-### Setup Zookeeper
-
-Run the following command to install and start a local zookeeper cluster.
-
-```bash
-./bin/grid install zookeeper
-./bin/grid start zookeeper
-```
-### Setup Kafka
+#### Installing Zookeeper and Kafka
-Run the following command to install and start a local kafka cluster.
+We will use the `./bin/grid` script from the `hello-samza` project to setup up Zookeeper and Kafka locally.
```bash
-./bin/grid install kafka
+./bin/grid start zookeeper
./bin/grid start zookeeper
```
-### Build binaries
-Before you can run the standalone job, you need to build a package for it using the following command.
+#### Building the binaries
+
+Let us now build the `hello-samza` project from its sources.
```bash
mvn clean package
@@ -166,52 +67,44 @@ mkdir -p deploy/samza
tar -xvf ./target/hello-samza-1.0.0-dist.tar.gz -C deploy/samza
```
-### Deploy binaries
+#### Running the application
-To run the sample standalone application [WikipediaZkLocalApplication](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/wikipedia/application/WikipediaZkLocalApplication.java)
+We are ready to run the example application [WikipediaZkLocalApplication](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/wikipedia/application/WikipediaZkLocalApplication.java). This application reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It emits these results to another topic named `wikipedia-stats`.
```bash
-./bin/deploy.sh
./deploy/samza/bin/run-class.sh samza.examples.wikipedia.application.WikipediaZkLocalApplication --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-application-local-runner.properties
```
-### Inspect results
+You can run the above command again to spin up a new instance of your application.
-The standalone application reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It outputs these counts to the local wikipedia-stats kafka topic. To inspect events in output topic, run the following command.
+#### Inspecting results
+
+To inspect events in output topic, run the following command.
```bash
./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats
```
-Events produced to the output topic from the standalone application launched above will be of the following form:
+You should see the output messages emitted to the Kafka topic.
```
{"is-talk":2,"bytes-added":5276,"edits":13,"unique-titles":13}
{"is-bot-edit":1,"is-talk":3,"bytes-added":4211,"edits":30,"unique-titles":30,"is-unpatrolled":1,"is-new":2,"is-minor":7}
```
-# Coordinator internals
-
-A samza application is comprised of multiple stream processors. A processor can become part of a standalone application by setting its app.name(Ex: app.name=group\_1) and joining the group. In samza standalone, the input topic partitions are distributed between the available processors dynamically at runtime. If the number of available processors changes(for example, if some processors are shutdown or added), then the partition assignments will be regenerated and re-distributed to all the processors. One processor will be elected as leader and it will generate the partition assignments and distribute it to the other processors in the group.
-
-To mediate the partition assignments between processors, samza standalone relies upon a coordination service. The main responsibilities of coordination service are the following:
-
-**Leader Election** - Elects a single processor to generate the partition assignments and distribute it to other processors in the group.
-
-**Distributed barrier** - Coordination primitive used by the processors to reach consensus(agree) on an partition assignment.
-
-By default, embedded samza uses Zookeeper for coordinating between processors of an application and store the partition assignment state. Coordination sequence for a standalone application is listed below:
-
-1. Each processor(participant) will register with the coordination service(e.g: Zookeeper) with its participant ID.
+### Standalone Coordinator Internals
-2. One of the participants will be elected as the leader.
+Samza runs your application by logically breaking its execution down into multiple tasks. A task is the unit of parallelism for your application, with each task consumeing data from one or more partitions of your input streams.
+Obviously, there is a limit to the throughput you can achieve with a single instance. You can increase parallelism by simply running multiple instances of your application. The individual instances can be executed on the same machine, or distributed across
+machines. Likewise, to scale down your parallelism, you can shut down a few instances of your application. Samza will coordinate among available instances and dynamically assign the tasks them. The coordination logic itself is
+pluggable - with a Zookeeper-based implementation provided out of the box.
-3. The leader will monitor the list of all the active participants.
+Here is a typical sequence of actions when using Zookeeper for coordination.
-4. Whenever the list of the participants changes in a group, the leader will generate a new partition assignments for the current participants and persist it to a common storage.
+1. Everytime you spawn a new instance of your application, it registers itself with Zookeeper as a participant.
-5. Participants are notified that the new partition assignment is available. Notification is done through the coordination service(e.g. ZooKeeper).
+2. There is always a single leader - which acts as the coordinator. The coordinator which manages the assignment of tasks across the individual containers. The coordinator also monitors the liveness of individual containers and redistributes the tasks among the remaining ones during a failure.
-6. The participants will stop processing, pick up the new partition assignment, and then resume processing.
+3. Whenever a new instance joins or leaves the group, it triggers a notification to the leader. The leader can then recompute assignments of tasks to the live instances.
-In order to ensure that no two partitions are processed by different processors, processing is paused and all the processors will synchronize on a distributed barrier. Once all the processors are paused, the new partition assignments are applied, after which processing resumes.
\ No newline at end of file
+4. Once the leader publishes new partition assignments, all running instances pick up the new assignment and resume processing.
[18/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/473b6c3a
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/473b6c3a
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/473b6c3a
Branch: refs/heads/master
Commit: 473b6c3af511302b572bc6d455f6261c4c6914ae
Parents: 590eb14 7384103
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 22 19:27:24 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 22 19:27:24 2018 -0700
----------------------------------------------------------------------
RELEASE.md | 16 +-
.../versioned/deployment/standalone.md | 2 +-
.../versioned/standalone/standalone.md | 217 -------------------
.../versioned/deploy-samza-job-from-hdfs.md | 2 +-
.../tutorials/versioned/deploy-samza-to-CDH.md | 4 +-
.../versioned/hello-samza-high-level-yarn.md | 2 +-
.../versioned/hello-samza-high-level-zk.md | 2 +-
.../versioned/remote-debugging-samza.md | 2 +-
.../versioned/run-in-multi-node-yarn.md | 4 +-
.../versioned/samza-rest-getting-started.md | 2 +-
docs/startup/hello-samza/versioned/index.md | 2 +-
gradle.properties | 2 +-
samza-test/src/main/config/join/README | 8 +-
.../config/standalone.failure.test.properties | 1 -
.../TestStandaloneIntegrationApplication.java | 6 +-
samza-test/src/main/python/configs/tests.json | 2 +-
.../src/main/python/samza_job_yarn_deployer.py | 3 +-
samza-test/src/main/python/stream_processor.py | 2 +-
samza-test/src/main/python/tests/zk_client.py | 8 +-
19 files changed, 41 insertions(+), 246 deletions(-)
----------------------------------------------------------------------
[42/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/ed196c7b
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/ed196c7b
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/ed196c7b
Branch: refs/heads/master
Commit: ed196c7b1f5b06d0d9d59aa04cfe035e7c892cea
Parents: 429277b a528cc6
Author: Jagadish <jv...@linkedin.com>
Authored: Thu Nov 1 15:38:32 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Thu Nov 1 15:38:32 2018 -0700
----------------------------------------------------------------------
.../samza/config/KafkaConsumerConfig.java | 14 +
.../org/apache/samza/config/KafkaConfig.scala | 18 -
.../samza/config/RegExTopicGenerator.scala | 4 +-
.../samza/config_deprecated/KafkaConfig.scala | 400 +++++++++++++++++++
.../kafka_deprecated/KafkaSystemFactory.scala | 6 +-
.../apache/samza/config/TestKafkaConfig.scala | 42 +-
6 files changed, 425 insertions(+), 59 deletions(-)
----------------------------------------------------------------------
[02/50] samza git commit: More edits to the LinkedIn case-study. fix
rendering
Posted by ja...@apache.org.
More edits to the LinkedIn case-study. fix rendering
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/7942dbe4
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/7942dbe4
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/7942dbe4
Branch: refs/heads/master
Commit: 7942dbe4ac3b619040b543c79e6d2086c6bace36
Parents: 3dc9251
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 15 16:30:30 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 15 16:30:30 2018 -0700
----------------------------------------------------------------------
docs/_case-studies/linkedin.md | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/7942dbe4/docs/_case-studies/linkedin.md
----------------------------------------------------------------------
diff --git a/docs/_case-studies/linkedin.md b/docs/_case-studies/linkedin.md
index 0f5fa6d..1ecb440 100644
--- a/docs/_case-studies/linkedin.md
+++ b/docs/_case-studies/linkedin.md
@@ -28,25 +28,25 @@ How LinkedIn built Air Traffic Controller, a stateful stream processing system t
<!--more-->
-LinkedIn is a professional networking company that offers various services and platform for job seekers, employers and sales professionals. With a growing user base and multiple product offerings, it becomes imperative to streamline communications to members. To ensure member experience comes first before individual product metrics, LinkedIn developed a new email and notifications platform called *Air Traffic Controller*.
+LinkedIn is a professional networking company that offers various services and platform for job seekers, employers and sales professionals. With a growing user base and multiple product offerings, it becomes imperative to streamline communications to members. To ensure member experience comes first, LinkedIn developed a new email and notifications platform called *Air Traffic Controller (ATC)*.
ATC is designed to be an intelligent platform that tracks all outgoing communications and delivers the communication through the right channe to the right member at the right time.
<img src="/img/{{site.version}}/case-studies/linkedin-atc-samza-pipeline.png" alt="architecture" style="max-width: 80%; height: auto;" onclick="window.open(this.src)"/>
-Any service that wants to send out a notification to members writes its request to a Kafka topic, which ATC later reads from. The ATC platform comprises of three components:
+Any service that wants to send out a notification to members writes its request to a Kafka topic, which ATC later reads from. The ATC platform comprises of three components: <br/>
-- **Partitioner**: _Partitioners_ read incoming communication requests from Kafka and distribute them across _Pipeline_ instances based on the hash of the recipient. It also does some
-filtering early-on to drop malformed messages.
-- **Relevance processor**: The _Relevance processors_ read personalized machine-learning models from Kafka and stores them in Samza's RocksDb store for evaluating them later. It uses them to score incoming requests and determine the right channel (eg: drop it vs sending an email vs push notification vs badge) for the notification.
-- **Pipeline**: The _pipeline_ processors aggregate the output of the _Relevance_ and the _Partitioners_, thereby making the final determination on the notification. It heavily leverages Samza's local state to batch and aggregate notifications. It decides the frequency of notifications (eg: duplicate notifications are merged, notifications are capped at a certain threshold). The _Pipeline_ also implements a _scheduler_ on top of Samza's local-store so that it can schedule messages for delivery later (For eg: it makes no sense to send notifications to a member at midnight)
+_Partitioners_ read incoming communication requests from Kafka and distribute them across _Pipeline_ instances based on the hash of the recipient. It also does some
+filtering early-on to drop malformed messages. <br/><br/>
+The _Relevance processors_ read personalized machine-learning models from Kafka and stores them in Samza's state store for evaluating them later. It uses them to score incoming requests and determine the right channel for the notification (eg: drop it vs sending an email vs push notification) . <br/><br/>
+The _ATC pipeline_ processors aggregate the output from the _Relevance_ and the _Partitioners_, thereby making the final call on the notification. It heavily leverages Samza's local state to batch and aggregate notifications. It decides the frequency of notifications - duplicate notifications are merged, notifications are capped at a certain threshold. The _Pipeline_ also implements a _scheduler_ on top of Samza's local-store so that it can schedule messages for delivery later. As an example, it may not be helpful to send a push-notification at midnight. <br/><br/>
-ATC, leverages Samza extensively and uses a lot of features including but not limited to:
+ATC uses several of Samza features:
-- **Stateful processing**: The ML models in the relevance module are stored locally in RocksDb which are updated realtime time based on user feedback.
-- **Async APIs and Multi-threading**: Samza’s multi-threading and Async APIs allows ATC to perform remote calls with high-throughput. This helps bring down the 90th percentile (P90) end-to-end latency for end to end latency for push notifications from about 12 seconds to about 1.5 seconds.
-- **Host affinity**: Co-location of local state stores along with host awareness helps ATC to achieve zero downtime and instant recovery.
+**Stateful processing**: The ML models in the relevance module are stored locally in RocksDb and are updated realtime time based on user feedback. <br/>
+**Async APIs and Multi-threading**: Samza’s multi-threading and Async APIs allow ATC to perform remote calls with high throughput. This helps bring down the 90th percentile end-to-end latency for push notifications. <br/>
+**Host affinity**: Samza's incremental checkpointing and host-affinity enable ATC to achieve zero downtime during upgrades and instant recovery during failures. <br/>
Key Samza Features: *Stateful processing*, *Async API*, *Host affinity*
[49/50] samza git commit: Commit for website publish for 1.0.0
Posted by ja...@apache.org.
Commit for website publish for 1.0.0
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/2d9836df
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/2d9836df
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/2d9836df
Branch: refs/heads/master
Commit: 2d9836df520a626c97364fdbd057b75daf2d3417
Parents: 32af8e0
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Nov 27 01:46:54 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Nov 27 01:46:54 2018 -0800
----------------------------------------------------------------------
docs/_config.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/2d9836df/docs/_config.yml
----------------------------------------------------------------------
diff --git a/docs/_config.yml b/docs/_config.yml
index d581b26..0b028d3 100644
--- a/docs/_config.yml
+++ b/docs/_config.yml
@@ -25,7 +25,7 @@ exclude: [_docs]
baseurl: http://samza.apache.org
version: latest
# this is the version you will go if you click 'switch version' in "latest" pages.
-latest-release: '0.14'
+latest-release: '1.0.0'
collections:
menu:
output: false
@@ -42,4 +42,4 @@ collections:
releases:
output: true
powered-by:
- output: false
\ No newline at end of file
+ output: false
[43/50] samza git commit: Cleanup docs for HDFS connector
Posted by ja...@apache.org.
Cleanup docs for HDFS connector
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/f8470b1e
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/f8470b1e
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/f8470b1e
Branch: refs/heads/master
Commit: f8470b1ed796d00888e4ba176a0105c9b07b2938
Parents: ed196c7
Author: Jagadish <jv...@linkedin.com>
Authored: Fri Nov 2 17:33:26 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Fri Nov 2 17:33:26 2018 -0700
----------------------------------------------------------------------
.../documentation/versioned/connectors/hdfs.md | 134 +++++++------------
1 file changed, 50 insertions(+), 84 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/f8470b1e/docs/learn/documentation/versioned/connectors/hdfs.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/hdfs.md b/docs/learn/documentation/versioned/connectors/hdfs.md
index 9692d18..9b79f24 100644
--- a/docs/learn/documentation/versioned/connectors/hdfs.md
+++ b/docs/learn/documentation/versioned/connectors/hdfs.md
@@ -21,133 +21,99 @@ title: HDFS Connector
## Overview
-Samza applications can read and process data stored in HDFS. Likewise, you can also write processed results to HDFS.
-
-### Environment Requirement
-
-Your job needs to run on the same YARN cluster which hosts the HDFS you want to consume from (or write into).
+The HDFS connector allows your Samza jobs to read data stored in HDFS files. Likewise, you can write processed results to HDFS.
+To interact with HDFS, Samza requires your job to run on the same YARN cluster.
## Consuming from HDFS
+### Input Partitioning
-You can configure your Samza job to read from HDFS files with the [HdfsSystemConsumer](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/HdfsSystemConsumer.java). Avro encoded records are supported out of the box and it is easy to extend to support other formats (plain text, csv, json etc). See Event Format section below.
-
-### Partitioning
+Partitioning works at the level of individual directories and files. Each directory is treated as its own stream and each of its files is treated as a _partition_. For example, Samza creates 5 partitions when it's reading from a directory containing 5 files. There is no way to parallelize the consumption when reading from a single file - you can only have one container to process the file.
-Partitioning works at the level of individual directories and files. Each directory is treated as its own stream, while each of its files is treated as a partition. For example, when reading from a directory on HDFS with 10 files, there will be 10 partitions created. This means that you can have up-to 10 containers to process them. If you want to read from a single HDFS file, there is currently no way to break down the consumption - you can only have one container to process the file.
+### Input Event format
+Samza supports avro natively, and it's easy to extend to other serialization formats. Each avro record read from HDFS is wrapped into a message-envelope. The [envelope](../api/javadocs/org/apache/samza/system/IncomingMessageEnvelope.html) contains these 3 fields:
-### Event format
+- The key, which is empty
-Samza's HDFS consumer wraps each avro record read from HDFS into a message-envelope. The [Envelope](../api/javadocs/org/apache/samza/system/IncomingMessageEnvelope.html) contains three fields of interest:
+- The value, which is set to the avro [GenericRecord](https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericRecord.html)
-1. The key, which is empty
-2. The message, which is set to the avro [GenericRecord](https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericRecord.html)
-3. The stream partition, which is set to the name of the HDFS file
+- The partition, which is set to the name of the HDFS file
-To support input formats which are not avro, you can implement the [SingleFileHdfsReader](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/reader/SingleFileHdfsReader.java) interface (example: [AvroFileHdfsReader](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/reader/AvroFileHdfsReader.java))
+To support non-avro input formats, you can implement the [SingleFileHdfsReader](https://github.com/apache/samza/blob/master/samza-hdfs/src/main/java/org/apache/samza/system/hdfs/reader/SingleFileHdfsReader.java) interface.
-### End of stream support
+### EndOfStream
-While streaming sources like Kafka are unbounded, files on HDFS have finite data and have a notion of end-of-file.
+While streaming sources like Kafka are unbounded, files on HDFS have finite data and have a notion of EOF. When reading from HDFS, your Samza job automatically exits after consuming all the data. You can implement [EndOfStreamListenerTask](../api/javadocs/org/apache/samza/task/EndOfStreamListenerTask.html) to get a callback once EOF has been reached.
-When reading from HDFS, your Samza job automatically exits after consuming all the data. You can choose to implement [EndOfStreamListenerTask](../api/javadocs/org/apache/samza/task/EndOfStreamListenerTask.html) to receive a callback when reaching end of stream.
-### Basic Configuration
+### Defining streams
-Here is a few of the basic configs which are required to set up HdfsSystemConsumer:
+Samza uses the notion of a _system_ to describe any I/O source it interacts with. To consume from HDFS, you should create a new system that points to - `HdfsSystemFactory`. You can then associate multiple streams with this _system_. Each stream should have a _physical name_, which should be set to the name of the directory on HDFS.
{% highlight jproperties %}
-# The HDFS system consumer is implemented under the org.apache.samza.system.hdfs package,
-# so use HdfsSystemFactory as the system factory for your system
systems.hdfs.samza.factory=org.apache.samza.system.hdfs.HdfsSystemFactory
-# Define the hdfs stream
streams.hdfs-clickstream.samza.system=hdfs
-
-# You need to specify the path of files you want to consume
streams.hdfs-clickstream.samza.physical.name=hdfs:/data/clickstream/2016/09/11
-# You can specify a white list of files you want your job to process (in Java Pattern style)
-systems.hdfs.partitioner.defaultPartitioner.whitelist=.*avro
-
-# You can specify a black list of files you don't want your job to process (in Java Pattern style),
-# by default it's empty.
-# Note that you can have both white list and black list, in which case both will be applied.
-systems.hdfs.partitioner.defaultPartitioner.blacklist=somefile.avro
{% endhighlight %}
-### Security Configuration
+The above example defines a stream called `hdfs-clickstream` that reads data from the `/data/clickstream/2016/09/11` directory.
-The following additional configs are required when accessing HDFS clusters that have kerberos enabled:
+#### Whitelists & Blacklists
+If you only want to consume from files that match a certain pattern, you can configure a whitelist. Likewise, you can also blacklist consuming from certain files. When both are specified, the _whitelist_ selects the files to be filtered and the _blacklist_ is later applied on its results.
{% highlight jproperties %}
-# When the job is running in a secure environment, use the SamzaYarnSecurityManagerFactory, which fetches and renews the Kerberos delegation tokens
-job.security.manager.factory=org.apache.samza.job.yarn.SamzaYarnSecurityManagerFactory
-
-# Kerberos principal
-yarn.kerberos.principal=your-principal-name
-
-# Path of the keytab file (local path)
-yarn.kerberos.keytab=/tmp/keytab
-{% endhighlight %}
-
-### Advanced Configuration
-
-Some of the advanced configuration you might need to set up:
-
-{% highlight jproperties %}
-# Specify the group pattern for advanced partitioning.
-systems.hdfs-clickstream.partitioner.defaultPartitioner.groupPattern=part-[id]-.*
-
-# Specify the type of files your job want to process (support avro only for now)
-systems.hdfs-clickstream.consumer.reader=avro
-
-# Max number of retries (per-partition) before the container fails.
-system.hdfs-clickstream.consumer.numMaxRetries=10
+systems.hdfs.partitioner.defaultPartitioner.whitelist=.*avro
+systems.hdfs.partitioner.defaultPartitioner.blacklist=somefile.avro
{% endhighlight %}
-The advanced partitioning goes beyond the basic assumption that each file is a partition. With advanced partitioning you can group files into partitions arbitrarily. For example, if you have a set of files as [part-01-a.avro, part-01-b.avro, part-02-a.avro, part-02-b.avro, part-03-a.avro] that you want to organize into three partitions as (part-01-a.avro, part-01-b.avro), (part-02-a.avro, part-02-b.avro), (part-03-a.avro), where the numbers in the middle act as a “group identifier”, you can then set this property to be “part-[id]-.” (note that "[id]" is a reserved term here, i.e. you have to literally put it as [id]). The partitioner will apply this pattern to all file names and extract the “group identifier” (“[id]” in the pattern), then use the “group identifier” to group files into partitions.
## Producing to HDFS
-The samza-hdfs module implements a Samza Producer to write to HDFS. The current implementation includes a ready-to-use HdfsSystemProducer, and two HdfsWriters: One that writes messages of raw bytes to a SequenceFile of BytesWritable keys and values. Another writes out Avro data files including the schema automatically reflected from the POJO objects fed to it.
+#### Output format
-### Configuring an HdfsSystemProducer
-
-You can configure an HdfsSystemProducer like any other Samza system: using configuration keys and values set in a job.properties file. You might configure the system producer for use by your StreamTasks like this:
+Samza allows writing your output results to HDFS in AVRO format. You can either use avro's GenericRecords or have Samza automatically infer the schema for your object using reflection.
{% highlight jproperties %}
# set the SystemFactory implementation to instantiate HdfsSystemProducer aliased to 'hdfs'
systems.hdfs.samza.factory=org.apache.samza.system.hdfs.HdfsSystemFactory
+systems.hdfs.producer.hdfs.writer.class=org.apache.samza.system.hdfs.writer.AvroDataFileHdfsWriter
+{% endhighlight %}
+
-# Assign the implementation class for this system's HdfsWriter
+If your output is non-avro, you can describe its format by implementing your own serializer.
+{% highlight jproperties %}
systems.hdfs.producer.hdfs.writer.class=org.apache.samza.system.hdfs.writer.TextSequenceFileHdfsWriter
-#systems.hdfs.producer.hdfs.writer.class=org.apache.samza.system.hdfs.writer.AvroDataFileHdfsWriter
-# define a serializer/deserializer for the hdfs system
-# DO NOT define (i.e. comment out) a SerDe when using the AvroDataFileHdfsWriter so it can reflect the schema
-systems.hdfs.samza.msg.serde=some-serde-impl
+serializers.registry.my-serde-name.class=MySerdeFactory
+systems.hdfs.samza.msg.serde=my-serde-name
+{% endhighlight %}
-# Assign a serde implementation to be used for the stream called "metrics"
-systems.hdfs.streams.metrics.samza.msg.serde=some-metrics-impl
-# Set compression type supported by chosen Writer.
-# AvroDataFileHdfsWriter supports snappy, bzip2, deflate or none
-systems.hdfs.producer.hdfs.compression.type=snappy
+#### Output directory structure
-# The base dir for HDFS output. Output is structured into buckets. The default Bucketer for SequenceFile HdfsWriters
-# is currently /BASE/JOB_NAME/DATE_PATH/FILES, where BASE is set below
+Samza allows you to control the base HDFS directory to write your output. You can also organize the output into sub-directories depending on the time your application ran, by configuring a date-formatter.
+{% highlight jproperties %}
systems.hdfs.producer.hdfs.base.output.dir=/user/me/analytics/clickstream_data
-
-# Assign the implementation class for the HdfsWriter's Bucketer
-systems.hdfs.producer.hdfs.bucketer.class=org.apache.samza.system.hdfs.writer.JobNameDateTimeBucketer
-
-# Configure the DATE_PATH the Bucketer will set to bucket output files by day for this job run.
systems.hdfs.producer.hdfs.bucketer.date.path.format=yyyy_MM_dd
+{% endhighlight %}
-# Optionally set the max output bytes (records for AvroDataFileHdfsWriter) per file.
-# A new file will be cut and output continued on the next write call each time this many bytes
-# (records for AvroDataFileHdfsWriter) have been written.
+You can configure the maximum size of each file or the maximum number of records per-file. Once either limits have been reached, Samza will create a new file.
+
+{% highlight jproperties %}
systems.hdfs.producer.hdfs.write.batch.size.bytes=134217728
-#systems.hdfs.producer.hdfs.write.batch.size.records=10000
+systems.hdfs.producer.hdfs.write.batch.size.records=10000
{% endhighlight %}
-The above configuration assumes a Metrics and Serde implementation has been properly configured against the some-metrics-impl and some-serde-impl and labels somewhere else in the same job.properties file. Each of these properties has a reasonable default, so you can leave out the ones you don’t need to customize for your job run.
\ No newline at end of file
+## Security
+
+You can access Kerberos-enabled HDFS clusters by providing your principal and the path to your key-tab file. Samza takes care of automatically creating and renewing your Kerberos tokens periodically.
+
+{% highlight jproperties %}
+job.security.manager.factory=org.apache.samza.job.yarn.SamzaYarnSecurityManagerFactory
+
+# Kerberos principal
+yarn.kerberos.principal=your-principal-name
+
+# Path of the keytab file (local path)
+yarn.kerberos.keytab=/tmp/keytab
+{% endhighlight %}
[45/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/65f58bfa
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/65f58bfa
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/65f58bfa
Branch: refs/heads/master
Commit: 65f58bfa010ae60af0584e9a464028e338d1995f
Parents: e0e5e87 5f7a22c
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Nov 26 14:17:14 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Nov 26 14:17:14 2018 -0800
----------------------------------------------------------------------
build.gradle | 1 +
docs/_committers/hai-lu.md | 28 +++
.../versioned/jobs/configuration-table.html | 17 ++
.../apache/samza/config/JavaTableConfig.java | 118 +++++++++++++
.../apache/samza/metrics/SamzaHistogram.java | 79 +++++++++
.../org/apache/samza/table/TableProvider.java | 15 --
.../samza/table/TableProviderFactory.java | 10 +-
.../java/org/apache/samza/table/TableSpec.java | 172 -------------------
.../table/descriptors/BaseTableDescriptor.java | 68 +++-----
.../descriptors/CachingTableDescriptor.java | 70 ++++----
.../descriptors/GuavaCacheTableDescriptor.java | 32 ++--
.../descriptors/HybridTableDescriptor.java | 3 +-
.../table/descriptors/LocalTableDescriptor.java | 105 ++++++++---
.../descriptors/RemoteTableDescriptor.java | 122 ++++++-------
.../table/descriptors/TableDescriptor.java | 17 ++
.../samza/table/TestBaseTableDescriptor.java | 73 ++++++++
.../consumer/EventHubSystemConsumer.java | 2 +-
.../system/eventhub/metrics/SamzaHistogram.java | 83 ---------
.../eventhub/producer/AsyncSystemProducer.java | 2 +-
.../StreamApplicationDescriptorImpl.java | 10 +-
.../TaskApplicationDescriptorImpl.java | 8 +-
.../apache/samza/config/JavaTableConfig.java | 88 ----------
.../samza/execution/ExecutionPlanner.java | 35 ++--
.../org/apache/samza/execution/JobGraph.java | 14 +-
.../samza/execution/JobGraphJsonGenerator.java | 41 ++---
.../org/apache/samza/execution/JobNode.java | 10 +-
.../JobNodeConfigurationGenerator.java | 38 ++--
.../execution/OperatorSpecGraphAnalyzer.java | 29 ++--
.../samza/operators/MessageStreamImpl.java | 6 +-
.../org/apache/samza/operators/TableImpl.java | 18 +-
.../samza/operators/impl/OperatorImpl.java | 16 +-
.../operators/impl/SendToTableOperatorImpl.java | 2 +-
.../impl/StreamTableJoinOperatorImpl.java | 13 +-
.../samza/operators/spec/OperatorSpecs.java | 14 +-
.../operators/spec/SendToTableOperatorSpec.java | 14 +-
.../spec/StreamTableJoinOperatorSpec.java | 14 +-
.../apache/samza/table/BaseTableProvider.java | 40 ++---
.../samza/table/TableConfigGenerator.java | 66 +------
.../org/apache/samza/table/TableManager.java | 54 ++----
.../samza/table/caching/CachingTable.java | 3 -
.../table/caching/CachingTableProvider.java | 24 +--
.../caching/CachingTableProviderFactory.java | 7 +-
.../table/caching/guava/GuavaCacheTable.java | 3 -
.../caching/guava/GuavaCacheTableProvider.java | 13 +-
.../guava/GuavaCacheTableProviderFactory.java | 6 +-
.../table/remote/RemoteReadWriteTable.java | 33 ----
.../samza/table/remote/RemoteReadableTable.java | 24 ++-
.../samza/table/remote/RemoteTableProvider.java | 57 +++---
.../remote/RemoteTableProviderFactory.java | 10 +-
.../table/utils/DefaultTableReadMetrics.java | 2 +
.../apache/samza/task/EpochTimeScheduler.java | 6 +-
.../org/apache/samza/config/MetricsConfig.scala | 11 ++
.../apache/samza/container/SamzaContainer.scala | 2 +-
.../TestStreamApplicationDescriptorImpl.java | 9 +-
.../TestTaskApplicationDescriptorImpl.java | 3 -
.../org/apache/samza/context/MockContext.java | 6 +-
.../samza/execution/TestExecutionPlanner.java | 44 ++---
.../execution/TestJobGraphJsonGenerator.java | 15 ++
.../TestJobNodeConfigurationGenerator.java | 94 ++--------
.../samza/operators/TestJoinOperator.java | 1 +
.../samza/operators/TestMessageStreamImpl.java | 11 +-
.../samza/operators/impl/TestOperatorImpl.java | 5 +-
.../operators/impl/TestOperatorImplGraph.java | 1 +
.../impl/TestStreamTableJoinOperatorImpl.java | 5 +-
.../operators/impl/TestWindowOperator.java | 1 +
.../operators/spec/OperatorSpecTestUtils.java | 11 +-
.../samza/operators/spec/TestOperatorSpec.java | 26 +--
.../apache/samza/table/TestTableManager.java | 52 +-----
.../samza/table/caching/TestCachingTable.java | 53 +++---
.../descriptors/TestLocalTableDescriptor.java | 162 +++++++++++++++++
.../descriptors/TestRemoteTableDescriptor.java | 88 ++++++----
.../descriptors/InMemoryTableDescriptor.java | 38 ++--
.../descriptors/InMemoryTableProvider.java | 71 --------
.../InMemoryTableProviderFactory.java | 33 ----
.../inmemory/TestInMemoryTableDescriptor.java | 65 +++++++
.../TestInMemoryTableDescriptor.java | 55 ------
.../descriptors/TestInMemoryTableProvider.java | 67 --------
.../kv/descriptors/RocksDbTableDescriptor.java | 64 +++----
.../kv/descriptors/RocksDbTableProvider.java | 75 --------
.../RocksDbTableProviderFactory.java | 31 ----
.../descriptors/TestRocksDbTableDescriptor.java | 97 ++++++-----
.../descriptors/TestRocksDbTableProvider.java | 68 --------
.../samza/storage/kv/LocalReadWriteTable.java | 3 -
.../samza/storage/kv/LocalReadableTable.java | 8 +-
.../samza/storage/kv/LocalTableProvider.java | 103 +----------
.../storage/kv/LocalTableProviderFactory.java | 31 ++++
.../kv/descriptors/TestLocalTableProvider.java | 103 +----------
.../apache/samza/sql/avro/AvroRelConverter.java | 33 +++-
.../samza/sql/translator/JoinTranslator.java | 12 +-
.../translator/LogicalAggregateTranslator.java | 8 +-
.../samza/sql/translator/ProjectTranslator.java | 65 +++++--
.../samza/sql/translator/QueryTranslator.java | 21 +--
.../SamzaSqlRemoteTableJoinFunction.java | 9 +-
.../translator/SamzaSqlTableJoinFunction.java | 5 +
.../samza/sql/avro/TestAvroRelConversion.java | 52 +++++-
.../samza/sql/e2e/TestSamzaSqlRemoteTable.java | 95 +++++++++-
.../samza/sql/system/TestAvroSystemFactory.java | 13 +-
.../samza/sql/testutil/SamzaSqlTestConfig.java | 11 +-
.../sql/testutil/TestMetricsRegistryImpl.java | 117 +++++++++++++
.../sql/translator/TestJoinTranslator.java | 2 +-
.../sql/translator/TestProjectTranslator.java | 92 ++++++----
.../sql/translator/TestQueryTranslator.java | 24 +--
.../sql/translator/TranslatorTestBase.java | 11 +-
.../samza/example/TaskApplicationExample.java | 5 +-
.../test/functions/TestSchedulerFunction.java | 93 ++++++++++
.../test/samzasql/TestSamzaSqlEndToEnd.java | 29 +++-
106 files changed, 1967 insertions(+), 1977 deletions(-)
----------------------------------------------------------------------
[31/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/2ffedc9a
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/2ffedc9a
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/2ffedc9a
Branch: refs/heads/master
Commit: 2ffedc9a9ab376957371cc0068e85db6e1c62a17
Parents: 1befcad 08496c9
Author: Jagadish <jv...@linkedin.com>
Authored: Sat Oct 27 11:15:08 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sat Oct 27 11:15:08 2018 -0700
----------------------------------------------------------------------
.../sql/impl/ConfigBasedIOResolverFactory.java | 13 ++-
.../apache/samza/sql/planner/QueryPlanner.java | 3 +-
.../sql/runner/SamzaSqlApplicationConfig.java | 20 ++++-
.../samza/sql/translator/JoinTranslator.java | 44 ++++++---
.../translator/LogicalAggregateTranslator.java | 7 +-
.../samza/sql/translator/QueryTranslator.java | 6 +-
.../sql/testutil/TestIOResolverFactory.java | 12 ++-
.../sql/translator/TestJoinTranslator.java | 2 +-
.../sql/translator/TestQueryTranslator.java | 95 ++++++++++----------
.../apache/samza/test/framework/TestRunner.java | 23 ++++-
.../StreamApplicationIntegrationTest.java | 43 ++++++---
.../framework/StreamTaskIntegrationTest.java | 5 +-
.../test/samzasql/TestSamzaSqlEndToEnd.java | 34 ++++++-
13 files changed, 218 insertions(+), 89 deletions(-)
----------------------------------------------------------------------
[13/50] samza git commit: Cleanup website docs for 1.0 release
Posted by ja...@apache.org.
Cleanup website docs for 1.0 release
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/28da01c6
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/28da01c6
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/28da01c6
Branch: refs/heads/master
Commit: 28da01c68e3551ff9ea43b7a1642ca6eb490e9b6
Parents: 9412f3b
Author: Jagadish <jv...@linkedin.com>
Authored: Sun Oct 21 10:47:37 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sun Oct 21 10:47:37 2018 -0700
----------------------------------------------------------------------
.../2018-07-19-stream_processing-meetup-summary.md | 16 ++++++----------
docs/_blog/2018-09-26-stangeloop-recap.md | 3 ++-
docs/_case-studies/optimizely.md | 4 ++--
docs/_layouts/default.html | 10 +++++-----
docs/css/main.new.css | 5 +++++
5 files changed, 20 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/28da01c6/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
----------------------------------------------------------------------
diff --git a/docs/_blog/2018-07-19-stream_processing-meetup-summary.md b/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
index b1fedc5..de3fa06 100644
--- a/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
+++ b/docs/_blog/2018-07-19-stream_processing-meetup-summary.md
@@ -1,6 +1,6 @@
---
layout: blog
-title: Recap of Stream Processing with Apache Kafka & Apache Samza (July '18)
+title: Our July 2018 meetup - A Report
icon: analytics
authors:
- name:
@@ -25,18 +25,14 @@ excerpt_separator: <!--more-->
limitations under the License.
-->
-A look back at July edition of quarterly stream processing meetup
+On July 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies.
+The event featured technical deep dives by engineers from LinkedIn and Uber on the latest in the stream processing space...
<!--more-->
-
-On July 19th another successful stream processing meetup was hosted by LinkedIn!
-This event focused on Apache Kafka, Apache Samza, and related streaming technologies.
-This meetup was a full house and had techincal deep dives by engineers from LinkedIn and Uber on the latest
-and greatest in streaming tech
-
-<br>
-
+July 2018 - On July 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies.
+The event featured technical deep dives by engineers from LinkedIn and Uber on the latest in the stream processing space. Here is a brief summary
+of the talks that were presented.
### [Beam me up Samza: How we built a Samza Runner for Apache Beam](https://youtu.be/o5GaifLoZho)
http://git-wip-us.apache.org/repos/asf/samza/blob/28da01c6/docs/_blog/2018-09-26-stangeloop-recap.md
----------------------------------------------------------------------
diff --git a/docs/_blog/2018-09-26-stangeloop-recap.md b/docs/_blog/2018-09-26-stangeloop-recap.md
index 1229a58..4c5e262 100644
--- a/docs/_blog/2018-09-26-stangeloop-recap.md
+++ b/docs/_blog/2018-09-26-stangeloop-recap.md
@@ -1,12 +1,13 @@
---
layout: blog
-title: Recap of Chasing Stream Processing Utopia
+title: On chasing the Stream Processing Utopia
icon: analytics
authors:
- name:
website:
image:
excerpt_separator: <!--more-->
+exclude_from_loop: true
---
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
http://git-wip-us.apache.org/repos/asf/samza/blob/28da01c6/docs/_case-studies/optimizely.md
----------------------------------------------------------------------
diff --git a/docs/_case-studies/optimizely.md b/docs/_case-studies/optimizely.md
index c93c4b2..f259520 100644
--- a/docs/_case-studies/optimizely.md
+++ b/docs/_case-studies/optimizely.md
@@ -1,7 +1,7 @@
---
layout: case-study
hide_title: true # so we have control in case-study layout, but can still use page
-title: Real Time Session Aggregation
+title: On using Samza at Optimizely to compute analytics over session windows.
study_domain: optimizely.com
priority: 2
menu_title: Optimizely
@@ -25,7 +25,7 @@ excerpt_separator: <!--more-->
limitations under the License.
-->
-Real Time Session Aggregation
+On using Samza at Optimizely to compute analytics over session windows
<!--more-->
http://git-wip-us.apache.org/repos/asf/samza/blob/28da01c6/docs/_layouts/default.html
----------------------------------------------------------------------
diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html
index 988f536..6c7d0c8 100644
--- a/docs/_layouts/default.html
+++ b/docs/_layouts/default.html
@@ -132,7 +132,7 @@
High performance
</div>
<div class="section__item-features">
- <div class="section__item-feature">Samza provides extremely low latencies and high throughput to analyze your data instantly</div>
+ <div class="section__item-feature">Samza provides extremely low latencies and <a href="https://engineering.linkedin.com/performance/benchmarking-apache-samza-12-million-messages-second-single-node">high throughput</a> to analyze your data instantly</div>
</div>
</div>
@@ -143,7 +143,7 @@
Horizontally scalable
</div>
<div class="section__item-features">
- <div class="section__item-feature">Scales to several terabytes of state with features like incremental checkpoints and host-affinity</div>
+ <div class="section__item-feature">Scales to several terabytes of state with features like incremental checkpoints and <a href="/learn/documentation/latest/architecture/architecture-overview.html#host-affinity">host-affinity</a></div>
</div>
</div>
@@ -153,7 +153,7 @@
Easy to Operate
</div>
<div class="section__item-features">
- <div class="section__item-feature">Samza is easy to operate with flexible deployment options - YARN, Kubernetes or standalone</div>
+ <div class="section__item-feature">Samza is easy to operate with <a href="/learn/documentation/latest/deployment/deployment-model.html">flexible</a> deployment options - YARN, Kubernetes or standalone</div>
</div>
</div>
@@ -163,7 +163,7 @@
Powerful APIs
</div>
<div class="section__item-features">
- <div class="section__item-feature">Rich APIs to build your applications: Choose from low-level, Streams DSL, Samza SQL and Apache BEAM APIs</div>
+ <div class="section__item-feature">Rich APIs to build your applications: Choose from <a href="/learn/documentation/latest/api/low-level-api.html">low-level </a>, <a href="/learn/documentation/latest/api/high-level-api.html">Streams DSL </a>, <a href="/learn/documentation/latest/api/samza-sql.html">Samza SQL</a> and Apache BEAM APIs</div>
</div>
</div>
@@ -183,7 +183,7 @@
Pluggable architecture
</div>
<div class="section__item-features">
- <div class="section__item-feature">Integrates with several sources including Kafka, HDFS, AWS Kinesis, Azure Eventhubs, K-V stores and ElasticSearch</div>
+ <div class="section__item-feature">Integrates with several sources including <a href="/learn/documentation/latest/connectors/kafka.html">Kafka</a>, <a href="/learn/documentation/latest/connectors/hdfs.html">HDFS</a>, <a href="/learn/documentation/latest/connectors/kinesis.html">AWS Kinesis</a>, <a href="/learn/documentation/latest/connectors/eventhubs.html">Azure Eventhubs</a>, K-V stores and ElasticSearch</div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/samza/blob/28da01c6/docs/css/main.new.css
----------------------------------------------------------------------
diff --git a/docs/css/main.new.css b/docs/css/main.new.css
index 6773cc6..558dd29 100644
--- a/docs/css/main.new.css
+++ b/docs/css/main.new.css
@@ -86,6 +86,11 @@ a.side-navigation__group-title::after {
transition: transform 250ms ease-in-out;
}
+.section__items a::after {
+ border-bottom: none;
+}
+
+
.content a:hover::after,
.section a:hover::after,
.content a.active::after,
[12/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/9412f3ba
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/9412f3ba
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/9412f3ba
Branch: refs/heads/master
Commit: 9412f3ba2c7026e3b49683d65b9f80c46bf919ae
Parents: d36d6b6 880d2f0
Author: Jagadish <jv...@linkedin.com>
Authored: Fri Oct 19 14:13:43 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Fri Oct 19 14:13:43 2018 -0700
----------------------------------------------------------------------
.../versioned/api/high-level-api.md | 396 ++++++++++++++++++-
.../versioned/api/low-level-api.md | 308 ++++++++++++++-
.../documentation/versioned/api/overview.md | 160 --------
.../versioned/api/programming-model.md | 158 ++++++++
4 files changed, 841 insertions(+), 181 deletions(-)
----------------------------------------------------------------------
[04/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/9e6697b5
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/9e6697b5
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/9e6697b5
Branch: refs/heads/master
Commit: 9e6697b57bb5d431662cb4575a5d243656be67c0
Parents: f569a5c 7d3eb08
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 15 16:48:07 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 15 16:48:07 2018 -0700
----------------------------------------------------------------------
----------------------------------------------------------------------
[38/50] samza git commit: Cleanup documentation for Kinesis
Posted by ja...@apache.org.
Cleanup documentation for Kinesis
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/ff6032c3
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/ff6032c3
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/ff6032c3
Branch: refs/heads/master
Commit: ff6032c300fd301de90695f7d4ec03f852f091bb
Parents: 0b4c689
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 30 12:06:54 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 30 12:06:54 2018 -0700
----------------------------------------------------------------------
docs/learn/documentation/versioned/connectors/kinesis.md | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/ff6032c3/docs/learn/documentation/versioned/connectors/kinesis.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/connectors/kinesis.md b/docs/learn/documentation/versioned/connectors/kinesis.md
index 4f667d2..85149f6 100644
--- a/docs/learn/documentation/versioned/connectors/kinesis.md
+++ b/docs/learn/documentation/versioned/connectors/kinesis.md
@@ -22,9 +22,7 @@ title: Kinesis Connector
## Kinesis I/O: Quickstart
The Samza Kinesis connector allows you to interact with [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams),
-Amazon’s data streaming service.
-
-The `hello-samza` project includes an example of processing Kinesis streams using Samza. Here is the complete [source code](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/kinesis/KinesisHelloSamza.java) and [configs](https://github.com/apache/samza-hello-samza/blob/master/src/main/config/kinesis-hello-samza.properties).
+Amazon’s data streaming service. The `hello-samza` project includes an example of processing Kinesis streams using Samza. Here is the complete [source code](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/kinesis/KinesisHelloSamza.java) and [configs](https://github.com/apache/samza-hello-samza/blob/master/src/main/config/kinesis-hello-samza.properties).
You can build and run this example using this [tutorial](https://github.com/apache/samza-hello-samza#hello-samza).
[19/50] samza git commit: Stop requiring processor.id configuration
in PassThroughJC
Posted by ja...@apache.org.
Stop requiring processor.id configuration in PassThroughJC
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/b027b438
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/b027b438
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/b027b438
Branch: refs/heads/master
Commit: b027b438479153e1ed73ad3b1ecf9de148af8d2b
Parents: 473b6c3
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 22 22:09:20 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 22 22:09:20 2018 -0700
----------------------------------------------------------------------
.../org/apache/samza/standalone/PassthroughJobCoordinator.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/b027b438/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
----------------------------------------------------------------------
diff --git a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
index 737ac3e..7522106 100644
--- a/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
+++ b/samza-core/src/main/java/org/apache/samza/standalone/PassthroughJobCoordinator.java
@@ -119,7 +119,7 @@ public class PassthroughJobCoordinator implements JobCoordinator {
SystemAdmins systemAdmins = new SystemAdmins(config);
StreamMetadataCache streamMetadataCache = new StreamMetadataCache(systemAdmins, 5000, SystemClock.instance());
systemAdmins.start();
- String containerId = Integer.toString(config.getInt(JobConfig.PROCESSOR_ID()));
+ String containerId = config.get(JobConfig.PROCESSOR_ID());
/** TODO:
Locality Manager seems to be required in JC for reading locality info and grouping tasks intelligently and also,
[23/50] samza git commit: clean-up the quickstart page
Posted by ja...@apache.org.
clean-up the quickstart page
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/14e62205
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/14e62205
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/14e62205
Branch: refs/heads/master
Commit: 14e6220512e6922b89980bdbb6440c09fac380cb
Parents: 2615ad5
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:10:38 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:18:14 2018 -0700
----------------------------------------------------------------------
docs/_docs/replace-versioned.sh | 5 +-
docs/_menu/index.html | 2 +-
docs/startup/code-examples/versioned/index.md | 49 ++++++++++++++++++
docs/startup/quick-start/versioned/index.md | 59 +++++++++++++---------
4 files changed, 88 insertions(+), 27 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/14e62205/docs/_docs/replace-versioned.sh
----------------------------------------------------------------------
diff --git a/docs/_docs/replace-versioned.sh b/docs/_docs/replace-versioned.sh
index 24bf7ae..c454cac 100755
--- a/docs/_docs/replace-versioned.sh
+++ b/docs/_docs/replace-versioned.sh
@@ -44,4 +44,7 @@ echo "replaced startup/hello-samza/versioned to startup/hello-samza/"$version
mv -f $DIR/_site/startup/hello-samza/versioned $DIR/_site/startup/hello-samza/$version
echo "replaced startup/quick-start/versioned to startup/quick-start/"$version
-mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
\ No newline at end of file
+mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
+
+echo "replaced startup/code-examples/versioned to startup/code-examples/"$version
+mv -f $DIR/_site/startup/code-examples/versioned $DIR/_site/startup/code-examples/$version
http://git-wip-us.apache.org/repos/asf/samza/blob/14e62205/docs/_menu/index.html
----------------------------------------------------------------------
diff --git a/docs/_menu/index.html b/docs/_menu/index.html
index 0d1750f..a363bae 100644
--- a/docs/_menu/index.html
+++ b/docs/_menu/index.html
@@ -5,7 +5,7 @@ items:
- menu_title: QuickStart
url: /startup/quick-start/version/
- menu_title: Code Examples
- url: /learn/tutorials/version/
+ url: /startup/code-examples/version/
- menu_title: Documentation
has_sub: true
has_sub_subs: true
http://git-wip-us.apache.org/repos/asf/samza/blob/14e62205/docs/startup/code-examples/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/code-examples/versioned/index.md b/docs/startup/code-examples/versioned/index.md
new file mode 100644
index 0000000..ba1cc3e
--- /dev/null
+++ b/docs/startup/code-examples/versioned/index.md
@@ -0,0 +1,49 @@
+---
+layout: page
+title:
+---
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+
+### Checking out our examples
+
+The [hello-samza](https://github.com/apache/samza-hello-samza) project contains several examples to help you create your Samza applications. To checkout the hello-samza project:
+
+{% highlight bash %}
+> git clone https://git.apache.org/samza-hello-samza.git hello-samza
+{% endhighlight %}
+
+#### High-level API examples
+[The Samza Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook) contains various recipes using the Samza high-level API.
+These include:
+
+- The [Filter example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) demonstrates how to perform stateless operations on a stream.
+
+- The [Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/JoinExample.java]) demonstrates how you can join a Kafka stream of page-views with a stream of ad-clicks
+
+- The [Stream-Table Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/RemoteTableJoinExample.java) demonstrates how the Samza Table API. It joins a Kafka stream with a remote dataset accessed through a REST service.
+
+- The [SessionWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/SessionWindowExample.java) and [TumblingWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/TumblingWindowExample.java) examples illustrate Samza's rich windowing and triggering capabilities.
+
+
+In addition to the cookbook, you can also consult these:
+
+- [Wikipedia Parser](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): An advanced example that builds a streaming pipeline consuming a live-feed of wikipedia edits, parsing each message and generating statistics from them.
+
+
+- [Amazon Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis) and [Azure Eventhubs](https://github.com/apache/samza-hello-samza/tree/latest/src/main/java/samza/examples/azure) examples that cover how to consume input data from the respective systems.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/14e62205/docs/startup/quick-start/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/quick-start/versioned/index.md b/docs/startup/quick-start/versioned/index.md
index a046ee7..0c77a41 100644
--- a/docs/startup/quick-start/versioned/index.md
+++ b/docs/startup/quick-start/versioned/index.md
@@ -19,7 +19,7 @@ title: Quick Start
limitations under the License.
-->
-This tutorial will go through the steps of creating your first Samza application - `WordCount`. It demonstrates how to start writing a Samza application, consume from a kafka stream, tokenize the lines into words, and count the frequency of each word. For this tutorial we are going to use gradle 4.9 to build the projects. The full tutorial project tar file can be downloaded [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
+In this tutorial, we will create our first Samza application - `WordCount`. This application will consume messages from a Kafka stream, tokenize them into individual words and count the frequency of each word. Let us download the entire project from [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
### Setting up a Java Project
@@ -48,7 +48,7 @@ You can copy build.gradle and gradle.properties files from the downloaded tutori
### Create a Samza StreamApplication
-Now let’s write some code! The first step is to create your own Samza application by implementing the [StreamApplication](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplication.html) class:
+Now let’s write some code! An application written using Samza's [high-level API](/learn/documentation/{{site.version}}/api/api/high-level-api.html) implements the [StreamApplication](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplication.html) interface:
{% highlight java %}
package samzaapp;
@@ -63,11 +63,11 @@ public class WordCount implements StreamApplication {
}
{% endhighlight %}
-The StreamApplication interface provides an API method named describe() for you to specify your streaming pipeline. Using [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html), you can describe your entire data processing task from data inputs, operations and outputs.
+The interface provides a single method named `describe()`, which allows us to define our inputs, the processing logic and outputs for our application.
-### Input data source using Kafka
+### Describe your inputs and outputs
-In this example, we are going to use Kafka as the input data source and consume the text for word count line by line. We start by defining a KafkaSystemDescriptor, which specifies the properties to establishing the connection to the local Kafka cluster. Then we create a `KafkaInputDescriptor`/`KafkaOutputDescriptor` to set up the topic, Serializer and Deserializer. Finally we use this input in the [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html) so we can consume from this topic. The code is in the following:
+To interact with Kafka, we will first create a `KafkaSystemDescriptor` by providing the coordinates of the Kafka cluster. For each Kafka topic our application reads from, we create a `KafkaInputDescriptor` with the name of the topic and a serializer. Likewise, for each output topic, we instantiate a corresponding `KafkaOutputDescriptor`.
{% highlight java %}
public class WordCount implements StreamApplication {
@@ -99,23 +99,24 @@ public class WordCount implements StreamApplication {
}
{% endhighlight %}
-The resulting [MessageStream](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/operators/MessageStream.html) lines contains the data set that reads from Kafka and deserialized into string of each line. We also defined the output stream counts so we can write the word count results to it. Next let’s add processing logic.
+The above example creates a [MessageStream](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/operators/MessageStream.html) which reads from an input topic named `sample-text`. It also defines an output stream that emits results to a topic named `word-count-output`. Next let’s add our processing logic.
### Add word count processing logic
-First we are going to extract the value from lines. This is a one-to-one transform and we can use the Samza map operator as following:
+Kafka messages typically have a key and a value. Since we only care about the value here, we will apply the `map` operator on the input stream to extract the value.
{% highlight java %}
-lines .map(kv -> kv.value)
+lines.map(kv -> kv.value)
{% endhighlight %}
-Then we will split the line into words by using the flatmap operator:
+Next, we will tokenize the message into individual words using the `flatmap` operator.
{% highlight java %}
.flatMap(s -> Arrays.asList(s.split("\\W+")))
{% endhighlight %}
-Now let’s think about how to count the words. We need to aggregate the count based on the word as the key, and emit the aggregation results once there are no more data coming. Here we can use a session window which will trigger the output if there is no data coming within a certain interval.
+
+We now need to group the words, aggregate their respective counts and periodically emit our results. For this, we will use Samza's session-windowing feature.
{% highlight java %}
.window(Windows.keyedSessionWindow(
@@ -123,7 +124,11 @@ Now let’s think about how to count the words. We need to aggregate the count b
new StringSerde(), new IntegerSerde()), "count")
{% endhighlight %}
-The output will be captured in a [WindowPane](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/operators/windows/WindowPane.html) type, which contains the key and the aggregation value. We add a further map to transform that into a KV. To write the output to the output Kafka stream, we used the sentTo operator in Samza:
+Let's walk through each of the parameters to the above `window` function:
+The first parameter is a "key function", which defines the key to group messages by. In our case, we can simply use the word as the key. The second parameter is the windowing interval, which is set to 5 seconds. The third parameter is a function which provides the initial value for our aggregations. We can start with an initial count of zero for each word. The fourth parameter is an aggregation function for computing counts. The next two parameters specify the key and value serializers for our window.
+
+The output from the window operator is captured in a [WindowPane](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/operators/windows/WindowPane.html) type, which contains the word as the key and its count as the value. We add a further `map` to format this into a `KV`, that we can send to our Kafka topic. To write our results to the output topic, we use the `sendTo` operator in Samza.
+
{% highlight java %}
.map(windowPane ->
@@ -148,27 +153,31 @@ lines
{% endhighlight %}
-### Config your application
-
-In this section we will configure the word count example to run locally in a single JVM. Please add a file named “word-count.properties” under the config folder. We will add the job configs in this file.
+### Configure your application
-Since there is only a single Samza processor, there is no coordination required. We use the PassthroughJobCoordinator for the example. We also group all Samza tasks into this single processor. As for the Kafka topic, we will consume from the beginning. Here is the full config needed for the job:
+In this section, we will configure our word count example to run locally in a single JVM. Let us add a file named “word-count.properties” under the config folder.
{% highlight jproperties %}
job.name=word-count
+# Use a PassthroughJobCoordinator since there is no coordination needed
job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory
job.coordination.utils.factory=org.apache.samza.standalone.PassthroughCoordinationUtilsFactory
+
job.changelog.system=kafka
+
+# Use a single container to process all of the data
task.name.grouper.factory=org.apache.samza.container.grouper.task.SingleContainerGrouperFactory
processor.id=0
+
+# Read from the beginning of the topic
systems.kafka.default.stream.samza.offset.default=oldest
{% endhighlight %}
-For more details about Samza config, feel free to check out the latest config [here](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
+For more details on Samza's configs, feel free to check out the latest [configuration reference](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
### Run your application
-Let’s add a `main()` function to `WordCount` class first. The function reads the config file and factory from the args, and create a `LocalApplicationRunner` to run the application locally. Here is the function details:
+Let’s now add a `main()` function to the `WordCount` class. The function reads the config file and factory from the args, and creates a `LocalApplicationRunner` that run the application locally.
{% highlight java %}
public static void main(String[] args) {
@@ -189,28 +198,28 @@ apply plugin:'application'
mainClassName = "samzaapp.WordCount"
{% endhighlight %}
-Before running `main()`, we need to create the input Kafka topic with some sample data. Let’s start a local kafka broker first. Samza examples provides a script named “grid” which you can use to start zookeeper, kafka broker and yarn. Your can download it [here](https://github.com/apache/samza-hello-samza/blob/master/bin/grid) and put it under scripts/ folder, then issue the following command:
+Before running `main()`, we will create our input Kafka topic and populate it with sample data. You can download the scripts to interact with Kafka along with the sample data from [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
{% highlight bash %}
> ./scripts/grid install zookeeper && ./scripts/grid start zookeeper
> ./scripts/grid install kafka && ./scripts/grid start kafka
{% endhighlight %}
-Next we will create a Kafka topic named sample-text, and publish some sample data into it. A "sample-text.txt" file is included in the downloaded tutorial tgz file. In command line:
{% highlight bash %}
> ./deploy/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic sample-text --partition 1 --replication-factor 1
> ./deploy/kafka/bin/kafka-console-producer.sh --topic sample-text --broker localhost:9092 < ./sample-text.txt
{% endhighlight %}
-Now let’s fire up our application. Here we use gradle to run it. You can also run it directly within your IDE, with the same program arguments.
+Now let’s kick off our application and use gradle to run it. Alternately, you can also run it directly from your IDE, with the same program arguments.
{% highlight bash %}
> export BASE_DIR=`pwd`
> ./gradlew run --args="--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$BASE_DIR/src/main/config/word-count.properties"
{% endhighlight %}
-This application will output to a Kafka topic named "word-count-output". Let’s consume this topic to check out the results:
+
+The application will output to a Kafka topic named "word-count-output". Let’s fire up a Kafka consumer to read from this topic:
{% highlight bash %}
> ./deploy/kafka/bin/kafka-console-consumer.sh --topic word-count-output --zookeeper localhost:2181 --from-beginning
@@ -245,10 +254,10 @@ The [hello-samza](https://github.com/apache/samza-hello-samza) project contains
There are four main categories of examples in this project, including:
-1. [wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
+1. [Wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
-2. [cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
+2. [Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
-3. [asure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): this example shows how to run your application on Microsoft Asure.
+3. [Azure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): This example shows how to build an application that consumes input streams from Azure EventHubs.
-4. [kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): this example shows how to consume from Kinesis streams
\ No newline at end of file
+4. [Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): This example shows how to consume from Kinesis streams.
\ No newline at end of file
[21/50] samza git commit: clean-up the quickstart page
Posted by ja...@apache.org.
clean-up the quickstart page
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/cd34d98b
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/cd34d98b
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/cd34d98b
Branch: refs/heads/master
Commit: cd34d98bf638d84ec652f006844e05d343278d7a
Parents: 04a1276
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:10:38 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:16:13 2018 -0700
----------------------------------------------------------------------
docs/_docs/replace-versioned.sh | 5 ++-
docs/_menu/index.html | 2 +-
docs/startup/code-examples/versioned/index.md | 49 ++++++++++++++++++++++
docs/startup/quick-start/versioned/index.md | 26 +++++++-----
4 files changed, 70 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/cd34d98b/docs/_docs/replace-versioned.sh
----------------------------------------------------------------------
diff --git a/docs/_docs/replace-versioned.sh b/docs/_docs/replace-versioned.sh
index 24bf7ae..c454cac 100755
--- a/docs/_docs/replace-versioned.sh
+++ b/docs/_docs/replace-versioned.sh
@@ -44,4 +44,7 @@ echo "replaced startup/hello-samza/versioned to startup/hello-samza/"$version
mv -f $DIR/_site/startup/hello-samza/versioned $DIR/_site/startup/hello-samza/$version
echo "replaced startup/quick-start/versioned to startup/quick-start/"$version
-mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
\ No newline at end of file
+mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
+
+echo "replaced startup/code-examples/versioned to startup/code-examples/"$version
+mv -f $DIR/_site/startup/code-examples/versioned $DIR/_site/startup/code-examples/$version
http://git-wip-us.apache.org/repos/asf/samza/blob/cd34d98b/docs/_menu/index.html
----------------------------------------------------------------------
diff --git a/docs/_menu/index.html b/docs/_menu/index.html
index 0d1750f..a363bae 100644
--- a/docs/_menu/index.html
+++ b/docs/_menu/index.html
@@ -5,7 +5,7 @@ items:
- menu_title: QuickStart
url: /startup/quick-start/version/
- menu_title: Code Examples
- url: /learn/tutorials/version/
+ url: /startup/code-examples/version/
- menu_title: Documentation
has_sub: true
has_sub_subs: true
http://git-wip-us.apache.org/repos/asf/samza/blob/cd34d98b/docs/startup/code-examples/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/code-examples/versioned/index.md b/docs/startup/code-examples/versioned/index.md
new file mode 100644
index 0000000..ba1cc3e
--- /dev/null
+++ b/docs/startup/code-examples/versioned/index.md
@@ -0,0 +1,49 @@
+---
+layout: page
+title:
+---
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+
+### Checking out our examples
+
+The [hello-samza](https://github.com/apache/samza-hello-samza) project contains several examples to help you create your Samza applications. To checkout the hello-samza project:
+
+{% highlight bash %}
+> git clone https://git.apache.org/samza-hello-samza.git hello-samza
+{% endhighlight %}
+
+#### High-level API examples
+[The Samza Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook) contains various recipes using the Samza high-level API.
+These include:
+
+- The [Filter example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) demonstrates how to perform stateless operations on a stream.
+
+- The [Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/JoinExample.java]) demonstrates how you can join a Kafka stream of page-views with a stream of ad-clicks
+
+- The [Stream-Table Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/RemoteTableJoinExample.java) demonstrates how the Samza Table API. It joins a Kafka stream with a remote dataset accessed through a REST service.
+
+- The [SessionWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/SessionWindowExample.java) and [TumblingWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/TumblingWindowExample.java) examples illustrate Samza's rich windowing and triggering capabilities.
+
+
+In addition to the cookbook, you can also consult these:
+
+- [Wikipedia Parser](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): An advanced example that builds a streaming pipeline consuming a live-feed of wikipedia edits, parsing each message and generating statistics from them.
+
+
+- [Amazon Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis) and [Azure Eventhubs](https://github.com/apache/samza-hello-samza/tree/latest/src/main/java/samza/examples/azure) examples that cover how to consume input data from the respective systems.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/cd34d98b/docs/startup/quick-start/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/quick-start/versioned/index.md b/docs/startup/quick-start/versioned/index.md
index a046ee7..da20db2 100644
--- a/docs/startup/quick-start/versioned/index.md
+++ b/docs/startup/quick-start/versioned/index.md
@@ -65,7 +65,7 @@ public class WordCount implements StreamApplication {
The StreamApplication interface provides an API method named describe() for you to specify your streaming pipeline. Using [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html), you can describe your entire data processing task from data inputs, operations and outputs.
-### Input data source using Kafka
+### Describe your inputs and outputs
In this example, we are going to use Kafka as the input data source and consume the text for word count line by line. We start by defining a KafkaSystemDescriptor, which specifies the properties to establishing the connection to the local Kafka cluster. Then we create a `KafkaInputDescriptor`/`KafkaOutputDescriptor` to set up the topic, Serializer and Deserializer. Finally we use this input in the [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html) so we can consume from this topic. The code is in the following:
@@ -106,7 +106,7 @@ The resulting [MessageStream](/learn/documentation/{{site.version}}/api/javadocs
First we are going to extract the value from lines. This is a one-to-one transform and we can use the Samza map operator as following:
{% highlight java %}
-lines .map(kv -> kv.value)
+lines.map(kv -> kv.value)
{% endhighlight %}
Then we will split the line into words by using the flatmap operator:
@@ -152,23 +152,29 @@ lines
In this section we will configure the word count example to run locally in a single JVM. Please add a file named “word-count.properties” under the config folder. We will add the job configs in this file.
-Since there is only a single Samza processor, there is no coordination required. We use the PassthroughJobCoordinator for the example. We also group all Samza tasks into this single processor. As for the Kafka topic, we will consume from the beginning. Here is the full config needed for the job:
+In this section, we will configure our word count example to run locally in a single JVM. Let us add a file named “word-count.properties” under the config folder.
{% highlight jproperties %}
job.name=word-count
+# Use a PassthroughJobCoordinator since there is no coordination needed
job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory
job.coordination.utils.factory=org.apache.samza.standalone.PassthroughCoordinationUtilsFactory
+
job.changelog.system=kafka
+
+# Use a single container to process all of the data
task.name.grouper.factory=org.apache.samza.container.grouper.task.SingleContainerGrouperFactory
processor.id=0
+
+# Read from the beginning of the topic
systems.kafka.default.stream.samza.offset.default=oldest
{% endhighlight %}
-For more details about Samza config, feel free to check out the latest config [here](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
+For more details on Samza's configs, feel free to check out the latest [configuration reference](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
### Run your application
-Let’s add a `main()` function to `WordCount` class first. The function reads the config file and factory from the args, and create a `LocalApplicationRunner` to run the application locally. Here is the function details:
+Let’s now add a `main()` function to the `WordCount` class. The function reads the config file and factory from the args, and creates a `LocalApplicationRunner` that run the application locally.
{% highlight java %}
public static void main(String[] args) {
@@ -189,7 +195,7 @@ apply plugin:'application'
mainClassName = "samzaapp.WordCount"
{% endhighlight %}
-Before running `main()`, we need to create the input Kafka topic with some sample data. Let’s start a local kafka broker first. Samza examples provides a script named “grid” which you can use to start zookeeper, kafka broker and yarn. Your can download it [here](https://github.com/apache/samza-hello-samza/blob/master/bin/grid) and put it under scripts/ folder, then issue the following command:
+Before running `main()`, we will create our input Kafka topic and populate it with sample data. You can download the scripts to interact with Kafka along with the sample data from [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
{% highlight bash %}
> ./scripts/grid install zookeeper && ./scripts/grid start zookeeper
@@ -245,10 +251,10 @@ The [hello-samza](https://github.com/apache/samza-hello-samza) project contains
There are four main categories of examples in this project, including:
-1. [wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
+1. [Wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
-2. [cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
+2. [Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
-3. [asure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): this example shows how to run your application on Microsoft Asure.
+3. [Azure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): This example shows how to build an application that consumes input streams from Azure EventHubs.
-4. [kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): this example shows how to consume from Kinesis streams
\ No newline at end of file
+4. [Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): This example shows how to consume from Kinesis streams.
\ No newline at end of file
[05/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/75a53dbe
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/75a53dbe
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/75a53dbe
Branch: refs/heads/master
Commit: 75a53dbe8dd737f190ae1f420332a96aef32baab
Parents: 9e6697b 470a415
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 15 16:49:02 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 15 16:49:02 2018 -0700
----------------------------------------------------------------------
...ncing-the-release-of-apache-samza--0.13.1.md | 85 ++++++++++++++++++++
docs/_blog/2018-04-28-something-else.md | 36 ---------
...18-07-19-stream_processing-meetup-summary.md | 70 ++++++++++++++++
docs/_blog/2018-09-26-stangeloop-recap.md | 50 ++++++++++++
4 files changed, 205 insertions(+), 36 deletions(-)
----------------------------------------------------------------------
[15/50] samza git commit: Fix styling for LinkedIn logo;
Add support for loading logos from URLs
Posted by ja...@apache.org.
Fix styling for LinkedIn logo; Add support for loading logos from URLs
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/937d1de8
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/937d1de8
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/937d1de8
Branch: refs/heads/master
Commit: 937d1de845e838da8ea7b222ad745a1a2fe65a29
Parents: ce52acf
Author: Jagadish <jv...@linkedin.com>
Authored: Sun Oct 21 11:48:09 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Sun Oct 21 11:48:09 2018 -0700
----------------------------------------------------------------------
docs/_case-studies/linkedin.md | 1 +
docs/_layouts/default.html | 4 ++++
docs/css/main.new.css | 4 ----
3 files changed, 5 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/937d1de8/docs/_case-studies/linkedin.md
----------------------------------------------------------------------
diff --git a/docs/_case-studies/linkedin.md b/docs/_case-studies/linkedin.md
index 66e4108..f4dfe6b 100644
--- a/docs/_case-studies/linkedin.md
+++ b/docs/_case-studies/linkedin.md
@@ -5,6 +5,7 @@ title: Air Traffic Controller with Samza at LinkedIn
study_domain: linkedin.com # just the domain, not the protocol
priority: 4
menu_title: LinkedIn # what shows up in the menu
+logo_url: https://upload.wikimedia.org/wikipedia/commons/c/ca/LinkedIn_logo_initials.png
excerpt_separator: <!--more-->
---
<!--
http://git-wip-us.apache.org/repos/asf/samza/blob/937d1de8/docs/_layouts/default.html
----------------------------------------------------------------------
diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html
index 6c7d0c8..a3fa44b 100644
--- a/docs/_layouts/default.html
+++ b/docs/_layouts/default.html
@@ -206,7 +206,11 @@
<div class="section__item">
<a class="section__item-logo" href="{{ study.url }}" title="{{ study.menu_title }}">
+ {% if study.logo_url %}
+ <div class="section__item-logo-bg" style="background-image: url('{{ study.logo_url }}');"></div>
+ {% else %}
<div class="section__item-logo-bg" style="background-image: url('https://logo.clearbit.com/{{ study.study_domain}}?size=256');"></div>
+ {% endif %}
</a>
</div>
http://git-wip-us.apache.org/repos/asf/samza/blob/937d1de8/docs/css/main.new.css
----------------------------------------------------------------------
diff --git a/docs/css/main.new.css b/docs/css/main.new.css
index 468e10e..035fc7c 100644
--- a/docs/css/main.new.css
+++ b/docs/css/main.new.css
@@ -741,10 +741,6 @@ footer .side-by-side > * {
border-radius: 4px;
}
-.section__item-logo-bg {
- border-radius: 100%;
-}
-
.is-selected .section__item-logo-bg,
.section--brands .section__item:hover .section__item-logo-bg {
border-radius: 4px;
[50/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/6cdcdefd
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/6cdcdefd
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/6cdcdefd
Branch: refs/heads/master
Commit: 6cdcdefd448a197ae1634acf5ae8a33e7f15106a
Parents: 2d9836d 6f40414
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Nov 27 01:47:09 2018 -0800
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Nov 27 01:47:09 2018 -0800
----------------------------------------------------------------------
----------------------------------------------------------------------
[10/50] samza git commit: Minor: Fix a typo in architecture
documentation
Posted by ja...@apache.org.
Minor: Fix a typo in architecture documentation
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/e91223d1
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/e91223d1
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/e91223d1
Branch: refs/heads/master
Commit: e91223d1df61f5158027733030ac15cc2d844465
Parents: 19ededc
Author: Jagadish <jv...@linkedin.com>
Authored: Thu Oct 18 18:35:46 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Thu Oct 18 18:35:46 2018 -0700
----------------------------------------------------------------------
docs/learn/documentation/versioned/core-concepts/core-concepts.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/e91223d1/docs/learn/documentation/versioned/core-concepts/core-concepts.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/core-concepts/core-concepts.md b/docs/learn/documentation/versioned/core-concepts/core-concepts.md
index c1724fb..479ebcb 100644
--- a/docs/learn/documentation/versioned/core-concepts/core-concepts.md
+++ b/docs/learn/documentation/versioned/core-concepts/core-concepts.md
@@ -1,4 +1,4 @@
-.png
+---
layout: page
title: Core concepts
---
[36/50] samza git commit: Merge branch 'master' of
https://github.com/apache/samza
Posted by ja...@apache.org.
Merge branch 'master' of https://github.com/apache/samza
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/7d747819
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/7d747819
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/7d747819
Branch: refs/heads/master
Commit: 7d7478192fd2addc69f4dec2dec976e9be6301d8
Parents: d315cae ea0bec4
Author: Jagadish <jv...@linkedin.com>
Authored: Mon Oct 29 16:37:07 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Mon Oct 29 16:37:07 2018 -0700
----------------------------------------------------------------------
----------------------------------------------------------------------