You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by mm...@apache.org on 2013/02/25 18:15:36 UTC
[1/2] git commit: Add tuning documentation and update style
Add tuning documentation and update style
Project: http://git-wip-us.apache.org/repos/asf/incubator-s4/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-s4/commit/0634b5a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-s4/tree/0634b5a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-s4/diff/0634b5a0
Branch: refs/heads/S4-86
Commit: 0634b5a0929fa8008d209462fe5733580ad5bd33
Parents: 755ed6b
Author: Matthieu Morel <mm...@apache.org>
Authored: Sun Feb 24 16:43:59 2013 +0100
Committer: Matthieu Morel <mm...@apache.org>
Committed: Sun Feb 24 21:47:35 2013 +0100
----------------------------------------------------------------------
.gitignore | 1 -
website/Rules | 3 +
website/compass/config.rb | 1 +
website/content/doc/0.6.0/event_dispatch.md | 123 +++++++++++---
website/content/doc/0.6.0/fault_tolerance.md | 24 ++-
website/content/doc/0.6.0/overview.md | 32 ++--
website/content/doc/0.6.0/walkthrough.md | 130 +++++++++------
website/content/images/doc/0.6.0/executors.png | Bin 0 -> 58130 bytes
.../content/images/doc/0.6.0/sources/executors.odg | Bin 0 -> 11631 bytes
.../content/images/doc/0.6.0/sources/queues.odg | Bin 0 -> 10686 bytes
website/content/style/coderay.scss | 119 +++++++++++++
website/content/style/partials/_base.scss | 1 +
website/content/style/pygmentize.scss | 76 +++++++++
website/content/style/style.scss | 5 +-
website/layouts/default.haml | 2 +
15 files changed, 403 insertions(+), 114 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index a83d1b4..f42cc64 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,7 +7,6 @@
target/
build/
bin/
-doc/
tmp/
pepe/
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/Rules
----------------------------------------------------------------------
diff --git a/website/Rules b/website/Rules
index d69f9be..75ecf78 100644
--- a/website/Rules
+++ b/website/Rules
@@ -27,6 +27,9 @@ compile '/doc/*' do
# don’t filter binary items
else
filter :kramdown
+ filter :colorize_syntax,
+ :default_colorizer => :pygmentize,
+ :pygmentize => { :linenos => 'inline', :options => { :startinline => 'True' } }
layout 'default'
end
end
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/compass/config.rb
----------------------------------------------------------------------
diff --git a/website/compass/config.rb b/website/compass/config.rb
index 39317a1..04d9654 100644
--- a/website/compass/config.rb
+++ b/website/compass/config.rb
@@ -3,6 +3,7 @@ project_path = "."
css_dir = "output/style"
sass_dir = "content/style"
images_dir = "output/images"
+javascripts_dir = "assets/javascripts"
# when using SCSS:
sass_options = {
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/doc/0.6.0/event_dispatch.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/event_dispatch.md b/website/content/doc/0.6.0/event_dispatch.md
index ee61b23..7c12086 100644
--- a/website/content/doc/0.6.0/event_dispatch.md
+++ b/website/content/doc/0.6.0/event_dispatch.md
@@ -7,6 +7,7 @@ Events are dispatched according to their key.
The key is identified in an `Event` through a `KeyFinder`.
Dispatch can be configured for:
+
* dispatching events to partitions (_outgoing dispatch_)
* dispatching external events within a partition (_incoming dispatch_)
@@ -14,14 +15,18 @@ Dispatch can be configured for:
A stream can be defined with a KeyFinder, as :
-Stream<TopicEvent> topicSeenStream = createStream("TopicSeen", new KeyFinder<TopicEvent>() {
+~~~
- @Override
- public List<String> get(final TopicEvent arg0) {
- return ImmutableList.of(arg0.getTopic());
- }
- }, topicCountAndReportPE);
+#!java
+Stream<TopicEvent> topicSeenStream = createStream("TopicSeen", new KeyFinder<TopicEvent>() {
+ @Override
+ public List<String> get(final TopicEvent arg0) {
+ return ImmutableList.of(arg0.getTopic());
+ }
+ }, topicCountAndReportPE);
+
+~~~
When an event is sent to the "TopicSeen" stream, its key will be identified through the KeyFinder implementation, hashed and dispatched to the matching partition.
@@ -30,48 +35,110 @@ The same logic applies when defining _output streams_.
If we use an AdapterApp subclass, the `remoteStreamKeyFinder` should be defined in the `onInit()` method, _before_ calling `super.onInit()`:
- @Override
- protected void onInit() {
- ...
- remoteStreamKeyFinder = new KeyFinder<Event>() {
-
- @Override
- public List<String> get(Event event) {
- return ImmutableList.of(event.get("theKeyField"));
- }
- };
- super.onInit()
- ...
+~~~
+#!java
+ @Override
+ protected void onInit() {
+ ...
+ remoteStreamKeyFinder = new KeyFinder<Event>() {
+ @Override
+ public List<String> get(Event event) {
+ return ImmutableList.of(event.get("theKeyField"));
+ }
+ };
+ super.onInit()
+ ...
+~~~
If we use a standard App, we use the `createOutputStream(String name, KeyFinder<Event> keyFinder)` method.
-bq. If the KeyFinder is not defined for the output streams, events are sent to partitions of the connected cluster in a round robin fashion.
+> If the KeyFinder is not defined for the output streams, events are sent to partitions of the connected cluster in a round robin fashion.
# Incoming dispatch from external events
When receiving events from a remote application, we _must_ define how external events are dispatched internally, to which PEs and based on which keys. For that purpose, we simply define and _input stream_ with the corresponding KeyFinder:
-createInputStream("names", new KeyFinder<Event>() {
+~~~
+#!java
- @Override
- public List<String> get(Event event) {
- return Arrays.asList(new String[] { event.get("name") });
- }
- }, helloPE);
+createInputStream("names", new KeyFinder<Event>() {
+ @Override
+ public List<String> get(Event event) {
+ return Arrays.asList(new String[] { event.get("name") });
+ }
+ }, helloPE);
+~~~
In this case, a name is extracted from each event, the PE instance with this key is retrieved or created, and the event sent to that instance.
Alternatively, we can use a unique PE instance for processing events in a given node. For that we simply define the input stream without a KeyFinder, _and_ use a singleton PE:
- HelloPE helloPE = createPE(HelloPE.class);
- helloPE.setSingleton(true);
- createInputStream("names", helloPE);
+~~~
+#!java
+HelloPE helloPE = createPE(HelloPE.class);
+helloPE.setSingleton(true);
+createInputStream("names", helloPE);
+~~~
In this case, all events will be dispatched to the only HelloPE instance in this partition, regardless of the content of the event.
+
+# Internals and tuning
+
+S4 follows a staged event driven architecture and uses a pipeline of executors to process messages.
+
+## executors
+An executor is an object that executes tasks. It usually keeps a bounded queue of task items and schedules their execution through a pool of threads.
+
+When processing queues are full, executors may adopt various possible behaviours, in particular, in S4:
+ * **blocking**: the current thread simply waits until the queue is not full
+ * **shedding**: the current event is dropped
+
+**Throttling**, i.e. placing an upper bound on the maximum processing rate, is a convenient way to avoid sending too many messages too fast.
+
+S4 provides various default implementations of these behaviours and you can also define your own custom executors as appropriate.
+
+## workflow
+
+The following picture illustrates the pipeline of executors.
+
+
+
+### When a node receives a message:
+
+1. data is received on a socket and chunked into a message, in the form of an array of bytes
+1. the message is passed to a deserializer executor
+ * this executor is loaded with the application, and therefore has access to application classes, so that application specific messages can be deserialized
+ * by default it uses 1 thread and **blocks** if the processing queue is full
+1. the event (deserialized message) is dispatched to a stream executor
+ * the stream executor is selected according to the stream information contained in the event
+ * by default it **blocks** if the processing queue is full
+1. the event is processed in the PE instance that matches the key of the event
+
+### When a PE emits a message:
+
+1. an event is passed to a referenced stream
+1. if the target cluster is remote, the event is passed to a remote sender executor
+1. otherwise,
+ * if the target partition is the current one, the event is directly passed to the corresponding stream executor (see step 3 above)
+ * otherwise, the event is passed to a local sender executor
+1. remote or local sender executors serialize the event into an array of bytes
+ * remote sender executors are **blocking** by default, if their processing queue is full
+ * local sender executors are **throttling** by default, with a configurable maximum rate. If events arrive at a higher rate, they are **dropped**.
+
+
+## configuration parameters
+
+* blocking executors can lead to deadlocks, depending on the application graph
+* back pressure can be taken into account when using TCP: if downstream systems saturate, messages cannot be sent downstream, and sending queues fill up. With knowledge of the application, is is possible to add some mechanisms to react appropriately
+* executors can be replaced by other implementations in custom modules, by overriding the appropriate bindings (see `DefaultCommModule` and `DefaultCodeModule`)
+* the maximum number of threads to use to process a given stream can be specified **in the application**, using the `setParallelism()` method of `Stream`
+* default parameters are specified in `default.s4.comm.properties`
+
+
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/doc/0.6.0/fault_tolerance.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/fault_tolerance.md b/website/content/doc/0.6.0/fault_tolerance.md
index 241374c..9086582 100644
--- a/website/content/doc/0.6.0/fault_tolerance.md
+++ b/website/content/doc/0.6.0/fault_tolerance.md
@@ -4,7 +4,7 @@ title: Fault tolerance
Stream processing applications are typically long running applications, and they may accumulate state over extended periods of time.
-Running a distributed system over a long period of time implies there will be:
+When running a distributed system over a long period of time, expect:
- failures
@@ -138,21 +138,25 @@ We provide a default module (FileSystemBackendCheckpointingModule) that uses a f
##### Customizing the checkpointing backend
-It is quite straightforward to implement backends for other kinds of storage (key value stores, datagrid, cache, RDBMS). Using an alternative backend is as simple as providing a new module to the S4 node. Here is an example of a module using a 'Cool' backend implementation:
+It is quite straightforward to implement backends for other kinds of storage (key value stores, datagrid, cache, RDBMS). Writing a checkpointing backend consists of implementing a simple interface (`StateStorage`) matching your infrastructure or system.
+ Using an alternative backend is as simple as providing a new module to the S4 node. Here is an example of a module using a 'Cool' backend implementation:
- public class CoolBackendCheckpointingModule extends AbstractModule {
- @Override
- protected void configure() {
- bind(StateStorage.class).to(CoolStateStorage.class);
- bind(CheckpointingFramework.class).to(SafeKeeper.class);
- }
- }
+~~~
+#!java
+public class CoolBackendCheckpointingModule extends AbstractModule {
+ @Override
+ protected void configure() {
+ bind(StateStorage.class).to(CoolStateStorage.class);
+ bind(CheckpointingFramework.class).to(SafeKeeper.class);
+ }
+}
+~~~
##### Overriding checkpointing and recovery operations
-By default, S4 uses [kryo](http://code.google.com/p/kryo) to serialize and deserialize checkpoints, but it is possible to use a different mechanism, by overriding the `checkpoint()`, `serializeState()` and `restoreState()` methods of the `ProcessingElement` class.
+By default, S4 keeps all non transient fields as part of the state, and uses [kryo](http://code.google.com/p/kryo) to serialize and deserialize checkpoints, but it is possible to use a different mechanism, by overriding the `checkpoint()`, `serializeState()` and `restoreState()` methods of the `ProcessingElement` class.
PEs are eligible for checkpointing when their state is 'dirty'. The dirty flag is checked through the `isDirty()` method, and cleared by calling the `clearDirty()` method. In some cases, dependent on the application code, only some of the events may actually change the state of the PE. You should override these methods in order to avoid unjustified checkpointing operations.
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/doc/0.6.0/overview.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/overview.md b/website/content/doc/0.6.0/overview.md
index 709795b..44eeb2e 100644
--- a/website/content/doc/0.6.0/overview.md
+++ b/website/content/doc/0.6.0/overview.md
@@ -13,9 +13,9 @@ S4 0.5 focused on providing a functional complete refactoring.
S4 0.6 builds on this basis and brings plenty of exciting features, in particular:
-* *performance improvements*: stream throughput improved by 1000 % (~200k messages / s / stream)
+* **performance improvements**: stream throughput improved by 1000 % (~200k messages / s / stream)
* improved [configurability](S4:Configuration - 0.6.0], for both the S4 platform and deployed applications
-* *elasticity* and fine partition tuning, through an integration with Apache Helix
+* **elasticity** and fine partition tuning, through an integration with Apache Helix
# What are the cool features?
@@ -23,30 +23,30 @@ S4 0.6 builds on this basis and brings plenty of exciting features, in particula
**Flexible deployment**:
* By default keys are homogeneously sparsed over the cluster: helps balance the load, especially for fine grained partitioning
-* S4 also provides fine control over the partitioning
+* S4 also provides fine control over the partitioning (with Apache Helix)
* Features automatic rebalancing
**Modular design**:
* both the platform and the applications are built by dependency injection, and configured through independent modules.
-* makes it easy to customize the system according to specific requirements
-* pluggable event serving policies: load shedding, throttling, blocking
+* makes it **easy to customize** the system according to specific requirements
+* pluggable event serving policies: **load shedding, throttling, blocking**
**Dynamic and loose coupling of S4 applications**:
* through a pub-sub mechanism
* makes it easy to:
-** assemble subsystems into larger systems
-** reuse applications
-** separate pre-processing
-** provision, control and update subsystems independently
+ * assemble subsystems into larger systems
+ * reuse applications
+ * separate pre-processing
+ * provision, control and update subsystems independently
**[Fault tolerant](fault_tolerance)**
-* *Fail-over* mechanism for high availability
-* *Checkpointing and recovery* mechanism for minimizing state loss
+* **Fail-over** mechanism for high availability
+* **Checkpointing and recovery** mechanism for minimizing state loss
**Pure Java**: statically typed, easy to understand, to refactor, and to extend
@@ -68,19 +68,15 @@ S4 0.6 builds on this basis and brings plenty of exciting features, in particula
**Applications**
-
* Users develop applications and deploy them on S4 clusters
* Applications are built from:
-** *Processing elements* (PEs)
-** *Streams* that interconnect PEs
-
-* PEs communicate asynchronously by sending *events* on streams.
+ * **Processing elements** (PEs)
+ * **Streams** that interconnect PEs
+* PEs communicate asynchronously by sending **events** on streams.
* Events are dispatched to nodes according to their key
**External streams** are a special kind of stream that:
-
-
* send events outside of the application
* receive events from external sources
* can interconnect and assemble applications into larger systems.
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/doc/0.6.0/walkthrough.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/walkthrough.md b/website/content/doc/0.6.0/walkthrough.md
index f819331..437f580 100644
--- a/website/content/doc/0.6.0/walkthrough.md
+++ b/website/content/doc/0.6.0/walkthrough.md
@@ -57,45 +57,55 @@ S4 provides some scripts in order to simplify development and testing of applica
The src/main/java/hello directory contains 3 files:
* HelloPE.java : a very simple PE that simply prints the name contained in incoming events
- // ProcessingElement provides integration with the S4 platform
- public class HelloPE extends ProcessingElement {
-
- // you should define downstream streams here and inject them in the app definition
-
- // PEs can maintain some state
- boolean seen = false;
-
- // This method is called upon a new Event on an incoming stream.
- // You may overload it for handling instances of your own specialized subclasses of Event
- public void onEvent(Event event) {
- System.out.println("Hello " + (seen ? "again " : "") + event.get("name") + "!");
- seen = true;
- }
- // skipped remaining methods
+
+~~~
+
+#!java
+
+// ProcessingElement provides integration with the S4 platform
+public class HelloPE extends ProcessingElement {
+ // you should define downstream streams here and inject them in the app definition
+
+ // PEs can maintain some state
+ boolean seen = false;
+
+ // This method is called upon a new Event on an incoming stream.
+ // You may overload it for handling instances of your own specialized subclasses of Event
+ public void onEvent(Event event) {
+ System.out.println("Hello " + (seen ? "again " : "") + event.get("name") + "!");
+ seen = true;
+ }
+// skipped remaining methods
+~~~
* HelloApp.java: defines a simple application: exposes an input stream ("names"), connected to the HelloPE. See [the event dispatch configuration page](event_dispatch) for more information about how events are dispatched.
// App parent class provides integration with the S4 platform
public class HelloApp extends App {
- @Override
- protected void onStart() {
- }
-
- @Override
- protected void onInit() {
- // That's where we define PEs and streams
- // create a prototype
- HelloPE helloPE = createPE(HelloPE.class);
- // Create a stream that listens to the "lines" stream and passes events to the helloPE instance.
- createInputStream("names", new KeyFinder<Event>() {
- // the KeyFinder is used to identify keys
- @Override
- public List<String> get(Event event) {
- return Arrays.asList(new String[] { event.get("name") });
- }
- }, helloPE);
- }
- // skipped remaining methods
+~~~
+
+#!java
+
+@Override
+protected void onStart() {
+}
+
+@Override
+protected void onInit() {
+ // That's where we define PEs and streams
+ // create a prototype
+ HelloPE helloPE = createPE(HelloPE.class);
+ // Create a stream that listens to the "lines" stream and passes events to the helloPE instance.
+ createInputStream("names", new KeyFinder<Event>() {
+ // the KeyFinder is used to identify keys
+ @Override
+ public List<String> get(Event event) {
+ return Arrays.asList(new String[] { event.get("name") });
+ }
+ }, helloPE);
+}
+// skipped remaining methods
+~~~
* HelloInputAdapter is a simple adapter that reads character lines from a socket, converts them into events, and sends the events to interested S4 apps, through the "names" stream
@@ -112,37 +122,51 @@ In order to run an S4 application, you need :
* In 2 steps:
1. Start a Zookeeper server instance (-clean option removes previous ZooKeeper data, if any):
+
+
S4:incubator-s4$ ./s4 zkServer - clean
S4:myApp$ calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
[main] INFO org.apache.s4.tools.ZKServer - Starting zookeeper server on port [2181]
[main] INFO org.apache.s4.tools.ZKServer - cleaning existing data in [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/data] and [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/log]
- 1. Define a new cluster. Say a cluster named "cluster1" with 2 partitions, nodes listening to ports starting from 12000:
- S4:myApp$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
- calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
- [main] INFO org.apache.s4.tools.DefineCluster - preparing new cluster [cluster1] with [2] node(s)
- [main] INFO org.apache.s4.tools.DefineCluster - New cluster configuration uploaded into zookeeper
+
+1. Define a new cluster. Say a cluster named "cluster1" with 2 partitions, nodes listening to ports starting from 12000:
+
+ S4:myApp$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
+ calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
+ [main] INFO org.apache.s4.tools.DefineCluster - preparing new cluster [cluster1] with [2] node(s)
+ [main] INFO org.apache.s4.tools.DefineCluster - New cluster configuration uploaded into zookeeper
* Alternatively you may combine these two steps into a single one, by passing the cluster configuration inline with the `zkServer` command:
+
+~~~
+#!bash
- S4:incubator-s4$ ./s4 zkServer -clusters=c=cluster1:flp=12000:nbTasks=2 -clean
+S4:incubator-s4$ ./s4 zkServer -clusters=c=cluster1:flp=12000:nbTasks=2 -clean
+~~~
* Start 2 S4 nodes with the default configuration, and attach them to cluster "cluster1" :
- S4:myApp$ ./s4 node -c=cluster1
- calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
- 15:50:18.996 [main] INFO org.apache.s4.core.Main - Initializing S4 node with :
- - comm module class [org.apache.s4.comm.DefaultCommModule]
- - comm configuration file [default.s4.comm.properties from classpath]
- - core module class [org.apache.s4.core.DefaultCoreModule]
- - core configuration file[default.s4.core.properties from classpath]
- -extra modules: []
- [main] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to
+~~~
+#!bash
+
+S4:myApp$ ./s4 node -c=cluster1
+calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
+15:50:18.996 [main] INFO org.apache.s4.core.Main - Initializing S4 node with :
+- comm module class [org.apache.s4.comm.DefaultCommModule]
+- comm configuration file [default.s4.comm.properties from classpath]
+- core module class [org.apache.s4.core.DefaultCoreModule]
+- core configuration file[default.s4.core.properties from classpath]
+-extra modules: []
+[main] INFO org.apache.s4.core.Main - Starting S4 node.
+~~~
+
+This node will automatically download applications published for the cluster it belongs to
and again (maybe in another shell):
- S4:myApp$ ./s4 node -c=cluster1
+ S4:myApp$ ./s4 node -c=cluster1
* Build, package and publish the app to cluster1:
* This is done in 2 separate steps:
@@ -250,9 +274,9 @@ Have a look at the code in these directories. You'll note that:
> Note: You need a twitter4j.properties file in your home directory with the following content (debug is optional):
- debug=true
- user=<a twitter username>
- password=<matching password>
+ debug=true
+ user=<a twitter username>
+ password=<matching password>
* Start a Zookeeper instance. From the S4 base directory, do:
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/images/doc/0.6.0/executors.png
----------------------------------------------------------------------
diff --git a/website/content/images/doc/0.6.0/executors.png b/website/content/images/doc/0.6.0/executors.png
new file mode 100644
index 0000000..e654eb4
Binary files /dev/null and b/website/content/images/doc/0.6.0/executors.png differ
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/images/doc/0.6.0/sources/executors.odg
----------------------------------------------------------------------
diff --git a/website/content/images/doc/0.6.0/sources/executors.odg b/website/content/images/doc/0.6.0/sources/executors.odg
new file mode 100644
index 0000000..0fc69fa
Binary files /dev/null and b/website/content/images/doc/0.6.0/sources/executors.odg differ
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/images/doc/0.6.0/sources/queues.odg
----------------------------------------------------------------------
diff --git a/website/content/images/doc/0.6.0/sources/queues.odg b/website/content/images/doc/0.6.0/sources/queues.odg
new file mode 100644
index 0000000..b5c1b85
Binary files /dev/null and b/website/content/images/doc/0.6.0/sources/queues.odg differ
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/style/coderay.scss
----------------------------------------------------------------------
diff --git a/website/content/style/coderay.scss b/website/content/style/coderay.scss
new file mode 100644
index 0000000..82d083a
--- /dev/null
+++ b/website/content/style/coderay.scss
@@ -0,0 +1,119 @@
+.CodeRay {
+ background-color: hsl(0,0%,95%);
+ border: 1px solid silver;
+ color: black;
+}
+.CodeRay pre {
+ margin: 0px;
+}
+
+span.CodeRay { white-space: pre; border: 0px; padding: 2px; }
+
+table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px; }
+table.CodeRay td { padding: 2px 4px; vertical-align: top; }
+
+.CodeRay .line-numbers {
+ color: gray;
+ text-align: right;
+ -webkit-user-select: none;
+ -moz-user-select: none;
+ user-select: none;
+}
+.CodeRay .line-numbers a {
+ color: gray !important;
+ text-decoration: none !important;
+ padding-right: 5px;
+}
+.CodeRay .line-numbers a:target
+.CodeRay .line-numbers .highlighted
+.CodeRay .line-numbers .highlighted a
+.CodeRay span.line-numbers { padding: 0px 4px; }
+.CodeRay .line { display: block; float: left; width: 100%; }
+.CodeRay .code { width: 100%; }
+
+.CodeRay .debug { color: white !important; background: blue !important; }
+
+.CodeRay .annotation { color:#007 }
+.CodeRay .attribute-name { color:#b48 }
+.CodeRay .attribute-value { color:#700 }
+.CodeRay .binary { color:#509 }
+.CodeRay .char .content { color:#D20 }
+.CodeRay .char .delimiter { color:#710 }
+.CodeRay .char { color:#D20 }
+.CodeRay .class { color:#B06; font-weight:bold }
+.CodeRay .class-variable { color:#369 }
+.CodeRay .color { color:#0A0 }
+.CodeRay .comment { color:#777 }
+.CodeRay .comment .char { color:#444 }
+.CodeRay .comment .delimiter { color:#444 }
+.CodeRay .complex { color:#A08 }
+.CodeRay .constant { color:#036; font-weight:bold }
+.CodeRay .decorator { color:#B0B }
+.CodeRay .definition { color:#099; font-weight:bold }
+.CodeRay .delimiter { color:black }
+.CodeRay .directive { color:#088; font-weight:bold }
+.CodeRay .doc { color:#970 }
+.CodeRay .doc-string { color:#D42; font-weight:bold }
+.CodeRay .doctype { color:#34b }
+.CodeRay .entity { color:#800; font-weight:bold }
+.CodeRay .error { color:#F00; background-color:#FAA }
+.CodeRay .escape { color:#666 }
+.CodeRay .exception { color:#C00; font-weight:bold }
+.CodeRay .float { color:#60E }
+.CodeRay .function { color:#06B; font-weight:bold }
+.CodeRay .global-variable { color:#d70 }
+.CodeRay .hex { color:#02b }
+.CodeRay .imaginary { color:#f00 }
+.CodeRay .include { color:#B44; font-weight:bold }
+.CodeRay .inline { background-color: hsla(0,0%,0%,0.07); color: black }
+.CodeRay .inline-delimiter { font-weight: bold; color: #666 }
+.CodeRay .instance-variable { color:#33B }
+.CodeRay .integer { color:#00D }
+.CodeRay .key .char { color: #60f }
+.CodeRay .key .delimiter { color: #404 }
+.CodeRay .key { color: #606 }
+.CodeRay .keyword { color:#080; font-weight:bold }
+.CodeRay .label { color:#970; font-weight:bold }
+.CodeRay .local-variable { color:#963 }
+.CodeRay .namespace { color:#707; font-weight:bold }
+.CodeRay .octal { color:#40E }
+.CodeRay .operator { }
+.CodeRay .predefined { color:#369; font-weight:bold }
+.CodeRay .predefined-constant { color:#069 }
+.CodeRay .predefined-type { color:#0a5; font-weight:bold }
+.CodeRay .preprocessor { color:#579 }
+.CodeRay .pseudo-class { color:#00C; font-weight:bold }
+.CodeRay .regexp .content { color:#808 }
+.CodeRay .regexp .delimiter { color:#404 }
+.CodeRay .regexp .modifier { color:#C2C }
+.CodeRay .regexp { background-color:hsla(300,100%,50%,0.06); }
+.CodeRay .reserved { color:#080; font-weight:bold }
+.CodeRay .shell .content { color:#2B2 }
+.CodeRay .shell .delimiter { color:#161 }
+.CodeRay .shell { background-color:hsla(120,100%,50%,0.06); }
+.CodeRay .string .char { color: #b0b }
+.CodeRay .string .content { color: #D20 }
+.CodeRay .string .delimiter { color: #710 }
+.CodeRay .string .modifier { color: #E40 }
+.CodeRay .string { background-color:hsla(0,100%,50%,0.05); }
+.CodeRay .symbol .content { color:#A60 }
+.CodeRay .symbol .delimiter { color:#630 }
+.CodeRay .symbol { color:#A60 }
+.CodeRay .tag { color:#070 }
+.CodeRay .type { color:#339; font-weight:bold }
+.CodeRay .value { color: #088; }
+.CodeRay .variable { color:#037 }
+
+.CodeRay .insert { background: hsla(120,100%,50%,0.12) }
+.CodeRay .delete { background: hsla(0,100%,50%,0.12) }
+.CodeRay .change { color: #bbf; background: #007; }
+.CodeRay .head { color: #f8f; background: #505 }
+.CodeRay .head .filename { color: white; }
+
+.CodeRay .delete .eyecatcher { background-color: hsla(0,100%,50%,0.2); border: 1px solid hsla(0,100%,45%,0.5); margin: -1px; border-bottom: none; border-top-left-radius: 5px; border-top-right-radius: 5px; }
+.CodeRay .insert .eyecatcher { background-color: hsla(120,100%,50%,0.2); border: 1px solid hsla(120,100%,25%,0.5); margin: -1px; border-top: none; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; }
+
+.CodeRay .insert .insert { color: #0c0; background:transparent; font-weight:bold }
+.CodeRay .delete .delete { color: #c00; background:transparent; font-weight:bold }
+.CodeRay .change .change { color: #88f }
+.CodeRay .head .head { color: #f4f }
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/style/partials/_base.scss
----------------------------------------------------------------------
diff --git a/website/content/style/partials/_base.scss b/website/content/style/partials/_base.scss
index f7012ea..4106fe7 100644
--- a/website/content/style/partials/_base.scss
+++ b/website/content/style/partials/_base.scss
@@ -9,3 +9,4 @@ $blueprint-grid-margin: 10px;
// If you change your grid column dimensions
// you can make a new grid background image from the command line like this:
// compass grid-img 30+10x16
+
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/style/pygmentize.scss
----------------------------------------------------------------------
diff --git a/website/content/style/pygmentize.scss b/website/content/style/pygmentize.scss
new file mode 100644
index 0000000..c60e12b
--- /dev/null
+++ b/website/content/style/pygmentize.scss
@@ -0,0 +1,76 @@
+.highlight { background-color: #ffffcc }
+.c { color: #999988; font-style: italic } /* Comment */
+.err { color: #a61717; background-color: #e3d2d2 } /* Error */
+.k { color: #000000; font-weight: bold } /* Keyword */
+.o { color: #000000; font-weight: bold } /* Operator */
+.cm { color: #999988; font-style: italic } /* Comment.Multiline */
+.cp { color: #999999; font-weight: bold; font-style: italic } /* Comment.Preproc */
+.c1 { color: #999988; font-style: italic } /* Comment.Single */
+.cs { color: #999999; font-weight: bold; font-style: italic } /* Comment.Special */
+.gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */
+.ge { color: #000000; font-style: italic } /* Generic.Emph */
+.gr { color: #aa0000 } /* Generic.Error */
+.gh { color: #999999 } /* Generic.Heading */
+.gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
+.go { color: #888888 } /* Generic.Output */
+.gp { color: #555555 } /* Generic.Prompt */
+.gs { font-weight: bold } /* Generic.Strong */
+.gu { color: #aaaaaa } /* Generic.Subheading */
+.gt { color: #aa0000 } /* Generic.Traceback */
+.kc { color: #000000; font-weight: bold } /* Keyword.Constant */
+.kd { color: #000000; font-weight: bold } /* Keyword.Declaration */
+.kn { color: #000000; font-weight: bold } /* Keyword.Namespace */
+.kp { color: #000000; font-weight: bold } /* Keyword.Pseudo */
+.kr { color: #000000; font-weight: bold } /* Keyword.Reserved */
+.kt { color: #445588; font-weight: bold } /* Keyword.Type */
+.m { color: #009999 } /* Literal.Number */
+.s { color: #d01040 } /* Literal.String */
+.na { color: #008080 } /* Name.Attribute */
+.nb { color: #0086B3 } /* Name.Builtin */
+.nc { color: #445588; font-weight: bold } /* Name.Class */
+.no { color: #008080 } /* Name.Constant */
+.nd { color: #3c5d5d; font-weight: bold } /* Name.Decorator */
+.ni { color: #800080 } /* Name.Entity */
+.ne { color: #990000; font-weight: bold } /* Name.Exception */
+.nf { color: #990000; font-weight: bold } /* Name.Function */
+.nl { color: #990000; font-weight: bold } /* Name.Label */
+.nn { color: #555555 } /* Name.Namespace */
+.nt { color: #000080 } /* Name.Tag */
+.nv { color: #008080 } /* Name.Variable */
+.ow { color: #000000; font-weight: bold } /* Operator.Word */
+.w { color: #bbbbbb } /* Text.Whitespace */
+.mf { color: #009999 } /* Literal.Number.Float */
+.mh { color: #009999 } /* Literal.Number.Hex */
+.mi { color: #009999 } /* Literal.Number.Integer */
+.mo { color: #009999 } /* Literal.Number.Oct */
+.sb { color: #d01040 } /* Literal.String.Backtick */
+.sc { color: #d01040 } /* Literal.String.Char */
+.sd { color: #d01040 } /* Literal.String.Doc */
+.s2 { color: #d01040 } /* Literal.String.Double */
+.se { color: #d01040 } /* Literal.String.Escape */
+.sh { color: #d01040 } /* Literal.String.Heredoc */
+.si { color: #d01040 } /* Literal.String.Interpol */
+.sx { color: #d01040 } /* Literal.String.Other */
+.sr { color: #009926 } /* Literal.String.Regex */
+.s1 { color: #d01040 } /* Literal.String.Single */
+.ss { color: #990073 } /* Literal.String.Symbol */
+.bp { color: #999999 } /* Name.Builtin.Pseudo */
+.vc { color: #008080 } /* Name.Variable.Class */
+.vg { color: #008080 } /* Name.Variable.Global */
+.vi { color: #008080 } /* Name.Variable.Instance */
+.il { color: #009999 } /* Literal.Number.Integer.Long */
+
+
+pre {
+ counter-reset: line-numbering;
+ border: solid 3px #d9d9d9;
+ border-radius: 5px;
+ background: #fff;
+ padding: 5px;
+ line-height: 23px;
+ margin-bottom: 30px;
+ white-space: pre;
+ overflow-x: auto;
+ word-break: inherit;
+ word-wrap: inherit;
+}
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/content/style/style.scss
----------------------------------------------------------------------
diff --git a/website/content/style/style.scss b/website/content/style/style.scss
index 19a18fb..e2b3649 100644
--- a/website/content/style/style.scss
+++ b/website/content/style/style.scss
@@ -14,10 +14,7 @@ body {
font-size: 90%;
background-color: $bg_body;
color: #333;
- pre {
- background-color: $dark_bg;
- overflow: auto;
- }
+
}
a {
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/0634b5a0/website/layouts/default.haml
----------------------------------------------------------------------
diff --git a/website/layouts/default.haml b/website/layouts/default.haml
index 7cee573..43f66ff 100644
--- a/website/layouts/default.haml
+++ b/website/layouts/default.haml
@@ -11,6 +11,8 @@
%link{:rel=>"stylesheet", :type=>"text/css", :href=>"/style/style.css"}
%link{:rel=>"stylesheet", :type=>"text/css", :href=>"/style/nav.css"}
= google_analytics(@config[:google_analytics_account_id], @config[:google_analytics_domain])
+ %link{:rel => 'stylesheet', :type => 'text/css', :href => '/style/coderay.css'}
+ %link{:rel => 'stylesheet', :type => 'text/css', :href => '/style/pygmentize.css'}
%body
#header
.container