You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficcontrol.apache.org by oc...@apache.org on 2020/08/31 20:22:54 UTC
[trafficcontrol] branch master updated: New fakeOrigin testing tool
(#3567)
This is an automated email from the ASF dual-hosted git repository.
ocket8888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficcontrol.git
The following commit(s) were added to refs/heads/master by this push:
new 3bf2acf New fakeOrigin testing tool (#3567)
3bf2acf is described below
commit 3bf2acf72149f3f1ede089e52c6840787ccb6b3a
Author: Jonathan G <jh...@users.noreply.github.com>
AuthorDate: Mon Aug 31 14:22:45 2020 -0600
New fakeOrigin testing tool (#3567)
* Add fakeOrigin core
* Add fakeOrigin m3u8 library
* Updated Markdown licensing per feedback for consistency with the rest of the project.
* Shell script fix due to order in license insertion
* Removed rpm group per PR feedback as it appears to have been deprecated prior to CentOS7
* Cleanup of specfile per PR feedback
* Switch to string literals to make syntax highlighters happier per PR feedback
* Reordered to improve docker layer caching per PR feedback
* Consolidated docker layer caching per PR feedback
* Convert http bodies from slice bytes to bytes.buffer per PR feedback
* Convert int range request sizes to uint64 per PR feedback
* Added usage and required params to example external transcoder script
* Docker layer caching optimizations per PR feedback
* Convert implicit rpm dependency to explicit per PR feedback
* Change example syntax highlighter per PR feedback
* Wording change in documentation per PR feedback.
* Documentation nitpick fix per PR feedback
* Fix for stray character in example crossdomain.xml
* Fix ordering of license information in crossdomain.xml
* Add missing status code return in service
* Fix string formatting per PR feedback
* Improve error handling in case something goes wrong while resolving relative paths
* Fix file permissions per PR feedback
* Update config file write permissions per PR feedback
* Update posix compliance to config file output per PR feedback
* License Header fix per PR feedback
* Update gitignore for cert extension
* Bugfix for client header override capitalization
* Add missing license header
* Update Dockerignore file licensing
* Add more explicit public domain licensing for weasel
Co-authored-by: Robert Butts <ro...@users.noreply.github.com>
---
LICENSE | 4 +
test/fakeOrigin/.dockerignore | 27 +++
test/fakeOrigin/.gitignore | 29 +++
test/fakeOrigin/Dockerfile | 42 ++++
test/fakeOrigin/Dockerfile.build_rpm | 39 ++++
test/fakeOrigin/README.md | 74 ++++++
test/fakeOrigin/build/README.md | 35 +++
test/fakeOrigin/build/build_rpm.sh | 94 ++++++++
test/fakeOrigin/build/config.json | 24 ++
test/fakeOrigin/build/fakeOrigin.init | 121 ++++++++++
test/fakeOrigin/build/fakeOrigin.logrotate | 26 +++
test/fakeOrigin/build/fakeOrigin.spec | 70 ++++++
test/fakeOrigin/docker-compose.build_rpm.yml | 33 +++
test/fakeOrigin/docker-compose.yml | 36 +++
test/fakeOrigin/docs/Configuration.md | 140 ++++++++++++
test/fakeOrigin/docs/Endpoint.Examples.md | 223 ++++++++++++++++++
test/fakeOrigin/endpoint/endpoint.go | 216 ++++++++++++++++++
test/fakeOrigin/endpoint/endpoint_enums.go | 54 +++++
test/fakeOrigin/endpoint/type_jsonenums.go | 87 +++++++
test/fakeOrigin/example/crossdomain.xml | 20 ++
test/fakeOrigin/example/video/LICENSE | 5 +
test/fakeOrigin/example/video/kelloggs.mp4 | Bin 0 -> 2050262 bytes
test/fakeOrigin/fakeOrigin.go | 95 ++++++++
test/fakeOrigin/httpService/filter.go | 287 +++++++++++++++++++++++
test/fakeOrigin/httpService/handler.go | 178 +++++++++++++++
test/fakeOrigin/httpService/httpService.go | 277 ++++++++++++++++++++++
test/fakeOrigin/httpService/rangerequest.go | 130 +++++++++++
test/fakeOrigin/httpService/ssl.keygen.go | 42 ++++
test/fakeOrigin/m3u8/m3u8.go | 328 +++++++++++++++++++++++++++
test/fakeOrigin/shard.sh | 86 +++++++
test/fakeOrigin/transcode/transcode.go | 222 ++++++++++++++++++
test/fakeOrigin/version/VERSION | 5 +
test/fakeOrigin/version/version.go | 41 ++++
33 files changed, 3090 insertions(+)
diff --git a/LICENSE b/LICENSE
index f0485b8..83a7762 100644
--- a/LICENSE
+++ b/LICENSE
@@ -426,6 +426,10 @@ The ldap.v2 component is used under the MIT license:
@traffic_ops/vendor/gopkg.in/ldap.v2/*
./traffic_ops/vendor/gopkg.in/ldap.v2/LICENSE
+The fakeOrigin example video is used the Public Domain license:
+@test/fakeOrigin/example/video/*
+./test/fakeOrigin/example/video/LICENSE
+
The json-iterator/go component is used under the MIT license:
@vendor/github.com/json-iterator/go/*
./vendor/github.com/json-iterator/go/LICENSE
diff --git a/test/fakeOrigin/.dockerignore b/test/fakeOrigin/.dockerignore
new file mode 100644
index 0000000..cdeb839
--- /dev/null
+++ b/test/fakeOrigin/.dockerignore
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+out/*
+out
+*.avi
+*.crt
+*.key
+*.json
+dist/*
+dist
+docker_host/*
+docker_host
diff --git a/test/fakeOrigin/.gitignore b/test/fakeOrigin/.gitignore
new file mode 100644
index 0000000..26d9b47
--- /dev/null
+++ b/test/fakeOrigin/.gitignore
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+*.avi
+debug
+fakeOrigin
+config.json
+!build/config.json
+out/*
+origin/*
+*.crt
+*.cert
+*.key
+dist/*
+docker_host/*
diff --git a/test/fakeOrigin/Dockerfile b/test/fakeOrigin/Dockerfile
new file mode 100644
index 0000000..f79fd92
--- /dev/null
+++ b/test/fakeOrigin/Dockerfile
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM golang:1.11
+MAINTAINER dev@trafficcontrol.apache.org
+
+RUN apt-get update && \
+ apt-get install -y ffmpeg \
+ openssl \
+ gpac && \
+ apt-get autoremove -y && \
+ apt-get autoclean -y && \
+ rm -rf /var/lib/apt/lists/*
+
+COPY . /go/src/github.com/apache/trafficcontrol
+WORKDIR /go/src/github.com/apache/trafficcontrol/test/fakeOrigin
+
+RUN go get -d -v ./... && \
+ go install -v ./...
+
+RUN groupadd -g 999 fakeOrigin && \
+ useradd -r -u 999 -g fakeOrigin fakeOrigin
+RUN chown -R fakeOrigin:fakeOrigin .
+USER fakeOrigin
+
+VOLUME ["/host"]
+
+CMD ["fakeOrigin", "-cfg", "/host/config.json"]
diff --git a/test/fakeOrigin/Dockerfile.build_rpm b/test/fakeOrigin/Dockerfile.build_rpm
new file mode 100644
index 0000000..4a23560
--- /dev/null
+++ b/test/fakeOrigin/Dockerfile.build_rpm
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM golang:1.11
+MAINTAINER dev@trafficcontrol.apache.org
+
+RUN apt-get update && \
+ apt-get install -y \
+ cross-gcc-dev \
+ gccgo-aarch64-linux-gnu \
+ gccgo-arm-linux-gnueabi \
+ gccgo-arm-linux-gnueabihf \
+ gccgo-go \
+ zip \
+ rpm && \
+ apt-get autoremove -y && \
+ apt-get autoclean -y && \
+ rm -rf /var/lib/apt/lists/*
+
+COPY . /go/src/github.com/apache/trafficcontrol
+WORKDIR /go/src/github.com/apache/trafficcontrol/test/fakeOrigin
+
+VOLUME ["/go/src/github.com/apache/trafficcontrol/test/fakeOrigin/dist"]
+
+CMD ["/bin/bash","build/build_rpm.sh"]
diff --git a/test/fakeOrigin/README.md b/test/fakeOrigin/README.md
new file mode 100644
index 0000000..b6f07a1
--- /dev/null
+++ b/test/fakeOrigin/README.md
@@ -0,0 +1,74 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# fakeOrigin
+
+fakeOrigin is a simple HLS video server, capable of simulating live video traffic. It can:
+
+* Serve HLS Live video, by transcoding a static video file as VOD, then manipulating the manifest on the fly to serve an infinitely looping live manifest
+* Serve HLS VOD video, from a static video file
+* Serve static video and other files
+
+# How to install
+Local build pre-requesites:
+* Go 1.9+
+ * OSX: ```brew install go```
+ * CentOS: [Instructions]( https://www.itzgeek.com/how-tos/linux/centos-how-tos/install-go-1-7-ubuntu-16-04-14-04-centos-7-fedora-24.html)
+* FFMPEG 3.4+ (Optional)
+ * OSX: ```brew install ffmpeg --with-rtmp-dump```
+ * CentOS: [Instructions](https://linuxadmin.io/install-ffmpeg-on-centos-7/)
+
+and/or just a modern version of Docker & docker-compose
+
+If you're building locally, just run ```go get github.com/apache/trafficcontrol/test/fakeOrigin```
+
+If you're just using docker, clone this repository.
+
+# How to use
+Running locally:
+```
+Usage:
+fakeOrigin (generates a minimal config.json next to binary)
+fakeOrigin -cfg config.json (same as above, but specify the location)
+```
+Running in docker:
+```
+docker-compose build --no-cache
+docker-compose up --force-recreate
+... customize the config.json created in ./docker_host (maps to /host inside the container, it's really important to customize this appropriately)
+docker-compose up --force-recreate
+```
+
+On startup it will print any routes that are available after transcoding. You should just be able to plug those m3u8 url into VLC to start streaming.
+
+I'd *highly* recommend going and reading about the [Configuration](./docs/Configuration.md) to learn about what fakeOrigin can do.
+
+There is also another [set of instructions](build/README.md) if you're interested in building your own RPMs and binaries.
+
+# Features
+* Transcoding on startup only if the source file or transcoder configuration changes
+* Single & Multiple Static file serving support
+* HTTP & HTTPS support
+* RFC7232 support for CDN caching
+* RFC7233 support for Range requests
+ * Both single and multi-part ranges are supported
+* Arbitrary header response controls
+ * Controlled via config file or by client request headers
+* Optional in-memory caching
+* Support for arbitrary external commands to perform transcoding
+ * Supports vod, live, and event m3u8 HLS manifest types
diff --git a/test/fakeOrigin/build/README.md b/test/fakeOrigin/build/README.md
new file mode 100644
index 0000000..83f0b12
--- /dev/null
+++ b/test/fakeOrigin/build/README.md
@@ -0,0 +1,35 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# Building RPMs & Binaries
+To build the RPMs and binaries you will need docker.
+## Usage
+```bash
+docker-compose -f docker-compose.build_rpm.yml build --no-cache builder
+docker-compose -f docker-compose.build_rpm.yml up --force-recreate --exit-code-from builder
+```
+This should dump all artifacts in the %repository root%/dist directory when it's done (like all other ATC build jobs).
+
+## Version
+If you need to manipulate the version information, you can export a few environment variables to supply your own overrides for the [VERSION](../version/VERSION) file.
+* VER_MAJOR (integer)
+* VER_MINOR (integer)
+* VER_PATCH (integer)
+* VER_DESC (short string)
+* VER_COMMIT (git short hash string)
+* BUILD_NUMBER (integer)
diff --git a/test/fakeOrigin/build/build_rpm.sh b/test/fakeOrigin/build/build_rpm.sh
new file mode 100755
index 0000000..f6a9a23
--- /dev/null
+++ b/test/fakeOrigin/build/build_rpm.sh
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -ex
+env
+
+BUILDDIR="$HOME/rpmbuild"
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+pwd
+cd $DIR/..
+pwd
+
+if [ -z "${VER_MAJOR+set}" ]; then
+ VER_MAJOR=$(sed '1q;d' $DIR/../version/VERSION)
+fi
+if [ -z "${VER_MINOR+set}" ]; then
+ VER_MINOR=$(sed '2q;d' $DIR/../version/VERSION)
+fi
+if [ -z "${VER_PATCH+set}" ]; then
+ VER_PATCH=$(sed '3q;d' $DIR/../version/VERSION)
+fi
+if [ -z "${VER_DESC+set}" ]; then
+ VER_DESC=$(sed '4q;d' $DIR/../version/VERSION)
+fi
+if [ -z "${VER_COMMIT+set}" ]; then
+# VER_COMMIT=$(sed '5q;d' $DIR/../version/VERSION)
+ VER_COMMIT=$(git -C ${DIR}/../../.. rev-list --all --count)
+fi
+if [ -z "${BUILD_NUMBER+set}" ]; then
+ BUILD_NUMBER=1
+fi
+
+
+VERSION="${VER_MAJOR}.${VER_MINOR}.${VER_PATCH}_${VER_DESC}_${VER_COMMIT}"
+
+# prep build environment
+mkdir -p $DIR/../dist
+rm -rf $BUILDDIR
+mkdir -p $BUILDDIR/{BUILD,RPMS,SOURCES}
+echo "$BUILDDIR" > ~/.rpmmacros
+
+# build
+go build -v -ldflags "-X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerFull=${VERSION} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMajor=${VER_MAJOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMinor=${VER_MINOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerPatch=${VER_PATCH} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerDesc=${VER_DESC} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerCo [...]
+
+# tar
+tar -cvzf $BUILDDIR/SOURCES/fakeOrigin-${VERSION}-${BUILD_NUMBER}.tgz fakeOrigin build/config.json build/fakeOrigin.init build/fakeOrigin.logrotate example
+
+# build RPM
+rpmbuild --define "_topdir ${BUILDDIR}" --define "_version ${VERSION}" --define "_release ${BUILD_NUMBER}" -ba build/fakeOrigin.spec
+
+# copy build RPM to ../dist
+cp $BUILDDIR/RPMS/x86_64/*.rpm ./dist/
+
+# Cross compile because we can
+GOBINEXT=""
+for GOOS in darwin linux windows; do
+ for GOARCH in 386 amd64; do
+ if [[ "$GOOS" == "windows" ]]
+ then
+ GOBINEXT=".exe"
+ else
+ GOBINEXT=""
+ fi
+ GOOS=$GOOS GOARCH=$GOARCH go build -v -ldflags "-X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerFull=${VERSION} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMajor=${VER_MAJOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMinor=${VER_MINOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerPatch=${VER_PATCH} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerDesc=${VER_DESC} -X github.com/apache/trafficcontrol [...]
+ zip -r $DIR/../dist/fakeOrigin-$VERSION-$GOOS-$GOARCH.zip fakeOrigin$GOBINEXT example
+ done
+done
+
+# ARM Cross compile because we can
+GOOS=linux
+GOARCH=arm
+for GOARM in 5 6 7; do
+ GOOS=$GOOS GOARCH=$GOARCH GOARM=$GOARM go build -v -ldflags "-X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerFull=${VERSION} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMajor=${VER_MAJOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMinor=${VER_MINOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerPatch=${VER_PATCH} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerDesc=${VER_DESC} -X github.com/apache/tra [...]
+ zip -r $DIR/../dist/fakeOrigin-$VERSION-$GOOS-$GOARCH-ARM$GOARM.zip fakeOrigin example
+done
+GOARCH=arm64
+GOOS=$GOOS GOARCH=$GOARCH go build -v -ldflags "-X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerFull=${VERSION} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMajor=${VER_MAJOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerMinor=${VER_MINOR} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerPatch=${VER_PATCH} -X github.com/apache/trafficcontrol/test/fakeOrigin/version.VerDesc=${VER_DESC} -X github.com/apache/trafficcontrol/tes [...]
+zip -r $DIR/../dist/fakeOrigin-$VERSION-$GOOS-$GOARCH-ARM8.zip fakeOrigin example
diff --git a/test/fakeOrigin/build/config.json b/test/fakeOrigin/build/config.json
new file mode 100644
index 0000000..28ea049
--- /dev/null
+++ b/test/fakeOrigin/build/config.json
@@ -0,0 +1,24 @@
+{
+ "server": {
+ "http_port": 8080,
+ "https_port": 8443,
+ "ssl_cert": "/etc/fakeOrigin/server.cert",
+ "ssl_key": "/etc/fakeOrigin/server.key",
+ "binding_address": "127.0.0.1",
+ "crossdomain_xml_file": "/opt/fakeOrigin/example/crossdomain.xml"
+ },
+ "endpoints": [
+ {
+ "id": "SampleFile",
+ "source": "/opt/fakeOrigin/example/video/kelloggs.mp4",
+ "outputdir": "/opt/fakeOrigin/out",
+ "type": "static"
+ },
+ {
+ "id": "SampleDirectory",
+ "source": "/opt/fakeOrigin/example/video/",
+ "outputdir": "/opt/fakeOrigin/out",
+ "type": "dir"
+ }
+ ]
+}
diff --git a/test/fakeOrigin/build/fakeOrigin.init b/test/fakeOrigin/build/fakeOrigin.init
new file mode 100755
index 0000000..2fc5927
--- /dev/null
+++ b/test/fakeOrigin/build/fakeOrigin.init
@@ -0,0 +1,121 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Startup script for fakeOrigin
+#
+#
+# chkconfig: 345 99 10
+# description: fakeOrigin control script
+# processname: fakeOrigin
+
+### BEGIN INIT INFO
+# Provides: fakeOrigin
+# Required-Start: $network $local_fs $syslog
+# Required-Stop: $network $local_fs $syslog
+# Default-Start: 3 4 5
+# Default-Stop: 0 1 2 6
+# Short-Description: start and stop fakeOrigin
+# Description: Controls all fakeOrigin processes at once.
+### END INIT INFO
+
+# Source function library.
+. /etc/init.d/functions
+
+# Source networking configuration.
+. /etc/sysconfig/network
+
+name=fakeOrigin
+basepath=/opt/$name
+runpath=/var/run
+prog=$basepath/$name
+lockfile=$runpath/$name
+
+#options="-cfg /etc/${name}/config.json"
+options="-cfg /etc/fakeOrigin/config.json"
+
+start() {
+ [ "$NETWORKING" = "no" ] && exit 1
+ [ -x $prog ] || exit 5
+
+ #Set file limits
+ # Max open files
+ OPEN_FILE_LIMIT=65536
+ ulimit -n $OPEN_FILE_LIMIT
+ if [ $? -ne 0 ]; then
+ echo "Failed to set open file limit to $OPEN_FILE_LIMIT"
+ exit 1
+ fi
+
+ # Start daemons.
+ echo -n "Starting $name: "
+ daemon nohup $prog $options < /dev/null > /var/log/${name}/${name}.log 2>&1 &
+ RETVAL=$?
+ echo
+ [ $RETVAL -eq 0 ] && touch $lockfile
+ return $RETVAL
+}
+
+stop() {
+ echo -n "Shutting down $name: "
+ killproc $prog
+ RETVAL=$?
+ echo
+ [ $RETVAL -eq 0 ] && rm -f $lockfile
+ return $RETVAL
+}
+
+reload() {
+ echo -n "Reloading $name: "
+ if [ -n "`pidofproc $prog`" ]; then
+ killproc $prog -HUP
+ else
+ failure "Reloading $name"
+ fi
+ RETVAL=$?
+ echo
+ return $RETVAL
+}
+
+case "$1" in
+ start)
+ start
+ ;;
+ stop)
+ stop
+ ;;
+ status)
+ status $prog
+ ;;
+ restart|force-reload)
+ stop
+ start
+ ;;
+ try-restart|condrestart)
+ if status $prog > /dev/null; then
+ stop
+ start
+ fi
+ ;;
+ reload)
+ reload
+ ;;
+ *)
+ echo "Usage: $0 {start|stop|status|restart|try-restart|reload|force-reload}"
+ exit 2
+esac
diff --git a/test/fakeOrigin/build/fakeOrigin.logrotate b/test/fakeOrigin/build/fakeOrigin.logrotate
new file mode 100644
index 0000000..0ae33d3
--- /dev/null
+++ b/test/fakeOrigin/build/fakeOrigin.logrotate
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+/var/log/fakeOrigin/fakeOrigin.log {
+ compress
+ maxage 30
+ missingok
+ nomail
+ size 10M
+ rotate 5
+ copytruncate
+}
diff --git a/test/fakeOrigin/build/fakeOrigin.spec b/test/fakeOrigin/build/fakeOrigin.spec
new file mode 100644
index 0000000..11cfc58
--- /dev/null
+++ b/test/fakeOrigin/build/fakeOrigin.spec
@@ -0,0 +1,70 @@
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# RPM spec file for the test origin fakeOrigin
+#
+
+Summary: fakeOrigin CDN Origin
+Name: fakeOrigin
+Version: %{_version}
+Release: %{_release}
+Prefix: /usr/sbin/%{name}
+Source: %{_sourcedir}/%{name}-%{_version}-%{_release}.tgz
+Distribution: CentOS Linux
+BuildRoot: %{buildroot}
+License: Apache License, Version 2.0
+URL: https://github.com/apache/trafficcontrol
+Vendor: Apache Software Foundation
+Requires: initscripts
+
+%description
+A fake HTTP CDN Origin for testing
+
+%prep
+
+%build
+tar -xvzf %{_sourcedir}/%{name}-%{_version}-%{_release}.tgz --directory %{_builddir}
+
+%install
+rm -rf %{buildroot}/opt/%{name}
+mkdir -p %{buildroot}/opt/%{name}/example
+cp -p %{name} %{buildroot}/opt/%{name}
+cp -rp example/* %{buildroot}/opt/%{name}/example/
+
+rm -rf %{buildroot}/etc/%{name}
+mkdir -p -m 777 %{buildroot}/etc/%{name}
+cp -p build/config.json %{buildroot}/etc/%{name}
+
+rm -rf %{buildroot}/etc/logrotate.d/%{name}
+mkdir -p -m 777 %{buildroot}/etc/logrotate.d/%{name}
+cp -p build/%{name}.logrotate %{buildroot}/etc/logrotate.d/%{name}
+
+rm -rf %{buildroot}/var/log/%{name}
+mkdir -p -m 777 %{buildroot}/var/log/%{name}
+
+mkdir -p -m 777 %{buildroot}/etc/init.d/
+cp -p build/%{name}.init %{buildroot}/etc/init.d/%{name}
+
+%clean
+echo "cleaning"
+rm -r -f %{buildroot}
+
+%files
+/opt/%{name}/%{name}
+/opt/%{name}/example
+/var/log/%{name}
+%config(noreplace) /etc/%{name}
+%config(noreplace) /etc/logrotate.d/%{name}
+/etc/init.d/%{name}
diff --git a/test/fakeOrigin/docker-compose.build_rpm.yml b/test/fakeOrigin/docker-compose.build_rpm.yml
new file mode 100644
index 0000000..89c1d07
--- /dev/null
+++ b/test/fakeOrigin/docker-compose.build_rpm.yml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+version: '2'
+
+services:
+ builder:
+ build:
+ context: ../..
+ dockerfile: test/fakeOrigin/Dockerfile.build_rpm
+ volumes:
+ - ../../dist:/go/src/github.com/apache/trafficcontrol/test/fakeOrigin/dist
+ environment:
+ - VER_MAJOR
+ - VER_MINOR
+ - VER_PATCH
+ - VER_DESC
+ - VER_COMMIT
+ - BUILD_NUMBER
diff --git a/test/fakeOrigin/docker-compose.yml b/test/fakeOrigin/docker-compose.yml
new file mode 100644
index 0000000..58ba86b
--- /dev/null
+++ b/test/fakeOrigin/docker-compose.yml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+version: '2'
+
+services:
+ fake_origin:
+ build:
+ context: ../..
+ dockerfile: test/fakeOrigin/Dockerfile
+ volumes:
+ - ../../test/fakeOrigin/docker_host:/host
+ environment:
+ - VER_MAJOR
+ - VER_MINOR
+ - VER_PATCH
+ - VER_DESC
+ - VER_COMMIT
+ - BUILD_NUMBER
+ ports:
+ - "8080:8080"
+ - "8443:8443"
diff --git a/test/fakeOrigin/docs/Configuration.md b/test/fakeOrigin/docs/Configuration.md
new file mode 100644
index 0000000..9634f8e
--- /dev/null
+++ b/test/fakeOrigin/docs/Configuration.md
@@ -0,0 +1,140 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# Configuration
+The `config.json` file has basically 2 sections
+1. Server Info
+2. Endpoints
+
+**For Developers**
+
+The underlying data structures for these can be found in [endpoint.go](../endpoint/endpoint.go) and enums can be found in [endpoint_enums.go](../endpoint/endpoint_enums.go). We may periodically inject new configs into your config.json, but that should be only if they are required with a sane default or inert.
+
+## Server Info
+```json
+"server": {
+ "http_port": 8080,
+ "https_port": 8443,
+ "ssl_cert": "server.crt",
+ "ssl_key": "server.key",
+ "binding_address": "127.0.0.1",
+ "crossdomain_xml_file": "./example/crossdomain.xml"
+}
+```
+This allows you to set what ports and IP addresses you'd like to listen on. You can also supply where your SSL certificates and key should be located or created. They should be in PEM format.
+
+## Endpoints
+This is where the meat of your config will be.
+```json
+"endpoints": [
+ {
+ "id": "EXAMPLE_ENDPOINT",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out/EXAMPLE_ENDPOINT",
+ "type": "live",
+ "default_headers": {
+ "another-custom-header": [
+ "\"foo\", \"bar\""
+ ],
+ "my-custom-header": [
+ "foo",
+ "bar"
+ ]
+ },
+ "manual_command": [
+ "transcode.cli.executable",
+ "-arguments",
+ "-anotherArg",
+ "%SOURCE%",
+ "%OUTPUTDIRECTORY%/%DISKID%.out"
+ ]
+ }
+]
+```
+A few things you should know:
+1. `id` must be unique and will be the first segment of your endpoint URL path
+2. `override_disk_id` will default to be id and can usually be ommitted. This is only for when you need different endpoints that share the same transcoder output
+3. Be sure `outputdir` has enough disk space to handle whatever you're asking fakeOrigin to generate. This should also be unique to each `override_disk_id`.
+4. `manual_command` includes some string required token replacements to enable fakeOrigin to parse the output
+
+To help make things a bit easier, you can take a look at [some samples](./Endpoint.Examples.md) on how to build endpoints.
+The transcoder phase will be skipped for static and dir `type` as well as other `type` where the a hash of the input file and the `manual_command` are unchanged.
+
+## Manual Command Tokens
+You cannot use unescaped spaces in commands. Each segment of the command is a separate entry in the array.
+
+Since fakeOrigin still needs to know about where certain things are, such as the metadata generation, file organization, and the live manifest interceptor, a set of tokens are defined for use in a manual command. These are mostly taken from the endpoint properties themselves.
+```
+%DISKID% : Identifier for use in file paths, defaults to same as ID
+%ENDPOINTTYPE% : Endpoint Type [live, vod, event, static, dir]
+%ID% : Identifier for use in url paths
+%MASTERMANIFEST% : A primary video manifest path on disk [OutputDirectory + "/" + DiskID + ".m3u8"]
+%OUTPUTDIRECTORY% : The directory on disk for output files
+%SOURCE% : The original source file used in transcoding
+```
+When writing manual commands, you really want to aim for your transcoder to output all files into a single directory (%OUTPUTDIRECTORY%) after consuming your single source asset (%SOURCE%) with a primary manifest (%MASTERMANIFEST%). In fact, these 3 parameters are required.
+
+## Adaptive Bit Rate
+When using HLS with adaptive bitrate layers, the master manifest generator within fakeOrigin expects all layer manifests in the `%OUTPUTDIRECTORY%` with the format `%DISKID%.*(?<width>\d+)x(?<height>\d+)-(?<bandwidth>\d+).m3u8`. For example, `%DISKID%_1920x1080-40000.m3u8`.
+
+# Client-side Header Overrides
+Setting `default_headers` in the config file makes sense for cases like video players who can't supply custom HTTP headers. However for testing specific use cases, you can also supply custom headers as a client to override what gets returned in the fakeOrigin response. If you give a header with the Prefix `Fakeorigin-`, that prefix will be stripped and the rest of the client header string will be returned.
+
+Example (Normal):
+```bash session
+curl -vs4 -o /dev/null http://localhost:8080/SampleVideo/kelloggs.mp4
+* Trying 127.0.0.1...
+* TCP_NODELAY set
+* Connected to localhost (127.0.0.1) port 8080 (#0)
+> GET /SampleVideo/kelloggs.mp4 HTTP/1.1
+> Host: localhost:8080
+> User-Agent: curl/7.59.0
+> Accept: */*
+>
+< HTTP/1.1 200 OK
+< Another-Custom-Header: foo, bar
+< Content-Type: video/mp4
+< My-Custom-Header: foo
+< My-Custom-Header: bar
+< Date: Thu, 19 Jul 2018 16:08:35 GMT
+< Transfer-Encoding: chunked
+<
+```
+Example (With Client Header Override):
+```bash session
+curl -vs4 -o /dev/null http://localhost:8080/SampleVideo/kelloggs.mp4 -H 'Fakeorigin-A-Custom_header: foo' -H 'fakeOrigin-Another-Custom-Header: "baz"'
+* Trying 127.0.0.1...
+* TCP_NODELAY set
+* Connected to localhost (127.0.0.1) port 8080 (#0)
+> GET /SampleVideo/kelloggs.mp4 HTTP/1.1
+> Host: localhost:8080
+> User-Agent: curl/7.59.0
+> Accept: */*
+> Fakeorigin-A-Custom_header: foo
+> Fakeorigin-Another-Custom-Header: "baz"
+>
+< HTTP/1.1 200 OK
+< A-Custom_header: foo
+< Another-Custom-Header: "baz"
+< Content-Type: video/mp4
+< My-Custom-Header: foo
+< My-Custom-Header: bar
+< Date: Thu, 19 Jul 2018 16:10:43 GMT
+< Transfer-Encoding: chunked
+<
+```
diff --git a/test/fakeOrigin/docs/Endpoint.Examples.md b/test/fakeOrigin/docs/Endpoint.Examples.md
new file mode 100644
index 0000000..b610880
--- /dev/null
+++ b/test/fakeOrigin/docs/Endpoint.Examples.md
@@ -0,0 +1,223 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# Endpoint Examples
+Some of these examples make use of [ffmpeg](https://www.ffmpeg.org/), however any script or application that can satisfy the required tokens is sufficient if transcoding is needed. The specific commands used in these examples may vary based on your individual requirements. Refer to the documentation of your transcode command for more information.
+
+## HLS VOD with Adaptive Bit Rate (ABR)
+```json
+{
+ "id": "HLS_ABR_VOD",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out/HLS_ABR_VOD",
+ "type": "vod",
+ "manual_command": [
+ "ffmpeg",
+ "-y",
+ "-i", "%SOURCE%",
+ "-vf", "scale=w=1280:h=720:force_original_aspect_ratio=decrease",
+ "-c:a", "aac",
+ "-ar", "48000",
+ "-c:v", "h264",
+ "-profile:v", "main",
+ "-crf", "20",
+ "-sc_threshold", "0",
+ "-g", "48",
+ "-keyint_min", "48",
+ "-hls_time", "4",
+ "-hls_playlist_type", "vod",
+ "-b:v", "2800k",
+ "-b:a", "128k",
+ "-hls_list_size", "0",
+ "-hls_segment_filename", "%OUTPUTDIRECTORY%/%DISKID%_seq_%03d_1280x720.ts",
+ "%OUTPUTDIRECTORY%/%DISKID%_1280x720-20000.m3u8",
+ "-vf", "scale=w=1920:h=1080:force_original_aspect_ratio=decrease",
+ "-c:a", "aac",
+ "-ar", "48000",
+ "-c:v", "h264",
+ "-profile:v", "main",
+ "-crf", "20",
+ "-sc_threshold", "0",
+ "-g", "48",
+ "-keyint_min", "48",
+ "-hls_time", "4",
+ "-hls_playlist_type", "vod",
+ "-b:v", "5000k",
+ "-b:a", "192k",
+ "-hls_list_size", "0",
+ "-hls_segment_filename", "%OUTPUTDIRECTORY%/%DISKID%_seq_%03d_1920x1080.ts",
+ "%OUTPUTDIRECTORY%/%DISKID%_1920x1080-40000.m3u8"
+ ]
+}
+
+```
+## Duplicate Endpoint
+```json
+{
+ "id": "DUPLICATE_HLS_ABR_VOD",
+ "override_disk_id": "HLS_ABR_VOD",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out/HLS_ABR_VOD",
+ "type": "vod"
+}
+```
+The important bits to notice here is that the `override_disk_id`, `outputdir`, & `type` match our first example. Only the `id` and `default_headers` are different. This will cause the transcoder metadata to match and therefore inform fakeOrigin that it shouldn't bother trying to transcode it again.
+
+## HLS with ABR live manifests
+```json
+{
+ "id": "HLS_ABR_LIVE",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out/HLS_ABR_LIVE",
+ "type": "live",
+ "manual_command": [
+ "ffmpeg",
+ "-y",
+ "-i", "%SOURCE%",
+ "-vf", "scale=w=1280:h=720:force_original_aspect_ratio=decrease",
+ "-c:a", "aac",
+ "-ar", "48000",
+ "-c:v", "h264",
+ "-profile:v", "main",
+ "-crf", "20",
+ "-sc_threshold", "0",
+ "-g", "48",
+ "-keyint_min", "48",
+ "-hls_time", "4",
+ "-hls_playlist_type", "vod",
+ "-b:v", "2800k",
+ "-b:a", "128k",
+ "-hls_list_size", "0",
+ "-hls_segment_filename", "%OUTPUTDIRECTORY%/%DISKID%_seq_%03d_1280x720.ts",
+ "%OUTPUTDIRECTORY%/%DISKID%_1280x720-20000.m3u8",
+ "-vf", "scale=w=1920:h=1080:force_original_aspect_ratio=decrease",
+ "-c:a", "aac",
+ "-ar", "48000",
+ "-c:v", "h264",
+ "-profile:v", "main",
+ "-crf", "20",
+ "-sc_threshold", "0",
+ "-g", "48",
+ "-keyint_min", "48",
+ "-hls_time", "4",
+ "-hls_playlist_type", "vod",
+ "-b:v", "5000k",
+ "-b:a", "192k",
+ "-hls_list_size", "0",
+ "-hls_segment_filename", "%OUTPUTDIRECTORY%/%DISKID%_seq_%03d_1920x1080.ts",
+ "%OUTPUTDIRECTORY%/%DISKID%_1920x1080-40000.m3u8"
+ ]
+}
+```
+On live endpoint types, we intercept the m3u8 requests and rewrite them so that they loop.
+## HLS without ABR VOD
+```json
+{
+ "id": "HLS_VOD",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out/HLS_VOD",
+ "type": "vod",
+ "manual_command": [
+ "ffmpeg",
+ "-y",
+ "-i", "%SOURCE%",
+ "-vf", "scale=w=1920:h=1080:force_original_aspect_ratio=decrease",
+ "-c:a", "aac",
+ "-ar", "48000",
+ "-c:v", "h264",
+ "-profile:v", "main",
+ "-crf", "20",
+ "-sc_threshold", "0",
+ "-g", "48",
+ "-keyint_min", "48",
+ "-hls_time", "4",
+ "-hls_playlist_type", "vod",
+ "-b:v", "5000k",
+ "-b:a", "192k",
+ "-hls_list_size", "0",
+ "-hls_segment_filename", "%OUTPUTDIRECTORY%/%DISKID%_seq_%03d.ts",
+ "%OUTPUTDIRECTORY%/%DISKID%.m3u8"
+ ]
+}
+```
+
+## HLS Event
+```json
+{
+ "id": "HLS_EVENT",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out/HLS_EVENT",
+ "type": "event",
+ "manual_command": [
+ "ffmpeg",
+ "-y",
+ "-i", "%SOURCE%",
+ "-vf", "scale=w=1920:h=1080:force_original_aspect_ratio=decrease",
+ "-c:a", "aac",
+ "-ar", "48000",
+ "-c:v", "h264",
+ "-profile:v", "main",
+ "-crf", "20",
+ "-sc_threshold", "0",
+ "-g", "48",
+ "-keyint_min", "48",
+ "-hls_time", "4",
+ "-hls_playlist_type", "event",
+ "-b:v", "5000k",
+ "-b:a", "192k",
+ "-hls_list_size", "0",
+ "-hls_segment_filename", "%OUTPUTDIRECTORY%/%DISKID%_seq_%03d.ts",
+ "%OUTPUTDIRECTORY%/%DISKID%.m3u8"
+ ]
+}
+```
+## Static file
+```json
+{
+ "id": "SampleVideo",
+ "source": "./example/video/kelloggs.mp4",
+ "outputdir": "./out",
+ "type": "static"
+}
+```
+This shows how to serve single files with fakeOrigin. Also, `id` must still be unique even if they are different source files.
+
+## Directory
+```json
+{
+ "id": "SampleDir",
+ "source": "./example/video",
+ "outputdir": "./out",
+ "type": "dir"
+}
+```
+This shows how to serve all files in a given directory recursively. Also, `id` must still be unique even if they are different source files. This type serves each file it finds at startup as a static file.
+
+## Player Troubleshooting
+If you're running into issues with a Javascript-based test player, there is a good chance you may need to get add some default CORS headers to your endpoint config.
+```json
+"default_headers": {
+ "Access-Control-Allow-Headers": [
+ "*"
+ ],
+ "Access-Control-Allow-Origin": [
+ "*"
+ ]
+}
+```
+This is a wide open example and should be tailored to the domains of your test player appropriately.
diff --git a/test/fakeOrigin/endpoint/endpoint.go b/test/fakeOrigin/endpoint/endpoint.go
new file mode 100644
index 0000000..14b4341
--- /dev/null
+++ b/test/fakeOrigin/endpoint/endpoint.go
@@ -0,0 +1,216 @@
+package endpoint
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/ioutil"
+ "path/filepath"
+ "strings"
+)
+
+// DefaultConfigFile is the default configuration file path
+const DefaultConfigFile = "config.json"
+
+// DefaultOutputDirectory is the default output directory for generated content
+const DefaultOutputDirectory = "./out"
+
+// DefaultHTTPSKeyFile is the default path to the SSL key
+const DefaultHTTPSKeyFile = "server.key"
+
+// DefaultHTTPSCertFile is the default path to the SSL certificate
+const DefaultHTTPSCertFile = "server.cert"
+
+// ServerInfo contains relevant info for serving content
+type ServerInfo struct {
+ HTTPListeningPort int `json:"http_port"`
+ HTTPSListeningPort int `json:"https_port"`
+ SSLCert string `json:"ssl_cert"`
+ SSLKey string `json:"ssl_key"`
+ BindingAddress string `json:"binding_address"`
+ CrossdomainFile string `json:"crossdomain_xml_file"`
+}
+
+// Endpoint defines all kinds of endpoints to be served
+type Endpoint struct {
+ ID string `json:"id"`
+ DiskID string `json:"override_disk_id,omitempty"`
+ Source string `json:"source"`
+ OutputDirectory string `json:"outputdir,omitifempty"`
+ EndpointType Type `json:"type"`
+ ManualCommand []string `json:"manual_command,omitempty"`
+ DefaultHeaders map[string][]string `json:"default_headers,omitempty"`
+ NoCache bool `json:"no_cache,omitempty"`
+ ABRManifests []string `json:"abr_manifests,omitempty"`
+}
+
+// Config defines the application configuration
+type Config struct {
+ ServerConf ServerInfo `json:"server"`
+ Endpoints []Endpoint `json:"endpoints"`
+}
+
+func contains(args []string, param string) bool {
+ for _, str := range args {
+ if strings.Contains(str, param) {
+ return true
+ }
+ }
+ return false
+}
+
+func replace(tokenDict map[string]string, input string) string {
+ for k, v := range tokenDict {
+ input = strings.Replace(input, k, v, -1)
+ }
+ return input
+}
+
+// DefaultConfig generates a basic default configuration
+func DefaultConfig() Config {
+ return Config{
+ ServerConf: ServerInfo{
+ HTTPListeningPort: 8080,
+ HTTPSListeningPort: 8443,
+ SSLCert: DefaultHTTPSCertFile,
+ SSLKey: DefaultHTTPSKeyFile,
+ BindingAddress: "",
+ CrossdomainFile: "./example/crossdomain.xml",
+ },
+ Endpoints: []Endpoint{
+ {
+ ID: "SampleFile",
+ Source: "./example/video/kelloggs.mp4",
+ OutputDirectory: DefaultOutputDirectory,
+ EndpointType: Static,
+ },
+ {
+ ID: "SampleDirectory",
+ Source: "./example/video/",
+ OutputDirectory: DefaultOutputDirectory,
+ EndpointType: Dir,
+ },
+ },
+ }
+}
+
+// WriteConfig saves a fakeOrigin config as a pretty-printed json file
+func WriteConfig(cfg Config, path string) error {
+ bts, err := json.MarshalIndent(cfg, "", "\t")
+ if err != nil {
+ return errors.New("marshalling JSON: " + err.Error())
+ }
+ if err = ioutil.WriteFile(path, append(bts, '\n'), 0644); err != nil {
+ return errors.New("writing file '" + path + "': " + err.Error())
+ }
+ return nil
+}
+
+// ProcessConfig processes the config loaded from disk, or generated the first time. This must be called before the config can be used to transcode or serve.
+func ProcessConfig(out Config) (Config, error) {
+ for i := range out.Endpoints {
+ var err error
+ // Resolve relative paths to absolute paths
+ out.Endpoints[i].Source, err = filepath.Abs(out.Endpoints[i].Source)
+ if err != nil {
+ return Config{}, errors.New("resolving relative path: " + err.Error())
+ }
+ out.Endpoints[i].OutputDirectory, _ = filepath.Abs(out.Endpoints[i].OutputDirectory)
+
+ if out.Endpoints[i].OutputDirectory == "" {
+ out.Endpoints[i].OutputDirectory = DefaultOutputDirectory
+ }
+ if out.Endpoints[i].DiskID == "" {
+ out.Endpoints[i].DiskID = out.Endpoints[i].ID
+ }
+ }
+ return out, nil
+}
+
+// LoadAndGenerateDefaultConfig loads the config from a given json file and puts a default value in place if you havn't stored anything
+func LoadAndGenerateDefaultConfig(path string) (Config, error) {
+ out := DefaultConfig()
+ defaultEndpoints := make([]Endpoint, len(out.Endpoints))
+ copy(defaultEndpoints, out.Endpoints)
+ raw, err := ioutil.ReadFile(path)
+ if err != nil || len(raw) == 0 {
+ raw = []byte("{}")
+ }
+ err = json.Unmarshal(raw, &out)
+ if err != nil {
+ return out, err
+ }
+ if err := WriteConfig(out, path); err != nil {
+ return out, errors.New("writing config to file: " + err.Error())
+ }
+ if fmt.Sprintf("%v", out.Endpoints) == fmt.Sprintf("%v", defaultEndpoints) {
+ return out, errors.New("default endpoints generated, please provide real input")
+ }
+ for _, ep := range out.Endpoints {
+ if len(ep.ManualCommand) > 0 {
+ //if !contains(ep.ManualCommand, "%MASTERMANIFEST%") {
+ // return out, errors.New("Manual commands must include the %MASTERMANIFEST% token")
+ //}
+ if !contains(ep.ManualCommand, `%OUTPUTDIRECTORY%`) {
+ return out, errors.New(`Manual commands must include the %OUTPUTDIRECTORY% token`)
+ }
+ if !contains(ep.ManualCommand, `%SOURCE%`) {
+ return out, errors.New(`Manual commands must include the %SOURCE% token`)
+ }
+ }
+ if len(ep.ABRManifests) > 0 {
+ return out, errors.New("Paths of ABR Layer manifests must not be set via configuration")
+ }
+ }
+ out, err = ProcessConfig(out)
+ if err = WriteConfig(out, path); err != nil {
+ return out, errors.New("processing config file: " + err.Error())
+ }
+ return out, nil
+}
+
+// GetTranscoderCommand produces an instruction for the transcode phase to execute
+func GetTranscoderCommand(ep Endpoint) (string, []string, error) {
+ out := ""
+ args := []string{}
+ if ep.EndpointType.String() == Static.String() {
+ out = "static"
+ } else if ep.EndpointType.String() == Dir.String() {
+ out = "dir"
+ } else if len(ep.ManualCommand) > 0 {
+ tokenmap := map[string]string{
+ `%DISKID%`: ep.DiskID,
+ `%ENDPOINTTYPE%`: ep.EndpointType.String(),
+ `%ID%`: ep.ID,
+ `%MASTERMANIFEST%`: ep.OutputDirectory + "/" + ep.DiskID + ".m3u8",
+ `%OUTPUTDIRECTORY%`: ep.OutputDirectory,
+ `%SOURCE%`: ep.Source,
+ }
+ for _, cmdPart := range ep.ManualCommand {
+ args = append(args, replace(tokenmap, cmdPart))
+ }
+ out = args[0]
+ args = args[1:]
+ }
+
+ return out, args, nil
+}
diff --git a/test/fakeOrigin/endpoint/endpoint_enums.go b/test/fakeOrigin/endpoint/endpoint_enums.go
new file mode 100644
index 0000000..8da24c2
--- /dev/null
+++ b/test/fakeOrigin/endpoint/endpoint_enums.go
@@ -0,0 +1,54 @@
+package endpoint
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//go:generate jsonenums -type=Type
+
+// Type models the supported types of endpoints
+type Type int
+
+// Type models the supported types of endpoints
+const (
+ InvalidType Type = iota + 1
+ Vod
+ Live
+ Event
+ Static
+ Dir
+)
+
+func (e Type) String() string {
+ switch e {
+ case InvalidType:
+ return "invalid type"
+ case Vod:
+ return "vod"
+ case Live:
+ return "live"
+ case Event:
+ return "event"
+ case Static:
+ return "static"
+ case Dir:
+ return "dir"
+ default:
+ return "invalid type"
+ }
+}
diff --git a/test/fakeOrigin/endpoint/type_jsonenums.go b/test/fakeOrigin/endpoint/type_jsonenums.go
new file mode 100644
index 0000000..e251c8d
--- /dev/null
+++ b/test/fakeOrigin/endpoint/type_jsonenums.go
@@ -0,0 +1,87 @@
+// generated by jsonenums -type Type; DO NOT EDIT
+
+package endpoint
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "encoding/json"
+ "fmt"
+)
+
+var (
+ _TypeNameToValue = map[string]Type{
+ "InvalidType": InvalidType,
+ "Vod": Vod,
+ "Live": Live,
+ "Event": Event,
+ "Static": Static,
+ "Dir": Dir,
+ }
+
+ _TypeValueToName = map[Type]string{
+ InvalidType: "InvalidType",
+ Vod: "Vod",
+ Live: "Live",
+ Event: "Event",
+ Static: "Static",
+ Dir: "Dir",
+ }
+)
+
+func init() {
+ var v Type
+ if _, ok := interface{}(v).(fmt.Stringer); ok {
+ _TypeNameToValue = map[string]Type{
+ interface{}(InvalidType).(fmt.Stringer).String(): InvalidType,
+ interface{}(Vod).(fmt.Stringer).String(): Vod,
+ interface{}(Live).(fmt.Stringer).String(): Live,
+ interface{}(Event).(fmt.Stringer).String(): Event,
+ interface{}(Static).(fmt.Stringer).String(): Static,
+ interface{}(Dir).(fmt.Stringer).String(): Dir,
+ }
+ }
+}
+
+// MarshalJSON is generated so Type satisfies json.Marshaler.
+func (r Type) MarshalJSON() ([]byte, error) {
+ if s, ok := interface{}(r).(fmt.Stringer); ok {
+ return json.Marshal(s.String())
+ }
+ s, ok := _TypeValueToName[r]
+ if !ok {
+ return nil, fmt.Errorf("invalid Type: %d", r)
+ }
+ return json.Marshal(s)
+}
+
+// UnmarshalJSON is generated so Type satisfies json.Unmarshaler.
+func (r *Type) UnmarshalJSON(data []byte) error {
+ var s string
+ if err := json.Unmarshal(data, &s); err != nil {
+ return fmt.Errorf("Type should be a string, got %s", data)
+ }
+ v, ok := _TypeNameToValue[s]
+ if !ok {
+ return fmt.Errorf("invalid Type %q", s)
+ }
+ *r = v
+ return nil
+}
diff --git a/test/fakeOrigin/example/crossdomain.xml b/test/fakeOrigin/example/crossdomain.xml
new file mode 100644
index 0000000..ab0d20a
--- /dev/null
+++ b/test/fakeOrigin/example/crossdomain.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" ?>
+<!--
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+<cross-domain-policy>
+ <site-control permitted-cross-domain-policies="master-only"/>
+ <allow-access-from domain="*"/>
+ <allow-http-request-headers-from domain="*" headers="*"/>
+</cross-domain-policy>
diff --git a/test/fakeOrigin/example/video/LICENSE b/test/fakeOrigin/example/video/LICENSE
new file mode 100644
index 0000000..f232c6f
--- /dev/null
+++ b/test/fakeOrigin/example/video/LICENSE
@@ -0,0 +1,5 @@
+SPDX-License-Identifier: CC-PDM-1.0
+The kelloggs.mp4 file is in the Public Domain, from the Prelinger Library, provided by archive.org:
+https://creativecommons.org/licenses/publicdomain/
+https://archive.org/details/prelinger_commercials
+https://archive.org/details/kelloggs_variety_pak
diff --git a/test/fakeOrigin/example/video/kelloggs.mp4 b/test/fakeOrigin/example/video/kelloggs.mp4
new file mode 100644
index 0000000..95c385d
Binary files /dev/null and b/test/fakeOrigin/example/video/kelloggs.mp4 differ
diff --git a/test/fakeOrigin/fakeOrigin.go b/test/fakeOrigin/fakeOrigin.go
new file mode 100644
index 0000000..e873a69
--- /dev/null
+++ b/test/fakeOrigin/fakeOrigin.go
@@ -0,0 +1,95 @@
+package main
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "flag"
+ "fmt"
+ "os"
+
+ "github.com/apache/trafficcontrol/test/fakeOrigin/endpoint"
+ "github.com/apache/trafficcontrol/test/fakeOrigin/httpService"
+ "github.com/apache/trafficcontrol/test/fakeOrigin/transcode"
+ "github.com/apache/trafficcontrol/test/fakeOrigin/version"
+)
+
+func printUsage() {
+ fmt.Print(`Usage:
+ fakeOrigin -cfg config.json
+`)
+}
+
+func main() {
+ cfg := flag.String("cfg", endpoint.DefaultConfigFile, "config file location")
+ printVersion := flag.Bool("version", false, "print fakeOrigin version")
+ flag.Parse()
+
+ if *printVersion {
+ fmt.Println(version.VerFull)
+ os.Exit(0)
+ }
+
+ config := endpoint.Config{}
+ err := error(nil)
+
+ fmt.Println("using config: " + *cfg)
+ if config, err = endpoint.LoadAndGenerateDefaultConfig(*cfg); err != nil {
+ fmt.Printf("An error occurred while loading configuration '%v': %s\n", *cfg, err)
+ os.Exit(1)
+ }
+
+ for i, ep := range config.Endpoints {
+ if ep.EndpointType == endpoint.Static || ep.EndpointType == endpoint.Dir {
+ continue
+ }
+ var cmd string
+ var args []string
+ cmd, args, err = endpoint.GetTranscoderCommand(config.Endpoints[i])
+ if err != nil {
+ fmt.Printf("An error occurred while fetching transcoder commands: %s\n", err)
+ os.Exit(1)
+ }
+ if cmd == "" {
+ fmt.Println("Skipping Transcode for endpoint: " + config.Endpoints[i].ID)
+ } else if err = transcode.Do(&config.Endpoints[i], cmd, args); err != nil {
+ fmt.Printf("An error occurred while performing transcoder commands: %s\n", err)
+ os.Exit(1)
+ }
+ }
+
+ routes, err := httpService.GetRoutes(config)
+ if err != nil {
+ fmt.Println("Error getting routes: " + err.Error())
+ os.Exit(1)
+ }
+ httpService.PrintRoutes(os.Stdout, routes, "", "Serving ", false)
+
+ go func() {
+ if err := httpService.StartHTTPSListener(config, routes); err != nil {
+ fmt.Printf("Error serving HTTPS: %s\n", err)
+ os.Exit(1)
+ }
+ }()
+
+ if err := httpService.StartHTTPListener(config, routes); err != nil {
+ fmt.Printf("Error serving HTTP: %s\n", err)
+ os.Exit(1)
+ }
+}
diff --git a/test/fakeOrigin/httpService/filter.go b/test/fakeOrigin/httpService/filter.go
new file mode 100644
index 0000000..e2379a6
--- /dev/null
+++ b/test/fakeOrigin/httpService/filter.go
@@ -0,0 +1,287 @@
+package httpService
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "fmt"
+ "hash/fnv"
+ "io"
+ "net/http"
+ "os"
+ "path"
+ "strconv"
+ "strings"
+ "time"
+)
+
+// BodyInterceptor is a container for a http writer body so we can write headers after the real writes are issued
+type BodyInterceptor struct {
+ w http.ResponseWriter
+ body bytes.Buffer
+ responseCode int
+}
+
+// WriteHeader doesn't actually write a header, just a response code. Thanks go.
+func (i *BodyInterceptor) WriteHeader(rc int) {
+ i.responseCode = rc
+}
+
+// Write is used for interface compatability with http response writer
+func (i *BodyInterceptor) Write(b []byte) (int, error) {
+ i.body.Write(b)
+ return i.body.Len(), nil
+}
+
+// Header is used for interface compatability with http response writer
+func (i *BodyInterceptor) Header() http.Header {
+ return i.w.Header()
+}
+
+// RealWrite is called to perform the actual write operation already done further in the chain
+func (i *BodyInterceptor) RealWrite() (int, error) {
+ if i.responseCode != 0 {
+ i.w.WriteHeader(i.responseCode)
+ }
+ c := i.body.Len()
+ io.Copy(i.w, &i.body)
+ return c, nil
+}
+
+// Body is used for interface compatability with http response writer
+func (i *BodyInterceptor) Body() []byte {
+ return i.body.Bytes()
+}
+
+// ParseHTTPDate parses the given RFC7231ยง7.1.1 HTTP-date
+func ParseHTTPDate(d string) (time.Time, bool) {
+ if t, err := time.Parse(time.RFC1123, d); err == nil {
+ return t, true
+ }
+ if t, err := time.Parse(time.RFC850, d); err == nil {
+ return t, true
+ }
+ if t, err := time.Parse(time.ANSIC, d); err == nil {
+ return t, true
+ }
+ return time.Time{}, false
+}
+
+func AddFullDefaultHeader(w http.ResponseWriter, r *http.Request, newKey string, newVals []string) bool {
+ if w.Header().Get(newKey) != "" || len(newVals) == 0 {
+ return false
+ }
+ w.Header().Set(newKey, newVals[0])
+ for _, val := range newVals[1:] {
+ w.Header().Add(newKey, val)
+ }
+ return true
+}
+
+func GenerateETag(source string) string {
+ h := fnv.New32a()
+ h.Write([]byte(source))
+ return fmt.Sprintf("\"%d\"", h.Sum32())
+}
+
+func log(handler http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ startTime := time.Now()
+ iw := &BodyInterceptor{w: w}
+ handler.ServeHTTP(iw, r)
+ size := iw.body.Len()
+ rc := iw.responseCode
+ if rc == 0 {
+ rc = http.StatusOK
+ }
+ iw.RealWrite()
+ finishTime := time.Now()
+ remoteAddr := r.RemoteAddr
+ finishTS := finishTime.Format(time.RFC1123)
+ method := r.Method
+ rURI := r.URL.EscapedPath()
+ proto := r.Proto
+ dur := finishTime.Sub(startTime)
+ refer := strings.Replace(r.Referer(), `"`, `\"`, -1)
+ uas := strings.Replace(r.UserAgent(), `"`, `\"`, -1)
+ im := strings.Replace(r.Header.Get("If-Match"), `"`, `\"`, -1)
+ inm := strings.Replace(r.Header.Get("If-None-Match"), `"`, `\"`, -1)
+ ims := strings.Replace(r.Header.Get("If-Modified-Since"), `"`, `\"`, -1)
+ ius := strings.Replace(r.Header.Get("If-Unmodified-Since"), `"`, `\"`, -1)
+ ir := strings.Replace(r.Header.Get("If-Range"), `"`, `\"`, -1)
+
+ fmt.Printf("%s - [%s] \"%s %s %s\" %d %d %d \"%s\" \"%s\" \"%s\" \"%s\" \"%s\" \"%s\" \"%s\"\n", remoteAddr, finishTS, method, rURI, proto, rc, size, dur, refer, uas, im, inm, ims, ius, ir)
+ })
+}
+func strictTransportSecurity(handler http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ iw := &BodyInterceptor{w: w}
+ handler.ServeHTTP(iw, r)
+ w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
+ iw.RealWrite()
+ })
+}
+func originHeaderManipulation(handler http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ iw := &BodyInterceptor{w: w}
+ handler.ServeHTTP(iw, r)
+ for key, value := range r.Header {
+ if strings.HasPrefix(key, "Fakeorigin-") {
+ newKey := strings.TrimPrefix(key, "Fakeorigin-")
+ // Intentionally stomp on anything else you may otherwise have known
+ w.Header().Set(newKey, value[0])
+ for _, val := range value[1:len(value)] {
+ w.Header().Add(newKey, val)
+ }
+ }
+ }
+ iw.RealWrite()
+ })
+}
+
+func checkIfMatch(req, origineTag, diskpath string) bool {
+ if req == "*" {
+ if _, err := os.Stat(diskpath); err == nil {
+ return true
+ }
+ }
+ etags := strings.Split(req, ",")
+ for _, etag := range etags {
+ if strings.TrimSpace(etag) == origineTag {
+ return true
+ }
+ }
+ return false
+}
+
+func checkIfNoneMatch(req, origineTag, diskpath string) bool {
+ if req == "*" {
+ if _, err := os.Stat(diskpath); err == nil {
+ return false
+ }
+ }
+ etags := strings.Split(req, ",")
+ for _, etag := range etags {
+ if strings.TrimPrefix(strings.TrimSpace(etag), "W/") == origineTag {
+ return false
+ }
+ }
+ return true
+}
+
+func checkIfRange(req, origineTag string, lastUpdated time.Time) bool {
+ if req == "" {
+ return true
+ }
+ reqTime, timeOk := ParseHTTPDate(req)
+ if !timeOk && strings.TrimPrefix(strings.TrimSpace(req), "W/") == origineTag {
+ return true
+ }
+ if timeOk && reqTime == lastUpdated {
+ return true
+ }
+ return false
+}
+
+func checkIsFullRange(rr string, size int) bool {
+ if rr == "" {
+ return false
+ }
+ nospaces := strings.Replace(rr, " ", "", -1)
+ if strings.HasSuffix(nospaces, "0-") {
+ return true
+ }
+ i := strconv.Itoa(size - 1)
+ if strings.HasSuffix(nospaces, "0-"+i) {
+ return true
+ }
+
+ return false
+}
+
+// https://tools.ietf.org/html/rfc7232 - If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since, If-Range
+// https://tools.ietf.org/html/rfc7233 - Range Requests, If-Range
+func cacheOptimization(handler http.Handler, startTime time.Time, ep httpEndpoint) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ diskpath := ep.OutputDir + "/" + strings.TrimPrefix(path.Base(r.RequestURI), "/"+ep.ID)
+ eTag := GenerateETag(ep.OutputDir + ep.DiskID + ep.LastTranscodeTime.Format(time.RFC1123) + path.Base(r.RequestURI))
+ w.Header().Set("ETag", eTag)
+ w.Header().Set("Last-Modified", ep.LastTranscodeTime.Format(time.RFC1123))
+ if r.Header.Get("If-Match") != "" && !checkIfMatch(r.Header.Get("If-Match"), eTag, diskpath) {
+ w.WriteHeader(http.StatusPreconditionFailed)
+ w.Write([]byte(strconv.Itoa(http.StatusPreconditionFailed) + " " + http.StatusText(http.StatusPreconditionFailed)))
+ }
+ if r.Header.Get("If-None-Match") != "" && !checkIfNoneMatch(r.Header.Get("If-None-Match"), eTag, diskpath) {
+ if r.Method == http.MethodGet || r.Method == http.MethodHead {
+ w.WriteHeader(http.StatusNotModified)
+ w.Write([]byte(strconv.Itoa(http.StatusNotModified) + " " + http.StatusText(http.StatusNotModified)))
+ } else {
+ w.WriteHeader(http.StatusPreconditionFailed)
+ w.Write([]byte(strconv.Itoa(http.StatusPreconditionFailed) + " " + http.StatusText(http.StatusPreconditionFailed)))
+ }
+ }
+ imsTime, timeOk := ParseHTTPDate(r.Header.Get("If-Modified-Since"))
+ if r.Header.Get("If-None-Match") == "" && (r.Method == http.MethodGet || r.Method == http.MethodHead) && timeOk {
+ if imsTime.Before(ep.LastTranscodeTime) || imsTime == ep.LastTranscodeTime {
+ w.WriteHeader(http.StatusNotModified)
+ w.Write([]byte(strconv.Itoa(http.StatusNotModified) + " " + http.StatusText(http.StatusNotModified)))
+ }
+ }
+ iusTime, timeOk := ParseHTTPDate(r.Header.Get("If-Unmodified-Since"))
+ if r.Header.Get("If-Match") == "" && (r.Method == http.MethodGet || r.Method == http.MethodHead) && timeOk {
+ if ep.LastTranscodeTime.After(iusTime) {
+ w.WriteHeader(http.StatusPreconditionFailed)
+ w.Write([]byte(strconv.Itoa(http.StatusPreconditionFailed) + " " + http.StatusText(http.StatusPreconditionFailed)))
+ }
+ }
+ iw := &BodyInterceptor{w: w}
+ handler.ServeHTTP(iw, r)
+ w.Header().Set("Accept-Ranges", "bytes")
+ rrange := r.Header.Get("Range")
+ irrange := r.Header.Get("If-Range")
+ // TODO: ensure this doesn't trigger on anything but 200
+ if rrange != "" && checkIfRange(irrange, eTag, ep.LastTranscodeTime) && !checkIsFullRange(rrange, len(iw.Body())) {
+ // Generate a 206 Paritial Content Range Request
+ if ranges, err := parseRange(rrange, uint64(len(iw.Body()))); err != nil {
+ iw.body.Reset()
+ iw.responseCode = http.StatusRequestedRangeNotSatisfiable
+ } else {
+ b, headers, err := clipToRange(ranges, iw.body.Bytes(), w.Header().Get("Content-Type"))
+ if err != nil {
+ iw.body.Reset()
+ iw.responseCode = http.StatusRequestedRangeNotSatisfiable
+ } else {
+ AddFullDefaultHeader(w, r, "Cache-Control", []string{})
+ AddFullDefaultHeader(w, r, "Expires", []string{(time.Now().Add(time.Minute * time.Duration(10))).Format(time.RFC1123)})
+ AddFullDefaultHeader(w, r, "Content-Location", []string{})
+ AddFullDefaultHeader(w, r, "Vary", []string{})
+ iw.responseCode = http.StatusPartialContent
+ // Reset the body in the body interceptor chain since we're explicitly clipping it to the requested ranges, otherwise it's just appending
+ iw.body.Reset()
+ iw.body.Write(b)
+ for key, val := range headers {
+ w.Header().Set(key, val)
+ }
+ }
+ }
+ }
+ iw.RealWrite()
+ })
+}
diff --git a/test/fakeOrigin/httpService/handler.go b/test/fakeOrigin/httpService/handler.go
new file mode 100644
index 0000000..801753f
--- /dev/null
+++ b/test/fakeOrigin/httpService/handler.go
@@ -0,0 +1,178 @@
+package httpService
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "encoding/xml"
+ "errors"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "os"
+ "path"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/apache/trafficcontrol/test/fakeOrigin/endpoint"
+ "github.com/apache/trafficcontrol/test/fakeOrigin/m3u8"
+)
+
+type httpEndpoint struct {
+ ID string
+ OutputDir string
+ Type endpoint.Type
+ DiskID string
+ IsLive bool
+ DefaultHeaders http.Header
+ NoCache bool
+ HTTPPath string
+ ContentType string
+ LastTranscodeTime time.Time
+}
+
+type staticHTTPEndpoint struct {
+ FilePath string
+ NoCache bool
+ Headers http.Header
+}
+
+// LiveM3U8MinFiles is used to govern the minimum number of files to be present in a live m3u8
+const LiveM3U8MinFiles = 20 // TODO make configurable
+// LiveM3U8MinDuration is used to govern the minimum number of seconds to be present in a live m2u8
+const LiveM3U8MinDuration = time.Duration(40) * time.Second // TODO make configurable
+
+func httpEndpointHandler(hend httpEndpoint) (http.HandlerFunc, error) {
+ start := time.Now() // used to compute the live manifest offset
+ diskpath := filepath.Join(hend.OutputDir, path.Base(hend.HTTPPath))
+ data, err := ioutil.ReadFile(diskpath) // read even if we're not caching, so we can error out before serving
+ if err != nil {
+ return nil, errors.New("reading file '" + diskpath + "': " + err.Error())
+ }
+
+ return func(w http.ResponseWriter, r *http.Request) {
+ for key, values := range hend.DefaultHeaders {
+ w.Header().Set(key, values[0])
+ for _, val := range values[1:] {
+ w.Header().Add(key, val)
+ }
+ }
+
+ if hend.NoCache {
+ if data, err = ioutil.ReadFile(diskpath); err != nil {
+ w.WriteHeader(http.StatusNotFound)
+ // Remove the error message in production, this is just for debug purposes
+ w.Write([]byte(strconv.Itoa(http.StatusNotFound) + " " + http.StatusText(http.StatusNotFound) + " - " + err.Error()))
+ return
+ }
+ }
+
+ if hend.IsLive {
+ vodM3U8, err := m3u8.Parse(data)
+ if err != nil {
+ fmt.Println("Error reading file: " + err.Error())
+ w.WriteHeader(http.StatusInternalServerError)
+ return
+ }
+ w.Header().Set("Cache-Control", "no-cache")
+ if len(vodM3U8.VARs) > 0 {
+ w.Header().Set("Content-Type", "application/x-mpegURL")
+ w.Write(data)
+ return
+ }
+ handleLiveM3U8(w, r, vodM3U8, LiveM3U8MinFiles, LiveM3U8MinDuration, start)
+ return
+ }
+
+ w.Header().Set("Content-Type", hend.ContentType)
+ w.Write(data)
+ }, nil
+}
+
+func handleLiveM3U8(w http.ResponseWriter, r *http.Request, vod m3u8.M3U8, minFiles int64, minDuration time.Duration, start time.Time) {
+ fmt.Println("path '" + r.URL.Path + "'")
+ offset := time.Since(start)
+ // TODO cache for 1s?
+ live, err := m3u8.TransformVodToLive(vod, offset, minFiles, minDuration)
+ if err != nil {
+ fmt.Println("Error transforming : " + err.Error())
+ w.WriteHeader(http.StatusInternalServerError)
+ return
+ }
+ liveBts := m3u8.SerializeLive(live)
+ w.Header().Set("Content-Type", "application/x-mpegURL")
+ w.Write(liveBts)
+}
+
+func staticHandler(e staticHTTPEndpoint) (http.HandlerFunc, error) {
+ data, err := ioutil.ReadFile(e.FilePath) // read even if we're not caching, so we can error out before serving
+ if err != nil {
+ return nil, errors.New("reading file '" + e.FilePath + "': " + err.Error())
+ }
+ return func(w http.ResponseWriter, r *http.Request) {
+ for key, values := range e.Headers {
+ w.Header().Set(key, values[0])
+ for _, val := range values[1:] {
+ w.Header().Add(key, val)
+ }
+ }
+ if e.NoCache {
+ if data, err = ioutil.ReadFile(e.FilePath); err != nil {
+ w.WriteHeader(http.StatusNotFound)
+ // Remove the error message in production, this is just for debug purposes
+ w.Write([]byte(strconv.Itoa(http.StatusNotFound) + " " + http.StatusText(http.StatusNotFound) + " - " + err.Error()))
+ return
+ }
+ }
+ w.Header().Set("Content-Type", http.DetectContentType(data)) // TODO add Content Type to config, to allow users to override detected type
+ w.Write(data)
+ }, nil
+
+}
+
+func crossdomainHandler(customCrossdomainFile string) http.HandlerFunc {
+ return func(w http.ResponseWriter, r *http.Request) {
+ var buf []byte
+ if customCrossdomainFile != "" && strings.HasSuffix(customCrossdomainFile, ".xml") {
+ _, ExistErr := os.Stat(customCrossdomainFile)
+ if os.IsNotExist(ExistErr) {
+ fmt.Println("Error Crossdomain File Does Not Exist")
+ w.WriteHeader(http.StatusInternalServerError)
+ return
+ }
+ var err error
+ buf, err = ioutil.ReadFile(customCrossdomainFile)
+ if err != nil {
+ fmt.Println("Error Reading Crossdomain File : " + err.Error())
+ w.WriteHeader(http.StatusInternalServerError)
+ return
+ }
+ tmp := ""
+ err = xml.Unmarshal(buf, &tmp)
+ if err != nil {
+ fmt.Println("Error Crossdomain File is not valid XML: " + err.Error())
+ w.WriteHeader(http.StatusInternalServerError)
+ return
+ }
+ }
+ w.Write(buf)
+ }
+}
diff --git a/test/fakeOrigin/httpService/httpService.go b/test/fakeOrigin/httpService/httpService.go
new file mode 100644
index 0000000..5b1dd2c
--- /dev/null
+++ b/test/fakeOrigin/httpService/httpService.go
@@ -0,0 +1,277 @@
+package httpService
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "crypto/tls"
+ "errors"
+ "fmt"
+ "io"
+ "io/ioutil"
+ "net/http"
+ "os"
+ "path"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/apache/trafficcontrol/test/fakeOrigin/endpoint"
+ "github.com/apache/trafficcontrol/test/fakeOrigin/transcode"
+)
+
+// EndpointRoutes contains the paths of all HTTP routes for a given config "endpoint"
+// TODO come up with a better structure. The static and abr checks are hackish, the structure should only contain the necessary variables for the given route. Maybe there's a better way with interfaces?
+type EndpointRoutes struct {
+ MasterPath string
+ VariantPaths []string
+ FragmentPaths []string
+ MetaJSONPaths []string
+ IsABR bool
+}
+
+// GetRoutes returns the map of config IDs, to the full HTTP paths, e.g. for each m3u8 and ts created for that config object.
+func GetRoutes(cfg endpoint.Config) (map[string]EndpointRoutes, error) {
+ // TODO return both HTTP route and file path, so the handler doesn't have to rebuild the file path to load the file from disk
+ allRoutes := map[string]EndpointRoutes{}
+ for _, ep := range cfg.Endpoints {
+ routes := EndpointRoutes{}
+ if ep.EndpointType == endpoint.Static {
+ routes.MasterPath = path.Join("/", ep.ID, filepath.Base(ep.Source))
+ allRoutes[ep.ID] = routes
+ continue
+ }
+ if ep.EndpointType == endpoint.Dir {
+ fileList := []string{}
+ err := filepath.Walk(ep.Source, func(path string, f os.FileInfo, err error) error {
+ if err != nil {
+ return err
+ }
+ // Skip directories since that's not a static file
+ if !f.IsDir() {
+ fileList = append(fileList, path)
+ }
+ return nil
+ })
+ if err != nil {
+ return nil, errors.New("reading files '" + ep.Source + "': " + err.Error())
+ }
+
+ for _, file := range fileList {
+ pathsuffix := strings.TrimPrefix(file, ep.Source)
+ routes.VariantPaths = append(routes.VariantPaths, path.Join("/", ep.ID, pathsuffix))
+ }
+ allRoutes[ep.ID] = routes
+ continue
+ }
+
+ files, err := ioutil.ReadDir(ep.OutputDirectory)
+ if err != nil {
+ return nil, errors.New("reading files '" + ep.Source + "': " + err.Error())
+ }
+ routes.IsABR = false
+ for _, f := range files {
+ if !f.IsDir() {
+ if strings.ToLower(filepath.Ext(f.Name())) == ".ts" {
+ routes.FragmentPaths = append(routes.FragmentPaths, path.Join("/", ep.ID, f.Name()))
+ } else if f.Name() == ep.OutputDirectory+"/"+ep.DiskID+".m3u8" {
+ routes.MasterPath = path.Join("/", ep.ID, f.Name())
+ } else if strings.ToLower(filepath.Ext(f.Name())) == ".m3u8" {
+ routes.IsABR = true
+ routes.VariantPaths = append(routes.VariantPaths, path.Join("/", ep.ID, f.Name()))
+ } else if strings.HasSuffix(strings.ToLower(f.Name()), ".meta.json") {
+ routes.MetaJSONPaths = append(routes.MetaJSONPaths, path.Join("/", ep.ID, ep.DiskID+".meta.json"))
+ }
+ }
+ }
+
+ allRoutes[ep.ID] = routes
+ }
+ return allRoutes, nil
+}
+
+// PrintRoutes writes the routes across multiple lines in human-readable format to the given writer.
+// The header is written at the beginning of each endpoint. The routePrefix is written at the beginning of each route.
+// If showFragments is false, only manifests and meta files are printed, not video fragments.
+func PrintRoutes(w io.Writer, routes map[string]EndpointRoutes, endpointPrefix, routePrefix string, showFragments bool) {
+ s := ""
+ for endpointID, endpointRoutes := range routes {
+ s += endpointPrefix + endpointID + "\n"
+ if endpointRoutes.MasterPath != "" {
+ s += routePrefix + endpointRoutes.MasterPath + "\n"
+ }
+ for _, path := range endpointRoutes.VariantPaths {
+ s += routePrefix + path + "\n"
+ }
+ for _, path := range endpointRoutes.MetaJSONPaths {
+ s += routePrefix + path + "\n"
+ }
+ if showFragments {
+ for _, path := range endpointRoutes.FragmentPaths {
+ s += routePrefix + path + "\n"
+ }
+ }
+ }
+ w.Write([]byte(s))
+}
+
+const ContentTypeJSON = `application/json`
+const ContentTypeM3U8 = `application/x-mpegURL`
+const ContentTypeTS = `video/MP2T`
+
+func registerRoute(mux *http.ServeMux, e endpoint.Endpoint, httpPath string, isLive bool, contentType string, isSSL bool) error {
+ startTime := time.Now()
+ if e.EndpointType == endpoint.Static || e.EndpointType == endpoint.Dir {
+ diskpath := e.Source
+ if e.EndpointType == endpoint.Dir {
+ diskpath = path.Join("/", e.Source, strings.TrimPrefix(httpPath, "/"+e.ID+"/"))
+ }
+ h, err := staticHandler(staticHTTPEndpoint{
+ FilePath: diskpath,
+ NoCache: e.NoCache,
+ Headers: e.DefaultHeaders,
+ })
+ if err != nil {
+ return errors.New("creating handler '" + httpPath + "': " + err.Error())
+ }
+ if isSSL {
+ mux.Handle(httpPath, log(strictTransportSecurity(originHeaderManipulation(h))))
+ mux.Handle(httpPath+"/", log(strictTransportSecurity(originHeaderManipulation(h))))
+ } else {
+ mux.Handle(httpPath, log(originHeaderManipulation(h)))
+ mux.Handle(httpPath+"/", log(originHeaderManipulation(h)))
+ }
+ return nil
+ }
+
+ var lastTranscodeDT time.Time
+ if diskmeta, err := transcode.GetMeta(e); err != nil {
+ lastTranscodeDT = startTime
+ } else {
+ var timeOk bool
+ if lastTranscodeDT, timeOk = ParseHTTPDate(diskmeta.LastTranscodeDT); !timeOk {
+ lastTranscodeDT = startTime
+ }
+ }
+ ep := httpEndpoint{
+ ID: e.ID,
+ OutputDir: e.OutputDirectory,
+ Type: e.EndpointType,
+ DiskID: e.DiskID,
+ IsLive: isLive,
+ DefaultHeaders: e.DefaultHeaders,
+ NoCache: e.NoCache,
+ HTTPPath: httpPath,
+ ContentType: contentType,
+ LastTranscodeTime: lastTranscodeDT,
+ }
+ h, err := httpEndpointHandler(ep)
+ if err != nil {
+ return errors.New("registering route '" + httpPath + "': " + err.Error())
+ }
+ if isSSL {
+ mux.Handle(httpPath, log(strictTransportSecurity(originHeaderManipulation(cacheOptimization(h, startTime, ep)))))
+ mux.Handle(httpPath+"/", log(strictTransportSecurity(originHeaderManipulation(cacheOptimization(h, startTime, ep)))))
+ } else {
+ mux.Handle(httpPath, log(originHeaderManipulation(cacheOptimization(h, startTime, ep))))
+ mux.Handle(httpPath+"/", log(originHeaderManipulation(cacheOptimization(h, startTime, ep))))
+ }
+ return nil
+}
+
+func registerRoutes(mux *http.ServeMux, conf endpoint.Config, routes map[string]EndpointRoutes, isSSL bool) error {
+ for _, e := range conf.Endpoints {
+ endpointRoutes, ok := routes[e.ID]
+ if !ok {
+ return errors.New("no routes found for endpoint '" + e.ID + "'")
+ }
+ if endpointRoutes.MasterPath != "" {
+ err := registerRoute(mux, e, endpointRoutes.MasterPath, e.EndpointType == endpoint.Live && !endpointRoutes.IsABR, ContentTypeM3U8, isSSL)
+ if err != nil {
+ return errors.New("Error registering endpoint '" + e.ID + "': " + err.Error())
+ }
+ }
+ for _, path := range endpointRoutes.VariantPaths {
+ err := registerRoute(mux, e, path, e.EndpointType == endpoint.Live && endpointRoutes.IsABR, ContentTypeM3U8, isSSL)
+ if err != nil {
+ return errors.New("Error registering endpoint '" + e.ID + "': " + err.Error())
+ }
+ }
+ for _, path := range endpointRoutes.FragmentPaths {
+ err := registerRoute(mux, e, path, false, ContentTypeTS, isSSL)
+ if err != nil {
+ return errors.New("Error registering endpoint '" + e.ID + "': " + err.Error())
+ }
+ }
+ for _, path := range endpointRoutes.MetaJSONPaths {
+ err := registerRoute(mux, e, path, false, ContentTypeJSON, isSSL)
+ if err != nil {
+ return errors.New("Error registering endpoint '" + e.ID + "': " + err.Error())
+ }
+ }
+ }
+ mux.Handle("/crossdomain.xml", log(crossdomainHandler(conf.ServerConf.CrossdomainFile)))
+ mux.Handle("/", log(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusNotFound)
+ w.Write([]byte(http.StatusText(http.StatusNotFound)))
+ })))
+ return nil
+}
+
+// StartHTTPListener kicks off the HTTPS stack
+func StartHTTPListener(conf endpoint.Config, routes map[string]EndpointRoutes) error {
+ mux := http.NewServeMux()
+ if err := registerRoutes(mux, conf, routes, false); err != nil {
+ return errors.New("registering routes: " + err.Error())
+ }
+ fmt.Println("Serving HTTP on " + conf.ServerConf.BindingAddress + ":" + strconv.Itoa(conf.ServerConf.HTTPListeningPort))
+ return http.ListenAndServe(conf.ServerConf.BindingAddress+":"+strconv.Itoa(conf.ServerConf.HTTPListeningPort), mux)
+}
+
+// StartHTTPSListener kicks off the HTTPS stack
+func StartHTTPSListener(conf endpoint.Config, routes map[string]EndpointRoutes) error {
+ if err := assertSSLCerts(conf.ServerConf.SSLCert, conf.ServerConf.SSLKey); err != nil {
+ return fmt.Errorf("asserting SSL info Cert:'%+v' Key:'%+v': %+v", conf.ServerConf.SSLCert, conf.ServerConf.SSLKey, err)
+ }
+ mux := http.NewServeMux()
+ if err := registerRoutes(mux, conf, routes, true); err != nil {
+ return errors.New("registering routes: " + err.Error())
+ }
+
+ cfg := &tls.Config{
+ MinVersion: tls.VersionTLS12,
+ CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
+ PreferServerCipherSuites: true,
+ CipherSuites: []uint16{
+ tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
+ tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
+ tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
+ tls.TLS_RSA_WITH_AES_256_CBC_SHA,
+ },
+ }
+ srv := &http.Server{
+ Addr: conf.ServerConf.BindingAddress + ":" + strconv.Itoa(conf.ServerConf.HTTPSListeningPort),
+ Handler: mux,
+ TLSConfig: cfg,
+ TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
+ }
+ fmt.Println("Serving HTTPS on " + conf.ServerConf.BindingAddress + ":" + strconv.Itoa(conf.ServerConf.HTTPSListeningPort))
+ return srv.ListenAndServeTLS(conf.ServerConf.SSLCert, conf.ServerConf.SSLKey)
+}
diff --git a/test/fakeOrigin/httpService/rangerequest.go b/test/fakeOrigin/httpService/rangerequest.go
new file mode 100644
index 0000000..6d330f3
--- /dev/null
+++ b/test/fakeOrigin/httpService/rangerequest.go
@@ -0,0 +1,130 @@
+package httpService
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+)
+
+type httpRange struct {
+ startOffset uint64
+ length uint64
+}
+
+const multipartSeparator = "3d6b6a416f9b5fakeOrigin"
+
+func parseRange(rawRange string, size uint64) ([]httpRange, error) {
+ var out []httpRange
+ if rawRange == "" {
+ return nil, nil
+ }
+ if !strings.HasPrefix(rawRange, "bytes=") {
+ return nil, errors.New("invalid range")
+ }
+ for _, r := range strings.Split(strings.TrimPrefix(rawRange, "bytes="), ",") {
+ r = strings.TrimSpace(r)
+ if r == "" {
+ continue
+ }
+ if !strings.Contains(r, "-") {
+ return nil, errors.New("invalid range")
+ }
+ offsets := strings.Split(r, "-")
+ if len(offsets) > 2 || len(offsets) <= 0 {
+ return nil, errors.New("invalid range")
+ }
+ start := strings.TrimSpace(offsets[0])
+ startI, err := strconv.ParseUint(start, 10, 64)
+ if err != nil || startI >= size {
+ return nil, errors.New("invalid range")
+ }
+ end := strings.TrimSpace(offsets[1])
+ var endI uint64
+ if end != "" {
+ endI, err = strconv.ParseUint(end, 10, 64)
+ if err != nil || endI < startI {
+ return nil, errors.New("invalid range")
+ }
+ // clip to size of body
+ if endI >= size {
+ endI = size - 1
+ }
+ }
+ var iout httpRange
+ if start == "" {
+ // relative to end of file
+ iout.startOffset = size - endI
+ iout.length = size - iout.startOffset
+ } else {
+ iout.startOffset = startI
+ if end == "" {
+ // all to end of file
+ iout.length = size - startI
+ } else {
+ // mid-range
+ iout.length = endI - startI + 1
+ }
+ }
+ out = append(out, iout)
+ }
+
+ return out, nil
+}
+
+func getContentRangeHeader(start, length, totalSize uint64) string {
+ var end uint64
+ if start+length > totalSize-1 {
+ end = totalSize - 1
+ } else {
+ end = start + length
+ }
+ return fmt.Sprintf("bytes=%d-%d/%d", start, end, totalSize)
+}
+
+func clipToRange(ranges []httpRange, obody []byte, contentHeader string) ([]byte, map[string]string, error) {
+ totalSize := uint64(len(obody))
+ // need to use existing instead of appending since we have to lookup 206 time mismatches
+ // this also means we have to start dealing with slice string values
+
+ if len(ranges) == 0 {
+ return nil, nil, errors.New("No ranges supplied")
+ } else if len(ranges) == 1 {
+ // single part ranges
+ r := ranges[0]
+ b := obody[r.startOffset : r.startOffset+r.length]
+ // Update response code and other headers based on isTimeMatch
+ return b, map[string]string{"Content-Range": getContentRangeHeader(r.startOffset, r.startOffset+r.length-1, totalSize)}, nil
+ }
+ // multipart
+ var b []byte
+ for i := range ranges {
+ b = append(b, []byte("--"+multipartSeparator+"\n")...)
+ b = append(b, []byte("Content-Type: "+contentHeader+"\n")...)
+ b = append(b, []byte("Content-Range: "+getContentRangeHeader(ranges[i].startOffset, ranges[i].startOffset+ranges[i].length-1, totalSize)+"\n\n")...)
+ b = append(b, obody[ranges[i].startOffset:ranges[i].startOffset+ranges[i].length]...)
+ b = append(b, []byte("\n")...)
+ }
+ b = append(b, []byte("--"+multipartSeparator+"--\n")...)
+ //w.Header().Set("Content-Type", "multipart/byteranges; boundary="+multipartSeparator)
+ return b, map[string]string{"Content-Type": "multipart/byteranges; boundary=" + multipartSeparator}, nil
+}
diff --git a/test/fakeOrigin/httpService/ssl.keygen.go b/test/fakeOrigin/httpService/ssl.keygen.go
new file mode 100644
index 0000000..652f3f7
--- /dev/null
+++ b/test/fakeOrigin/httpService/ssl.keygen.go
@@ -0,0 +1,42 @@
+package httpService
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "os"
+
+ "github.com/apache/trafficcontrol/test/fakeOrigin/transcode"
+)
+
+func assertSSLCerts(crtPath, keyPath string) error {
+ if _, err := os.Stat(keyPath); os.IsNotExist(err) {
+ if err = transcode.RunSynchronousCmd("openssl", []string{"genrsa", "-out", keyPath, "2048"}); err != nil {
+ return errors.New("generating new ssl key '" + os.Args[0] + "': " + err.Error())
+ }
+ }
+ if _, err := os.Stat(crtPath); os.IsNotExist(err) {
+ if err = transcode.RunSynchronousCmd("openssl", []string{"req", "-new", "-x509", "-sha256", "-key", keyPath, "-out", crtPath, "-days", "3650", "-subj", "/CN=localhost"}); err != nil {
+ return errors.New("generating new ssl cert '" + os.Args[0] + "': " + err.Error())
+ }
+ }
+
+ return nil
+}
diff --git a/test/fakeOrigin/m3u8/m3u8.go b/test/fakeOrigin/m3u8/m3u8.go
new file mode 100644
index 0000000..475a523
--- /dev/null
+++ b/test/fakeOrigin/m3u8/m3u8.go
@@ -0,0 +1,328 @@
+package m3u8
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "errors"
+ "fmt"
+ "io/ioutil"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "time"
+)
+
+type M3U8 struct {
+ Version string
+ HasVersion bool
+ TargetDuration time.Duration
+ HasTargetDuration bool
+ MediaSequence int64
+ HasMediaSequence bool
+ TSes []M3U8TS
+ VARs []M3U8VAR
+}
+
+type M3U8TS struct {
+ Duration time.Duration
+ Discontinuity bool
+ Filepath string
+}
+
+type M3U8VAR struct {
+ Bandwidth int
+ Resolution string
+ Filepath string
+}
+
+func Parse(fileBytes []byte) (M3U8, error) {
+ lines := bytes.Split(fileBytes, []byte("\n"))
+
+ // if bytes.TrimSpace(lines[0]) != []byte("#EXTM3U") {
+ // fmt.Println("Error parsing m3u8: file must begin with '#EXTM3U'")
+ // return
+ // }
+
+ m3u8 := M3U8{}
+
+ inTarget := false
+ ts := M3U8TS{}
+ variantStream := M3U8VAR{}
+linesLoop:
+ for _, line := range lines {
+ line = bytes.TrimSpace(line)
+ if len(line) == 0 {
+ continue
+ }
+ if !inTarget {
+ switch tagType(line) {
+ case TagTypeHeader:
+ continue
+ case TagTypeVersion:
+ m3u8.Version = string(bytes.TrimSpace(line[len(VersionPrefix):]))
+ m3u8.HasVersion = true
+ continue
+ case TagTypeTargetDuration:
+ targetDurationStr := string(bytes.TrimSpace(line[len(TargetDurationPrefix):]))
+ targetDurationSeconds, err := strconv.ParseFloat(targetDurationStr, 64)
+ if err != nil {
+ return M3U8{}, errors.New("parsing m3u8: " + TargetDurationPrefix + " not a number")
+ }
+ m3u8.TargetDuration = time.Duration(targetDurationSeconds * float64(time.Second))
+ m3u8.HasTargetDuration = true
+ continue
+ case TagTypeMediaSequence:
+ mediaSequenceStr := string(bytes.TrimSpace(line[len(MediaSequencePrefix):]))
+ mediaSequence, err := strconv.ParseInt(mediaSequenceStr, 10, 64)
+ if err != nil {
+ return M3U8{}, errors.New("parsing m3u8: " + MediaSequencePrefix + " not a number")
+ }
+ m3u8.MediaSequence = mediaSequence
+ m3u8.HasMediaSequence = true
+ continue
+ case TagTypeTSInfo:
+ fields := bytes.Split(line[len(TSInfoPrefix):], []byte(","))
+ durationStr := bytes.TrimSpace(fields[0])
+ if len(durationStr) == 0 {
+ return M3U8{}, errors.New("parsing m3u8: malformed line '" + string(line))
+ }
+ durationSeconds, err := strconv.ParseFloat(string(durationStr), 64)
+ if err != nil {
+ return M3U8{}, errors.New("parsing m3u8: line '" + string(line) + "' duration '" + string(durationStr) + "' not a number")
+ }
+ ts.Duration = time.Duration(durationSeconds * float64(time.Second))
+ inTarget = true
+ continue
+ case TagTypeVariantStream:
+ fields := bytes.Split(line[len(VariantPrefix):], []byte(","))
+ variantArgs := make(map[string]string)
+ for _, kvp := range fields {
+ temp := bytes.Split(kvp, []byte("="))
+ k := string(temp[0])
+ v := string(temp[1])
+ k = strings.ToLower(k)
+ v = strings.ToLower(v)
+ if string(temp[0]) != "" {
+ variantArgs[strings.ToLower(string(temp[0]))] = strings.ToLower(string(temp[1]))
+ }
+ }
+ if _, ok := variantArgs["bandwidth"]; ok {
+ var strconverror error
+ variantStream.Bandwidth, strconverror = strconv.Atoi(variantArgs["bandwidth"])
+ if strconverror != nil {
+ return M3U8{}, errors.New("parsing m3u8: variant stream bandwidth malformed: " + variantArgs["bandwidth"] + " error: " + strconverror.Error())
+ }
+ } else {
+ return M3U8{}, errors.New("parsing m3u8: variant stream bandwidth missing '" + string(line) + "'")
+ }
+ variantStream.Resolution = variantArgs["resolution"]
+ if variantStream.Resolution == "" {
+ return M3U8{}, errors.New("parsing m3u8: variant stream resolution missing '" + string(line) + "'")
+ }
+
+ inTarget = true
+ continue
+ case TagTypeEndList:
+ break linesLoop
+ // TODO handle #EXT-X-DISCONTINUITY
+ default:
+ if line[0] == '#' {
+ fmt.Println("Warning: parsing m3u8: line '" + string(line) + "' unknown directive")
+ continue
+ }
+ return M3U8{}, errors.New("parsing m3u8: unknown line '" + string(line))
+ }
+ } else {
+ switch targetType(line) {
+ case TargetTypeTS:
+ ts.Filepath = string(line)
+ m3u8.TSes = append(m3u8.TSes, ts)
+ inTarget = false
+ ts = M3U8TS{}
+ variantStream = M3U8VAR{}
+ case TargetTypeManifest:
+ variantStream.Filepath = string(line)
+ m3u8.VARs = append(m3u8.VARs, variantStream)
+ inTarget = false
+ ts = M3U8TS{}
+ variantStream = M3U8VAR{}
+ default:
+ return M3U8{}, errors.New("parsing m3u8: unknown target line '" + string(line))
+ }
+ }
+ }
+ return m3u8, nil
+}
+
+type TagType int
+
+const (
+ TagTypeInvalid TagType = iota
+ TagTypeHeader
+ TagTypeVersion
+ TagTypeTargetDuration
+ TagTypeMediaSequence
+ TagTypeTSInfo
+ TagTypeEndList
+ TagTypeVariantStream
+)
+
+type TargetType int
+
+const (
+ TargetTypeInvalid TargetType = iota
+ TargetTypeManifest
+ TargetTypeTS
+)
+
+const HeaderPrefix = "#EXTM3U"
+const VersionPrefix = "#EXT-X-VERSION:"
+const TargetDurationPrefix = "#EXT-X-TARGETDURATION:"
+const MediaSequencePrefix = "#EXT-X-MEDIA-SEQUENCE:"
+const TSInfoPrefix = "#EXTINF:"
+const EndListPrefix = "#EXT-X-ENDLIST"
+const VariantPrefix = "#EXT-X-STREAM-INF:"
+const TSSuffix = ".ts"
+const ManifestSuffix = ".m3u8"
+
+func tagType(line []byte) TagType {
+ switch {
+ case bytes.HasPrefix(line, []byte(HeaderPrefix)):
+ return TagTypeHeader
+ case bytes.HasPrefix(line, []byte(VersionPrefix)):
+ return TagTypeVersion
+ case bytes.HasPrefix(line, []byte(TargetDurationPrefix)):
+ return TagTypeTargetDuration
+ case bytes.HasPrefix(line, []byte(MediaSequencePrefix)):
+ return TagTypeMediaSequence
+ case bytes.HasPrefix(line, []byte(TSInfoPrefix)):
+ return TagTypeTSInfo
+ case bytes.HasPrefix(line, []byte(EndListPrefix)):
+ return TagTypeEndList
+ case bytes.HasPrefix(line, []byte(VariantPrefix)):
+ return TagTypeVariantStream
+ }
+ return TagTypeInvalid
+}
+
+func targetType(line []byte) TargetType {
+ switch {
+ case bytes.HasSuffix(line, []byte(TSSuffix)):
+ return TargetTypeTS
+ case bytes.HasSuffix(line, []byte(ManifestSuffix)):
+ return TargetTypeManifest
+ }
+ return TargetTypeInvalid
+}
+
+// TransformVodToLive creates a live m3u8 from a given VOD m3u8.
+// The created m3u8 will have TS files, of the lesser of MinLiveFiles and MinLiveDuration
+func TransformVodToLive(vod M3U8, offset time.Duration, minFiles int64, minDuration time.Duration) (M3U8, error) {
+ if !vod.HasVersion {
+ return M3U8{}, errors.New("vod must have version") // TODO default version
+ }
+ if !vod.HasTargetDuration {
+ return M3U8{}, errors.New("vod must have taget duration") // TODO compute target duration
+ }
+ if vod.HasMediaSequence && vod.MediaSequence != 0 {
+ return M3U8{}, errors.New("vod must not have media sequence")
+ }
+ if len(vod.TSes) < 1 {
+ return M3U8{}, errors.New("vod must have at least 1 TS")
+ }
+
+ live := M3U8{}
+ live.Version = vod.Version
+ live.TargetDuration = vod.TargetDuration
+ live.MediaSequence = GetLiveMediaSequence(vod, offset)
+
+ totalDuration := time.Duration(0)
+ totalFiles := int64(0)
+ for i := live.MediaSequence; totalFiles < minFiles && totalDuration < minDuration; i++ {
+ vodIdx := i % int64(len(vod.TSes))
+ ts := vod.TSes[vodIdx]
+ ts.Discontinuity = vodIdx == 0
+ live.TSes = append(live.TSes, ts)
+ totalDuration += ts.Duration
+ totalFiles++
+ }
+ return live, nil
+}
+
+// GetLiveMediaSequence gets the live media sequence, from the vod m3u8.
+// That is, if vod has 3 TSes each 2 seconds long, and offset is 3s, this returns 2, because the 3rd second occurs within the 2nd TS (which starts at second 2 and is 2 seconds long, i.e. contains absolute seconds 2-4 of the video).
+// The returned MediaSequence should be modded on the length of the TSes, to get the proper TSes to serve.
+// Each time the mod is 0, the 0 TS should also insert a #EXT-X-DISCONTINUITY into the manifest, via M3U8TS.Discontinuity, in order to play correctly, assuming the TS files have correct continuity counters.
+func GetLiveMediaSequence(vod M3U8, offset time.Duration) int64 {
+ if len(vod.TSes) == 0 {
+ return 0
+ }
+ i := int64(0)
+ for sum := vod.TSes[0].Duration; sum <= offset; i++ {
+ nextTS := vod.TSes[(i+1)%int64(len(vod.TSes))]
+ sum += nextTS.Duration
+ }
+ return i
+}
+
+// SerializeLiveM3U8 serialized the given M3U8 object as a live m3u8 manifest file.
+// It assumes the given m3u8 is valid, and does not check validity or existence of fields. Callers should check validity before calling (a M3U8 returned by TransformVodToLive without error is guaranteed valid).
+func SerializeLive(m3u8 M3U8) []byte {
+ b := []byte(`#EXTM3U
+#EXT-X-VERSION:` + m3u8.Version + `
+#EXT-X-TARGETDURATION:` + strconv.FormatFloat(float64(m3u8.TargetDuration)/float64(time.Second), 'f', -1, 64) + `
+#EXT-X-MEDIA-SEQUENCE:` + strconv.FormatInt(m3u8.MediaSequence, 10) + `
+`)
+ for _, ts := range m3u8.TSes {
+ if ts.Discontinuity {
+ b = append(b, []byte(`#EXT-X-DISCONTINUITY
+`)...)
+ }
+ b = append(b, []byte(`#EXTINF:`+strconv.FormatFloat(float64(ts.Duration)/float64(time.Second), 'f', 6, 64)+`,
+`)...)
+ b = append(b, []byte(ts.Filepath+`
+`)...)
+ }
+ return b
+}
+
+// GetTotalTime returns the sum of the durations of all TSes in the given vod M3U8.
+func GetTotalTime(vod M3U8) time.Duration {
+ sum := time.Duration(0)
+ for _, ts := range vod.TSes {
+ sum += ts.Duration
+ }
+ return sum
+}
+
+func LoadTSes(m3u8 M3U8, m3u8Path string) (map[string][]byte, error) {
+ tses := map[string][]byte{}
+ dir := filepath.Dir(m3u8Path)
+ for _, ts := range m3u8.TSes {
+ path := filepath.Join(dir, ts.Filepath)
+ bts, err := ioutil.ReadFile(path)
+ if err != nil {
+ return nil, errors.New("loading '" + path + "':" + err.Error())
+ }
+ tses[ts.Filepath] = bts
+ }
+ return tses, nil
+}
diff --git a/test/fakeOrigin/shard.sh b/test/fakeOrigin/shard.sh
new file mode 100755
index 0000000..fa277f3
--- /dev/null
+++ b/test/fakeOrigin/shard.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+# brew install ffmpeg --with-rtmp-dump
+# brew install mp4box
+
+# Sample Invokation:
+# ./shard.sh ./example/video/kelloggs.mp4 ./out/shard.hls.vod HLS_VOD
+# ./shard.sh ./example/video/kelloggs.mp4 ./out/shard.hls.live HLS_LIVE
+# ./shard.sh ./example/video/kelloggs.mp4 ./out/shard.dash.vod DASH_VOD
+# ./shard.sh ./example/video/kelloggs.mp4 ./out/shard.dash.live DASH_LIVE
+
+# This script is to serve as an example of external transcoder for VOD assets that may be plumbed inside fakeOrigin as a VOD endpoint or static Directory of files
+
+
+src=$1
+destination=$2
+format=$3
+filename=$(basename "${destination}")
+
+if [ $# -ne 3 ]; then
+ echo 1>&2 "Usage: $0 <Sourcefile> <Destdirectory> <Type (HLS_VOD|HLS_LIVE|DASH_VOD|DASH_LIVE)>"
+ exit 3
+fi
+
+mkdir -p $destination
+
+set -x
+# HLS VOD
+if [[ $format == "HLS_VOD" ]]; then
+ffmpeg -y -i "${src}" \
+ -vf scale=w=640:h=360:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 800k -maxrate 856k -bufsize 1200k -b:a 96k -hls_segment_filename "${destination}/${filename}_seq_%03d_640x360-10000.ts" "${destination}/${filename}_640x360-10000.m3u8" \
+ -vf scale=w=842:h=480:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 1400k -maxrate 1498k -bufsize 2100k -b:a 128k -hls_segment_filename "${destination}/${filename}_seq_%03d_842x480-20000.ts" "${destination}/${filename}_842x480-20000.m3u8" \
+ -vf scale=w=1280:h=720:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 2800k -maxrate 2996k -bufsize 4200k -b:a 128k -hls_segment_filename "${destination}/${filename}_seq_%03d_1280x720-40000.ts" "${destination}/${filename}_1280x720-40000.m3u8" \
+ -vf scale=w=1920:h=1080:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 5000k -maxrate 5350k -bufsize 7500k -b:a 192k -hls_segment_filename "${destination}/${filename}_seq_%03d_1920x1080-80000.ts" "${destination}/${filename}_1920x1080-80000.m3u8"
+fi
+
+# HLS LIVE
+if [[ $format == "HLS_LIVE" ]]; then
+ffmpeg -y -i "${src}" \
+ -vf scale=w=640:h=360:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -b:v 800k -maxrate 856k -bufsize 1200k -b:a 96k -hls_segment_filename "${destination}/${filename}_seq_%03d_640x360-10000.ts" "${destination}/${filename}_640x360-10000.m3u8" \
+ -vf scale=w=842:h=480:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -b:v 1400k -maxrate 1498k -bufsize 2100k -b:a 128k -hls_segment_filename "${destination}/${filename}_seq_%03d_842x480-20000.ts" "${destination}/${filename}_842x480-20000.m3u8" \
+ -vf scale=w=1280:h=720:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -b:v 2800k -maxrate 2996k -bufsize 4200k -b:a 128k -hls_segment_filename "${destination}/${filename}_seq_%03d_1280x720-40000.ts" "${destination}/${filename}_1280x720-40000.m3u8" \
+ -vf scale=w=1920:h=1080:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -b:v 5000k -maxrate 5350k -bufsize 7500k -b:a 192k -hls_segment_filename "${destination}/${filename}_seq_%03d_1920x1080-80000.ts" "${destination}/${filename}_1920x1080-80000.m3u8"
+fi
+
+# DASH Live
+if [[ $format == "DASH_LIVE" ]]; then
+ffmpeg -i "${src}" -c:a copy -vn "${destination}/${filename}-audio.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 5300k -maxrate 5300k -bufsize 2650k -vf 'scale=-1:1080' "${destination}/${filename}-1080.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 2400k -maxrate 2400k -bufsize 1200k -vf 'scale=-1:720' "${destination}/${filename}-720.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 1060k -maxrate 1060k -bufsize 530k -vf 'scale=-1:480' "${destination}/${filename}-480.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 600k -maxrate 600k -bufsize 300k -vf 'scale=-1:360' "${destination}/${filename}-360.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 260k -maxrate 260k -bufsize 130k -vf 'scale=-1:240' "${destination}/${filename}-240.mp4"
+
+MP4Box -dash 1000 -rap -frag-rap -profile live -out "${destination}/$filename.mpd" "${destination}/${filename}-1080.mp4" "${destination}/${filename}-720.mp4" "${destination}/${filename}-480.mp4" "${destination}/${filename}-360.mp4" "${destination}/${filename}-240.mp4" "${destination}/${filename}-audio.mp4"
+fi
+
+# DASH VOD
+if [[ $format == "DASH_VOD" ]]; then
+ffmpeg -i "${src}" -c:a copy -vn "${destination}/${filename}-audio.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 5300k -maxrate 5300k -bufsize 2650k -vf 'scale=-1:1080' "${destination}/${filename}-1080.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 2400k -maxrate 2400k -bufsize 1200k -vf 'scale=-1:720' "${destination}/${filename}-720.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 1060k -maxrate 1060k -bufsize 530k -vf 'scale=-1:480' "${destination}/${filename}-480.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 600k -maxrate 600k -bufsize 300k -vf 'scale=-1:360' "${destination}/${filename}-360.mp4"
+ffmpeg -i "${src}" -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 260k -maxrate 260k -bufsize 130k -vf 'scale=-1:240' "${destination}/${filename}-240.mp4"
+
+MP4Box -dash 1000 -rap -frag-rap -profile onDemand -out "${destination}/$filename.mpd" "${destination}/${filename}-1080.mp4" "${destination}/${filename}-720.mp4" "${destination}/${filename}-480.mp4" "${destination}/${filename}-360.mp4" "${destination}/${filename}-240.mp4" "${destination}/${filename}-audio.mp4"
+fi
diff --git a/test/fakeOrigin/transcode/transcode.go b/test/fakeOrigin/transcode/transcode.go
new file mode 100644
index 0000000..57ca4b4
--- /dev/null
+++ b/test/fakeOrigin/transcode/transcode.go
@@ -0,0 +1,222 @@
+package transcode
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "hash/crc32"
+ "io"
+ "io/ioutil"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "reflect"
+ "regexp"
+ "time"
+
+ "github.com/apache/trafficcontrol/test/fakeOrigin/endpoint"
+)
+
+type TranscodeMeta struct {
+ CrcHash string `json:"crc_hash"`
+ Cmd string `json:"cmd"`
+ Args []string `json:"args"`
+ LastTranscodeDT string `json:"last_transcode_datetime"`
+}
+
+func Do(ep *endpoint.Endpoint, cmdStr string, args []string) error {
+ fmt.Printf("Creating directory: %s\n", ep.OutputDirectory)
+ err := os.MkdirAll(ep.OutputDirectory, os.ModePerm)
+ if err != nil {
+ return err
+ }
+ fmt.Println("Directory creation complete")
+
+ isMatch := false
+ isMatch, err = checkMeta((*ep), cmdStr, args)
+ if err != nil {
+ return err
+ }
+ if !isMatch {
+ fmt.Println("Transcode Meta Check Failed")
+ fmt.Println("Beginning Transcode")
+ err = RunSynchronousCmd(cmdStr, args)
+ fmt.Println("Transcode complete")
+ if err != nil {
+ return err
+ }
+ err = generateMasterManifest(ep)
+ if err != nil {
+ return err
+ }
+ err = generateMeta((*ep), cmdStr, args)
+ } else {
+ fmt.Println("Transcode Meta Check Matched")
+ }
+
+ return err
+}
+
+func RunSynchronousCmd(baseCmd string, args []string) error {
+ thisDir, err := filepath.Abs(filepath.Dir(os.Args[0]))
+ if err != nil {
+ return errors.New("getting absolute filepath '" + os.Args[0] + "': " + err.Error())
+ }
+ cmd := exec.Command(baseCmd, args...)
+ cmd.Dir = thisDir
+ fmt.Printf("Transcode Cmd: %#v\n", baseCmd)
+ fmt.Printf("Transcode Args: %#v\n", args)
+ fmt.Println("Working Directory:", cmd.Dir)
+ writer := io.MultiWriter(os.Stdout)
+ cmd.Stderr = writer
+ cmd.Stdout = writer
+ if err = cmd.Run(); err != nil {
+ return fmt.Errorf("running baseCmd '%+v' args '%+v': %+v", baseCmd, args, err)
+ }
+ return nil
+}
+
+func hashFileCrc32(filePath string, polynomial uint32) (string, error) {
+ out := ""
+
+ file, err := os.Open(filePath)
+ if err != nil {
+ return out, err
+ }
+ defer file.Close()
+
+ tp := crc32.MakeTable(polynomial)
+ hash := crc32.New(tp)
+
+ if _, err := io.Copy(hash, file); err != nil {
+ return out, err
+ }
+
+ hashInBytes := hash.Sum(nil)[:]
+ out = hex.EncodeToString(hashInBytes)
+
+ return out, nil
+}
+
+func generateMeta(ep endpoint.Endpoint, cmdStr string, args []string) error {
+ // CRC-32 reversed polynomial https://en.wikipedia.org/wiki/Cyclic_redundancy_check
+ crc, err := hashFileCrc32(ep.Source, 0xEDB88320)
+ if err != nil {
+ return err
+ }
+
+ metainfo := TranscodeMeta{
+ Cmd: cmdStr,
+ Args: args,
+ CrcHash: crc,
+ LastTranscodeDT: time.Now().Format(time.RFC1123),
+ }
+
+ bytes, err := json.MarshalIndent(metainfo, "", "\t")
+ if err != nil {
+ return err
+ }
+ err = ioutil.WriteFile(ep.OutputDirectory+"/"+ep.DiskID+".meta.json", bytes, 0755)
+
+ return err
+}
+
+// GetMeta retrieves the disk metadata for the given endpoint
+func GetMeta(ep endpoint.Endpoint) (TranscodeMeta, error) {
+ raw, err := ioutil.ReadFile(ep.OutputDirectory + "/" + ep.DiskID + ".meta.json")
+ if err != nil {
+ raw = []byte("{}")
+ }
+ var sourcemetainfo TranscodeMeta
+ if err = json.Unmarshal(raw, &sourcemetainfo); err != nil {
+ return TranscodeMeta{}, err
+ }
+ return sourcemetainfo, nil
+}
+
+func checkMeta(ep endpoint.Endpoint, cmdStr string, args []string) (bool, error) {
+ // CRC-32 reversed polynomial https://en.wikipedia.org/wiki/Cyclic_redundancy_check
+ crc, err := hashFileCrc32(ep.Source, 0xEDB88320)
+ if err != nil {
+ return false, err
+ }
+
+ destmetainfo := TranscodeMeta{
+ Cmd: cmdStr,
+ Args: args,
+ CrcHash: crc,
+ }
+
+ sourcemetainfo, err := GetMeta(ep)
+ if err != nil {
+ return false, err
+ }
+
+ if sourcemetainfo.CrcHash == destmetainfo.CrcHash && sourcemetainfo.Cmd == destmetainfo.Cmd && reflect.DeepEqual(sourcemetainfo.Args, destmetainfo.Args) {
+ return true, nil
+ }
+
+ return false, nil
+}
+
+func generateMasterManifest(ep *endpoint.Endpoint) error {
+ files, err := ioutil.ReadDir(ep.OutputDirectory)
+ if err != nil {
+ return err
+ }
+ type manifest struct {
+ resolution string
+ bandwidth string
+ name string
+ }
+ var manifests []manifest
+ for _, file := range files {
+ var r *regexp.Regexp
+ r, err = regexp.Compile(ep.DiskID + `.*?(?:(?P<res>\d+x\d+)-(?P<bw>\d+))?\.m3u8`)
+ if err != nil {
+ return err
+ }
+ if r.MatchString(file.Name()) {
+ match := r.FindStringSubmatch(file.Name())
+ ep.ABRManifests = append(ep.ABRManifests, file.Name())
+ manifests = append(manifests, manifest{
+ resolution: match[1],
+ bandwidth: match[2],
+ name: file.Name(),
+ })
+ }
+ }
+ if len(manifests) == 0 {
+ return errors.New("Master manifest detection failed")
+ } else if len(manifests) == 1 {
+ return nil
+ }
+
+ out := "#EXTM3U\n#EXT-X-VERSION:3\n"
+ for _, layer := range manifests {
+ fmt.Printf("DEBUG: %+v\n", layer)
+ out = out + "#EXT-X-STREAM-INF:BANDWIDTH=" + layer.bandwidth + ",RESOLUTION=" + layer.resolution + "\n" + layer.name + "\n"
+ }
+ err = ioutil.WriteFile(ep.OutputDirectory+"/"+ep.DiskID+".m3u8", []byte(out), 0755)
+ return err
+}
diff --git a/test/fakeOrigin/version/VERSION b/test/fakeOrigin/version/VERSION
new file mode 100644
index 0000000..b41aafb
--- /dev/null
+++ b/test/fakeOrigin/version/VERSION
@@ -0,0 +1,5 @@
+1
+0
+0
+dev
+0
diff --git a/test/fakeOrigin/version/version.go b/test/fakeOrigin/version/version.go
new file mode 100644
index 0000000..47f8c33
--- /dev/null
+++ b/test/fakeOrigin/version/version.go
@@ -0,0 +1,41 @@
+package version
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// These values are irrelevant during an rpm build as they will be forced to compile time values via ldflags in build_rpm.sh
+// If you need to update the defaults, do so in VERSION
+
+// VerMajor should be be incremented whenever backward breaking changes are made
+var VerMajor = "0"
+
+// VerMinor should be updated whenever new features and bugfix rollups are performed, a typical minor release
+var VerMinor = "1"
+
+// VerPatch should be incremented if patches are backported to old releases
+var VerPatch = "0"
+
+// VerDesc should be an arbitrary string prefix to the git commit to distinguish what release pipeline stage this is from
+var VerDesc = "dev"
+
+// VerCommit should represent the git hash that was compiled with the binary
+var VerCommit = "0"
+
+// VerFull should be the full version string
+var VerFull = VerMajor + "." + VerMinor + "." + VerPatch + "_" + VerDesc + "_" + VerCommit