You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@freemarker.apache.org by sg...@apache.org on 2020/06/30 05:35:10 UTC
[freemarker-generator] branch master updated: FREEMARKER-147
Complete Maven site documentation (#16)
This is an automated email from the ASF dual-hosted git repository.
sgoeschl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/freemarker-generator.git
The following commit(s) were added to refs/heads/master by this push:
new 4c39e8e FREEMARKER-147 Complete Maven site documentation (#16)
4c39e8e is described below
commit 4c39e8ea7a000139f53105bddf3e2048ecbc2219
Author: Siegfried Goeschl <si...@gmail.com>
AuthorDate: Tue Jun 30 07:35:03 2020 +0200
FREEMARKER-147 Complete Maven site documentation (#16)
---
README.md | 12 +-
freemarker-generator-base/README.md | 3 +-
.../generator/base/datasource/DataSources.java | 17 +-
.../generator/base/template/TemplateOutput.java | 4 +-
.../generator/base/template/TemplateSource.java | 23 +-
.../base/template/TemplateTransformations.java | 3 +
.../generator/base/uri/UriFragmentParser.java | 2 +-
.../freemarker/generator/base/util/ListUtils.java | 5 +-
.../freemarker/generator/base/util/Validate.java | 2 +-
.../src/site/markdown/index.md | 4 +-
.../generator/datasource/DataSourcesTest.java | 1 -
.../template/TemplateSourceFactoryTest.java | 11 +-
.../CHANGELOG.md | 9 +-
freemarker-generator-cli/README.md | 1584 +-------------------
.../examples/templates/dataframe/example.ftl | 14 +-
.../examples/templates/info.ftl | 67 -
.../templates/json/dataframe/github-users.ftl | 20 +
freemarker-generator-cli/pom.xml | 22 +-
.../src/main/assembly/dist.xml | 41 -
.../org/apache/freemarker/generator/cli/Main.java | 4 +-
.../markdown/cli/advanced/cli-configuration.md | 77 +
.../src/site/markdown/cli/concepts/data-models.md | 37 +-
.../src/site/markdown/cli/concepts/data-sources.md | 37 +-
.../src/site/markdown/cli/concepts/design-goals.md | 17 +
.../src/site/markdown/cli/concepts/named-uris.md | 10 +-
.../{user-parameters.md => passing-data.md} | 23 +-
.../site/markdown/cli/concepts/template-loading.md | 60 +
.../src/site/markdown/cli/concepts/tools.md | 22 +
.../site/markdown/cli/concepts/transformation.md | 4 +-
.../markdown/cli/introduction/getting-started.md | 142 ++
.../site/markdown/cli/usage/running-examples.md} | 644 ++------
.../site/markdown/cli/usage/transforming-csv.md | 80 +
.../markdown/cli/usage/transforming-directories.md | 26 +-
.../dataframe.md => usage/using-dataframes.md} | 49 +-
.../src/site/markdown/index.md | 29 +-
.../site/resources/images/examples}/contract.png | Bin
.../resources/images/examples}/excel-to-html.png | Bin
.../site/resources/images/examples}/github.png | Bin
.../images/examples}/locker-test-users-pdf.png | Bin
.../resources/images/examples}/transactions.png | Bin
.../freemarker/generator/cli/ManualTest.java | 5 +-
.../{examples => }/templates/cat.ftl | 1 -
.../templates/csv/csv/transform.ftl | 7 +-
freemarker-generator-cli/templates/info.ftl | 2 +-
.../templates/lib/commons-csv.ftl | 8 +-
freemarker-generator-maven-plugin/CHANGELOG.md | 12 +
freemarker-generator-maven-plugin/pom.xml | 23 -
freemarker-generator-tools/README.md | 36 +-
freemarker-generator-tools/pom.xml | 4 +-
.../generator/tools/commonscsv/CommonsCSVTool.java | 13 +-
.../{ => impl}/CommonsCSVPrinterFacade.java | 2 +-
.../generator/tools/dataframe/DataFrameTool.java | 16 +-
.../{converter => impl}/CSVConverter.java | 2 +-
.../{converter => impl}/ConverterUtils.java | 10 +-
.../{converter => impl}/ListConverter.java | 3 +-
.../{converter => impl}/MapConverter.java | 4 +-
.../generator/tools/excel/ExcelTool.java | 8 +-
.../generator/tools/freemarker/FreeMarkerTool.java | 19 +-
.../freemarker/generator/tools/grok/GrokTool.java | 1 +
.../tools/grok/{ => impl}/GrokWrapper.java | 2 +-
.../generator/tools/system/SystemTool.java | 4 +-
.../src/site/markdown/index.md | 67 +-
.../tools/commonscsv/CommonsCSVToolTest.java | 1 +
.../generator/tools/grok/GrokToolTest.java | 1 +
pom.xml | 7 +-
src/site/markdown/index.md | 2 -
travis.sh | 22 +-
67 files changed, 1010 insertions(+), 2377 deletions(-)
diff --git a/README.md b/README.md
index efc2fc2..1db8027 100644
--- a/README.md
+++ b/README.md
@@ -3,7 +3,6 @@ Apache FreeMarker Generator
For documentation or to report bugs visit: [https://freemarker.apache.org](https://freemarker.apache.org)
-
Regarding pull requests on Github
-----------------------------------------------------------------------------
@@ -15,7 +14,6 @@ a [Contributor License Agreement](https://www.apache.org/dev/new-committers-guid
For contributions that are judged to be non-trivial, you will be asked
to actually signing a Contributor License Agreement.
-
What is Apache FreeMarker Generator?
-----------------------------------------------------------------------------
@@ -28,6 +26,16 @@ Currently it can be invoked as a
* Command-line interface `freemarker-generator-cli`
* Maven plug-in `freemarker-generator-maven-plugin`
+Building Apache FreeMarker Generator
+-----------------------------------------------------------------------------
+
+To create the artefacts locally run
+
+> mvn clean install
+
+To build the documentation site run
+
+> mvn clean site site:stage
Licensing
-----------------------------------------------------------------------------
diff --git a/freemarker-generator-base/README.md b/freemarker-generator-base/README.md
index 0a420f6..4a5c119 100644
--- a/freemarker-generator-base/README.md
+++ b/freemarker-generator-base/README.md
@@ -1,4 +1,5 @@
-# Overview
+Apache FreeMarker Generator Base
+=============================================================================
This module provides common functionality for `freemarker-generator-cli` and `freemarker-generator-maven-plugin` such as
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/datasource/DataSources.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/datasource/DataSources.java
index 4c6e156..9db5552 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/datasource/DataSources.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/datasource/DataSources.java
@@ -16,8 +16,10 @@
*/
package org.apache.freemarker.generator.base.datasource;
+import org.apache.commons.io.FilenameUtils;
import org.apache.freemarker.generator.base.util.ClosableUtils;
import org.apache.freemarker.generator.base.util.StringUtils;
+import org.apache.freemarker.generator.base.util.Validate;
import java.io.Closeable;
import java.util.ArrayList;
@@ -25,7 +27,6 @@ import java.util.Collection;
import java.util.List;
import static java.util.stream.Collectors.toList;
-import static org.apache.commons.io.FilenameUtils.wildcardMatch;
/**
* Container for data sources with a couple of convenience functions to select
@@ -36,13 +37,14 @@ public class DataSources implements Closeable {
private final List<DataSource> dataSources;
public DataSources(Collection<DataSource> dataSources) {
+ Validate.notNull(dataSources, "No data sources provided");
this.dataSources = new ArrayList<>(dataSources);
}
/**
* Get the names of all data sources.
*
- * @return datas ource names
+ * @return data source names
*/
public List<String> getNames() {
return dataSources.stream()
@@ -109,24 +111,24 @@ public class DataSources implements Closeable {
/**
* Find data sources based on their name and globbing pattern.
*
- * @param wildcard globbing pattern
+ * @param wildcard the wildcard string to match against
* @return list of matching data sources
*/
public List<DataSource> find(String wildcard) {
return dataSources.stream()
- .filter(d -> wildcardMatch(d.getName(), wildcard))
+ .filter(d -> FilenameUtils.wildcardMatch(d.getName(), wildcard))
.collect(toList());
}
/**
* Find data sources based on their group and and globbing pattern.
*
- * @param wildcard globbing pattern
+ * @param wildcard the wildcard string to match against
* @return list of mathching data sources
*/
public List<DataSource> findByGroup(String wildcard) {
return dataSources.stream()
- .filter(d -> wildcardMatch(d.getGroup(), wildcard))
+ .filter(d -> FilenameUtils.wildcardMatch(d.getGroup(), wildcard))
.collect(toList());
}
@@ -139,9 +141,6 @@ public class DataSources implements Closeable {
public String toString() {
return "DataSource{" +
"dataSources=" + dataSources +
- ", names=" + getNames() +
- ", groups=" + getGroups() +
- ", size=" + size() +
'}';
}
}
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateOutput.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateOutput.java
index 1e5cb50..426c0f1 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateOutput.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateOutput.java
@@ -26,9 +26,11 @@ import java.io.Writer;
import static java.util.Objects.requireNonNull;
/**
- * Information about where to write the ouput of a template. Initially we
+ * Information about where to write the output of a template. Initially we
* wanted to use a <code>FileWriter</code> but it requires actually an
* existing output file (otherwise a FileNotFound exception is thrown).
+ * An alternative could be a "LazyFileWriter" which creates the file on
+ * the first write access.
*/
public class TemplateOutput {
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateSource.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateSource.java
index 42c36b6..c12d35a 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateSource.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateSource.java
@@ -23,7 +23,7 @@ import java.nio.charset.StandardCharsets;
/**
* Information how to load a template. The template is either
- * provided as source code or as template path resolved
+ * provided as literal text or as template path resolved
* by FreeMarker's template loader.
*/
public class TemplateSource {
@@ -64,17 +64,38 @@ public class TemplateSource {
this.encoding = encoding;
}
+ /**
+ * Template will be loaded from path using a file-base template loader.
+ *
+ * @param path template path
+ * @return file-based template source
+ */
public static TemplateSource fromPath(String path) {
Validate.notEmpty(path, "Template path is empty");
return new TemplateSource(path, path, StandardCharsets.UTF_8);
}
+ /**
+ * Template will be loaded from path using a file-base template loader.
+ *
+ * @param path template path
+ * @param encoding character encoding og template
+ * @return file-based template source
+ */
+
public static TemplateSource fromPath(String path, Charset encoding) {
Validate.notEmpty(path, "Template path is empty");
Validate.notNull(encoding, "Template encoding is null");
return new TemplateSource(path, path, encoding);
}
+ /**
+ * Template will be loaded from a literal content.
+ *
+ * @param name name of the template
+ * @param code template code
+ * @return template source
+ */
public static TemplateSource fromCode(String name, String code) {
Validate.notEmpty(name, "Template name is empty");
Validate.notEmpty(code, "Template code is empty");
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateTransformations.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateTransformations.java
index ed65111..6c09b60 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateTransformations.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/template/TemplateTransformations.java
@@ -22,6 +22,9 @@ import java.util.List;
import static java.util.Objects.requireNonNull;
+/**
+ * Keeps track of all transformations being executed.
+ */
public class TemplateTransformations {
private final List<TemplateTransformation> templateTransformations;
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/uri/UriFragmentParser.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/uri/UriFragmentParser.java
index 839cba7..2242b90 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/uri/UriFragmentParser.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/uri/UriFragmentParser.java
@@ -24,7 +24,7 @@ import static java.util.stream.Collectors.toMap;
import static org.apache.freemarker.generator.base.util.StringUtils.isEmpty;
/**
- * Parses the URI fragment as list of name/value pairs seperated by an ampersand.
+ * Parses the URI fragment as list of name/value pairs separated by an ampersand.
*/
public class UriFragmentParser {
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/ListUtils.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/ListUtils.java
index e04d16a..369bf90 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/ListUtils.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/ListUtils.java
@@ -58,7 +58,10 @@ public class ListUtils {
* @return copied array
*/
public static <T> T coalesce(List<T> list) {
- return list.stream().filter(Objects::nonNull).findFirst().orElseGet(() -> null);
+ return list.stream()
+ .filter(Objects::nonNull)
+ .findFirst()
+ .orElse(null);
}
/**
diff --git a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/Validate.java b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/Validate.java
index af2b497..114b61d 100644
--- a/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/Validate.java
+++ b/freemarker-generator-base/src/main/java/org/apache/freemarker/generator/base/util/Validate.java
@@ -19,7 +19,7 @@ package org.apache.freemarker.generator.base.util;
import java.io.File;
/**
- * Simple validation methods designed for interal use.
+ * Simple validation methods designed for internal use.
*/
public final class Validate {
diff --git a/freemarker-generator-base/src/site/markdown/index.md b/freemarker-generator-base/src/site/markdown/index.md
index 2fd9f95..212345d 100644
--- a/freemarker-generator-base/src/site/markdown/index.md
+++ b/freemarker-generator-base/src/site/markdown/index.md
@@ -1 +1,3 @@
-TBD
\ No newline at end of file
+# Apache FreeMarker CLI Base
+
+This package contains infrastructure code not depending on `Apache FreeMarker` directly since `DataSources` and their creation might be useful for un-related command-line tools.
diff --git a/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/datasource/DataSourcesTest.java b/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/datasource/DataSourcesTest.java
index b4cf812..c97de8f 100644
--- a/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/datasource/DataSourcesTest.java
+++ b/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/datasource/DataSourcesTest.java
@@ -90,7 +90,6 @@ public class DataSourcesTest {
assertEquals("unknown", dataSources().get(0).getName());
assertEquals("pom.xml", dataSources().get(1).getName());
assertEquals("server.invalid", dataSources().get(2).getName());
- assertEquals("unknown", dataSources().getFirst().getName());
assertEquals(3, dataSources.getList().size());
assertEquals(3, dataSources.size());
assertFalse(dataSources.isEmpty());
diff --git a/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/template/TemplateSourceFactoryTest.java b/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/template/TemplateSourceFactoryTest.java
index 1d84c90..7a40e52 100644
--- a/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/template/TemplateSourceFactoryTest.java
+++ b/freemarker-generator-base/src/test/java/org/apache/freemarker/generator/template/TemplateSourceFactoryTest.java
@@ -31,9 +31,8 @@ public class TemplateSourceFactoryTest {
private static final String ANY_TEMPLATE_PATH = "any/template/path.ftl";
private static final String ANY_FILE_NAME = "pom.xml";
- private static final String ANY_URL = "https://jsonplaceholder.typicode.com/posts/2";
+ private static final String ANY_URL = "https://raw.githubusercontent.com/apache/freemarker-generator/master/freemarker-generator-cli/templates/info.ftl";
private static final String ANY_ENVIRONMENT_VARIABLE = "env:///PWD";
- private static final String ANY_NAMED_URI = "content:www=https://www.google.com?foo=bar#contenttype=application/json";
@Test
public void shouldCreateFromTemplatePath() {
@@ -66,7 +65,7 @@ public class TemplateSourceFactoryTest {
}
@Test
- @Ignore("Requires internet access")
+ // @Ignore("Requires internet access")
public void shouldCreateFromUrl() {
final TemplateSource templateSource = TemplateSourceFactory.create(ANY_URL);
@@ -77,11 +76,11 @@ public class TemplateSourceFactoryTest {
}
@Test
- @Ignore("Requires internet access")
+ // @Ignore("Requires internet access")
public void shouldCreateFromNamedUri() {
- final TemplateSource templateSource = TemplateSourceFactory.create(ANY_NAMED_URI);
+ final TemplateSource templateSource = TemplateSourceFactory.create("info=" + ANY_URL);
- assertNotNull(templateSource.getName());
+ assertEquals("info", templateSource.getName());
assertEquals(Origin.CODE, templateSource.getOrigin());
assertNull(templateSource.getPath());
assertFalse(templateSource.getCode().isEmpty());
diff --git a/CHANGELOG.md b/freemarker-generator-cli/CHANGELOG.md
similarity index 88%
rename from CHANGELOG.md
rename to freemarker-generator-cli/CHANGELOG.md
index 2d33b61..32ab747 100644
--- a/CHANGELOG.md
+++ b/freemarker-generator-cli/CHANGELOG.md
@@ -12,7 +12,6 @@ All notable changes to this project will be documented in this file. We try to a
* [FREEMARKER-135] Support user-supplied names for `DataSource` on the command line
* [FREEMARKER-129] Support `DataSource` exclude pattern in addition to include pattern
* [FREEMARKER-129] User-defined parameters are passed as `-Pkey=value` instead of using system properties
-* [FREEMARKER-129] Add `freemarker-generator-maven-plugin-sample` for better testing of the Maven plugin
* [FREEMARKER-129] Migrate `freemarker-cli` into `freemarker-generator` project (see [https://github.com/sgoeschl/freemarker-cli](https://github.com/sgoeschl/freemarker-cli))
### Changed
@@ -24,12 +23,9 @@ All notable changes to this project will be documented in this file. We try to a
* [FREEMARKER-129] Use `freemarker.configuration.setting` in `freemarker-cli.properties` to configure FreeMarker
* [FREEMARKER-129] Provide a `toString()` metheod for all tools
* [FREEMARKER-129] Use version "0.X.Y" to cater for API changes according to [Semantic Versioning](https://semver.org)
-* [FREEMARKER-129] Change artifact `freemarker-maven-plugin` to `freemarker-generator-maven-plugin`
-* [FREEMARKER-128] Update `freemarker-maven-plugin` to Apache FreeMarker 2.3.29
-
-### Deleted
### Fixed
+* [FREEMARKER-147] Complete Maven site documenation
* [FREEMARKER-127] Site build fails with missing "org/apache/maven/doxia/siterenderer/DocumentContent"
[FREEMARKER-127]: https://issues.apache.org/jira/browse/FREEMARKER-127
@@ -42,4 +38,5 @@ All notable changes to this project will be documented in this file. We try to a
[FREEMARKER-139]: https://issues.apache.org/jira/browse/FREEMARKER-139
[FREEMARKER-142]: https://issues.apache.org/jira/browse/FREEMARKER-142
[FREEMARKER-144]: https://issues.apache.org/jira/browse/FREEMARKER-144
-[FREEMARKER-146]: https://issues.apache.org/jira/browse/FREEMARKER-146
\ No newline at end of file
+[FREEMARKER-146]: https://issues.apache.org/jira/browse/FREEMARKER-146
+[FREEMARKER-147]: https://issues.apache.org/jira/browse/FREEMARKER-147
\ No newline at end of file
diff --git a/freemarker-generator-cli/README.md b/freemarker-generator-cli/README.md
index 70179f2..097a4b5 100644
--- a/freemarker-generator-cli/README.md
+++ b/freemarker-generator-cli/README.md
@@ -1,211 +1,15 @@
-# Apache FreeMarker Generator CLI
+Apache FreeMarker Generator CLI
+=============================================================================
-# 1. Is This Project For You?
+This module provides provides the CLI for `Apache FreeMarker`.
-You somehow found this GitHub project and wonder if it solves a problem you might have?!
+* Requires JDK 1.8+ on Linux, Mac OSX and Windows
+* Add the bin/freemarker-cli or bin/freemarker-cli.bat to your PATH variable
-* You need to transform some structured text document (CSV, HTML, JSON, XML, YAML, Java Property files, access logs) into CSV, HTML, Markdown or Confluence markup?
-* You need to convert an Excel document into CSV, HTML or Markdown?
-* You need to create a nice-looking PDF from some boring-looking CSV or JSON content ?
+Now you can have a look at the command line options
-The goal of `freemarker-cli` is to automate repeated transformation tasks
-
-* Which are too boring to be done manually
-* Which happen not often enough to write a dedicated script or program
-
-# 2. Once Upon A Time
-
-In December 2015 I needed a little bit of test data management for a customer project - to make a long story short (after writing a few more Groovy scripts) it boiled down to transforming one or more JSON files to something human readable.
-
-What are the options?
-
-* The cool kids say 'Node.js' - but they always say 'Node.js'
-* Some fancy Groovy scripts using Groovy's markup builder - but the syntax looks a bit odd
-* Using 'JsonPath' and 'Velocity' to reuse good & old stuff
-
-So I went with 'Apache Groovy', 'JsonPath' and 'Apache Velocity'
-
-* Playing with Groovy over the public holidays
-* Groovy has a built-in package manager which makes distribution a breeze
-* Providing samples to transform JSON to Markdown
-
-Using Velocity actually created some minor issues so I migrated to [Apache FreeMarker](https://freemarker.apache.org) during Christmas 2016
-
-* Velocity 1.7 was released 2010 and only recently there was a new release
-* I was painful to get Velocity Tools working
-* Velocity XML processing support is also painful
-* Spring 4.3 deprecated Velocity support which could affect me in the long run
-* FreeMarker has no additional dependencies and things are just working :-)
-
-While I love Apache Velocity (Apache Turbine anyone?) I decided to give FreeMarker a chance and migrated my [velocity-cli](https://github.com/sgoeschl/velocity-cli) to FreeMarker.
-
-Some years later the not-so-small-any-longer-and-not-having-tests Groovy script was still growing so I decided
-
-* To ditch Groovy and migrate to plain JDK 8
-* Write unit tests since I had no more excuses
-* To ditch Commons CLI and migrate to [Picocli](https://picocli.info)
-
-# 3. Design Goals
-
-* Create a proper command-line tool which has Unix look & feel
-* Handle arbitrary large input and output data
-* Support multiple source files/directories for a single transformation
-* Support transformation of Property files using plain-vanilla JDK
-* Support transformation of CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/)
-* Support transformation of JSON using [Jayway's JSONPath](https://github.com/jayway/JsonPath) and [GSON](https://github.com/google/gson)
-* Support transformation of Excel using [Apache POI](https://poi.apache.org)
-* Support transformation of YAML using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home)
-* Support transformation of HTML using [JSoup](https://jsoup.org)
-* Support transformation of structured logfiles using [Grok](https://github.com/thekrakken/java-grok)
-* XML & XPath is supported by FreeMarker [out-of-the-box](http://freemarker.org/docs/xgui.html)
-* Support for reading a data source content from STDIN to integrate with command line tools
-* Support execution of arbitrary commands using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/)
-* Add some commonly useful information such as `System Properties`, `Enviroment Variables`
-* Support embedding the code in existing applications
-
-# 4. Installation
-
-Download the latest release from [GitHub](https://github.com/sgoeschl/freemarker-cli/releases), e.g. [freemarker-cli-2.0.0-BETA-5-app.tar.gz](https://github.com/sgoeschl/freemarker-cli/releases/download/v2.0.0-BETA-4/freemarker-cli-2.0.0-BETA-5-app.tar.gz) and unpack it into of a directory of your choice, e.g `/Application/Java/freemarker-cli`
-
-It is recommended
-
-* To add `bin/freemarker-cli` or `bin/freemarker-cli.bat` to your executable path
-* To create a `~/.freemarker-cli` directory to store your custom FTL templates
-
-You can test the installation by executing
-
-```text
-> ./bin/freemarker-cli -t templates/info.ftl
-FreeMarker CLI Information
-------------------------------------------------------------------------------
-FreeMarker version : 2.3.30
-Template name : info.ftl
-Language : en
-Locale : en_US
-Timestamp : Jun 18, 2020 10:55:04 AM
-Output encoding : UTF-8
-Output format : plainText
-
-FreeMarker CLI Template Directories
-------------------------------------------------------------------------------
-[#1] /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler
-[#2] /Users/sgoeschl/.freemarker-cli
-
-FreeMarker CLI Tools
-------------------------------------------------------------------------------
-- CSVTool : Process CSV files using Apache Commons CSV (see https://commons.apache.org/proper/commons-csv/)
-- DataFrameTool : Bridge to nRo/DataFrame (see https://github.com/nRo/DataFrame)
-- ExcelTool : Process Excels files (XLS, XLSX) using Apache POI (see https://poi.apache.org)
-- ExecTool : Execute command line tools using Apache Commons Exec (see https://commons.apache.org/proper/commons-exec/)
-- FreeMarkerTool : Expose useful Apache FreeMarker classes
-- GrokTool : Process text files using Grok expressions (see https://github.com/thekrakken/java-grok)
-- GsonTool : Process JSON files using GSON (see https://github.com/google/gson)
-- JsonPathTool : Process JSON files using Java JSON Path (see https://github.com/json-path/JsonPath)
-- JsoupTool : Process HTML files using Jsoup (see https://jsoup.org)
-- PropertiesTool : Process JDK properties files
-- SystemTool : Expose System-related utility methods
-- UUIDTool : Create UUIDs
-- XmlTool : Process XML files using Apache FreeMarker (see https://freemarker.apache.org/docs/xgui.html)
-- YamlTool : Process YAML files using SnakeYAML(see https://bitbucket.org/asomov/snakeyaml/wiki/Home)
-
-FreeMarker CLI Data Model
----------------------------------------------------------------------------
-- CSVTool
-- DataFrameTool
-- DataSources
-- ExcelTool
-- ExecTool
-- FreeMarkerTool
-- GrokTool
-- GsonTool
-- JsonPathTool
-- JsoupTool
-- PropertiesTool
-- SystemTool
-- UUIDTool
-- XmlTool
-- YamlTool
```
-
-There a many examples (see below) available you can execute - run `./run-examples.sh` and have a look at the generated output
-
-```text
-./run-examples.sh
-templates/info.ftl
-examples/templates/demo.ftl
-templates/csv/html/transform.ftl
-examples/templates/csv/md/transform.ftl
-examples/templates/csv/shell/curl.ftl
-examples/templates/csv/md/filter.ftl
-examples/templates/csv/fo/transform.ftl
-fop -fo target/out/locker-test-users.fo target/out/locker-test-users.pdf
-examples/templates/csv/fo/transactions.ftl
-fop -fo target/out/transactions.fo target/out/transactions-fo.pdf
-templates/csv/html/transform.ftl
-wkhtmltopdf -O landscape target/out/transactions.html target/out/transactions-html.pdf
-examples/templates/dataframe/example.ftl
-examples/templates/accesslog/combined-access.ftl
-examples/templates/excel/dataframe/transform.ftl
-templates/excel/html/transform.ftl
-templates/excel/md/transform.ftl
-templates/excel/csv/transform.ftl
-examples/templates/excel/csv/custom.ftl
-examples/templates/html/csv/dependencies.ftl
-examples/templates/json/csv/swagger-endpoints.ftl
-templates/json/yaml/transform.ftl
-examples/templates/json/md/github-users.ftl
-examples/templates/properties/csv/locker-test-users.ftl
-examples/data/template
-examples/templates/yaml/txt/transform.ftl
-templates/yaml/json/transform.ftl
-examples/templates/xml/txt/recipients.ftl
-Created the following sample files in ./target/out
-total 1576
--rw-r--r-- 1 sgoeschl staff 646 Jun 18 10:56 combined-access.log.txt
--rw-r--r-- 1 sgoeschl staff 22548 Jun 18 10:56 contract.html
--rw-r--r-- 1 sgoeschl staff 7933 Jun 18 10:56 contract.md
--rw-r--r-- 1 sgoeschl staff 784 Jun 18 10:56 curl.sh
--rw-r--r-- 1 sgoeschl staff 232 Jun 18 10:56 customer.txt
--rw-r--r-- 1 sgoeschl staff 6488 Jun 18 10:56 dataframe.txt
--rw-r--r-- 1 sgoeschl staff 15632 Jun 18 10:56 demo.txt
--rw-r--r-- 1 sgoeschl staff 1310 Jun 18 10:56 dependencies.csv
--rw-r--r-- 1 sgoeschl staff 2029 Jun 18 10:56 github-users-curl.md
--rw-r--r-- 1 sgoeschl staff 2630 Jun 18 10:56 info.txt
--rw-r--r-- 1 sgoeschl staff 8075 Jun 18 10:56 interactive-dataframe.txt
--rw-r--r-- 1 sgoeschl staff 66 Jun 18 10:56 interactive-html.txt
--rw-r--r-- 1 sgoeschl staff 16 Jun 18 10:56 interactive-json.txt
--rw-r--r-- 1 sgoeschl staff 25090 Jun 18 10:56 interactive-swagger.json
--rw-r--r-- 1 sgoeschl staff 16870 Jun 18 10:56 interactive-swagger.yaml
--rw-r--r-- 1 sgoeschl staff 10 Jun 18 10:56 interactive-xml.txt
--rw-r--r-- 1 sgoeschl staff 285 Jun 18 10:56 locker-test-users.csv
--rw-r--r-- 1 sgoeschl staff 6341 Jun 18 10:56 locker-test-users.fo
--rw-r--r-- 1 sgoeschl staff 5526 Jun 18 10:56 locker-test-users.pdf
--rw-r--r-- 1 sgoeschl staff 921 Jun 18 10:56 recipients.txt
--rw-r--r-- 1 sgoeschl staff 910 Jun 18 10:56 sales-records.md
--rw-r--r-- 1 sgoeschl staff 2453 Jun 18 10:56 swagger-spec.csv
--rw-r--r-- 1 sgoeschl staff 25090 Jun 18 10:56 swagger-spec.json
--rw-r--r-- 1 sgoeschl staff 16870 Jun 18 10:56 swagger-spec.yaml
-drwxr-xr-x 4 sgoeschl staff 128 Jun 18 10:49 template
--rw-r--r-- 1 sgoeschl staff 156 Jun 18 10:56 test-multiple-sheets.xlsx.csv
--rw-r--r-- 1 sgoeschl staff 1917 Jun 18 10:56 test-multiple-sheets.xlsx.html
--rw-r--r-- 1 sgoeschl staff 389 Jun 18 10:56 test-multiple-sheets.xlsx.md
--rw-r--r-- 1 sgoeschl staff 157 Jun 18 10:56 test-transform-xls.csv
--rw-r--r-- 1 sgoeschl staff 1439 Jun 18 10:56 test.xls.dataframe.txt
--rw-r--r-- 1 sgoeschl staff 1556 Jun 18 10:56 test.xls.html
--rw-r--r-- 1 sgoeschl staff 1558 Jun 18 10:56 test.xslx.html
--rw-r--r-- 1 sgoeschl staff 25760 Jun 18 10:56 transactions-fo.pdf
--rw-r--r-- 1 sgoeschl staff 66016 Jun 18 10:56 transactions-html.pdf
--rw-r--r-- 1 sgoeschl staff 330129 Jun 18 10:56 transactions.fo
--rw-r--r-- 1 sgoeschl staff 51008 Jun 18 10:56 transactions.html
-
-```
-
-Please note that generated PDF files are very likely not found since they require `wkhtmltopdf` and `Apache FOP` installation.
-
-# 5. Usage
-
-```text
-> ./bin/freemarker-cli -h
+freemarker-cli -h
Usage: freemarker-cli (-t=<templates> [-t=<templates>]... |
-i=<interactiveTemplate>) [-hV] [--stdin] [-b=<baseDir>]
[--config=<configFile>] [--data-source-exclude=<exclude>]
@@ -245,1317 +49,85 @@ Apache FreeMarker CLI
template to process
--times=<times> re-run X times for profiling
-V, --version Print version information and exit.
-
```
-# 6. Examples
+Check the version of the `Apache FreeMarker CLI`
-The examples were tested with JDK 1.8 on Mac OS X
-
-```text
-> java -version
-java version "1.8.0_192"
-Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
-Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
```
-It is assumed that you run the examples from the `freemarker-cli` installation directory.
-
-## 6.1 Transforming GitHub JSON To Markdown
-
-A simple example with real JSON data to be transformed into Markdown
-
-### Invocation
-
-You can either use the existing JSON sample
-
-> ./bin/freemarker-cli -t examples/templates/json/md/github-users.ftl examples/data/json/github-users.json
-
-or pipe a cURL response
-
-> curl -s https://api.github.com/users | ./bin/freemarker-cli -t examples/templates/json/md/github-users.ftl --stdin
-
-### FreeMarker Template
-
-```text
-<#ftl output_format="plainText" >
-<#assign json = JsonPathTool.parse(DataSources.get(0))>
-<#assign users = json.read("$[*]")>
-<#--------------------------------------------------------------------------->
-# GitHub Users
-
-Report generated at ${.now?iso_utc}
-
-<#compress>
-<#list users as user>
-<#assign userAvatarUrl = user.avatar_url>
-<#assign userHomeUrl = user.html_url>
-# ${user.login}
-
-| User | Homepage |
-|:--------------------------------------------------------|:----------------------------------------------|
-| <img src="${user.avatar_url}" width="48" height="48"/> | [${userHomeUrl}](${userHomeUrl}) |
-</#list>
-</#compress>
+freemarker-cli -V
+version=0.1.0-SNAPSHOT, time=2020-06-25T21:48:02+0200, commit=b320d00094be8789086ad6153d9d3fcaf4b8c75f
```
-creates the following output
-
-
-
-## 6.2 CSV to HTML/Markdown Transformation
-
-Sometimes you have a CSV file which needs to be translated in Markdown or HTML - there are on-line solutions available such as [CSV To Markdown Table Generator](https://donatstudios.com/CsvToMarkdownTable) but having a local solution gives you more flexibility.
-
-```text
-> ./bin/freemarker-cli -t templates/csv/md/transform.ftl examples/data/csv/contract.csv
-> ./bin/freemarker-cli -t templates/csv/html/transform.ftl examples/data/csv/contract.csv
-```
-
-The FreeMarker template is shown below
-
-```text
-<#ftl output_format="plainText">
-<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader()>
-<#assign csvParser = CSVTool.parse(DataSources.get(0), cvsFormat)>
-<#assign csvHeaders = csvParser.getHeaderMap()?keys>
-<#assign csvRecords = csvParser.records>
-<#--------------------------------------------------------------------------->
-<#compress>
-<@writeHeaders headers=csvHeaders/>
-<@writeColums columns=csvRecords/>
-</#compress>
-<#--------------------------------------------------------------------------->
-<#macro writeHeaders headers>
-| ${csvHeaders?join(" | ", "")} |
- <#list csvHeaders as csvHeader>| --------</#list>|
-</#macro>
-<#--------------------------------------------------------------------------->
-<#macro writeColums columns>
- <#list columns as column>
- | ${column.iterator()?join(" | ", "")} |
- </#list>
-</#macro>
-```
-
-The resulting file actually looks pleasant when compared to raw CSV
-
-
-
-## 6.3 Transform XML To Plain Text
-
-Of course you can also transform a XML document
-
-```text
-> ./bin/freemarker-cli -t examples/templates/xml/txt/recipients.ftl examples/data/xml/recipients.xml
-```
-
-using the following template
-
-```text
-<#ftl output_format="plainText" >
-<#assign xml = XmlTool.parse(DataSources.get(0))>
-<#list xml.recipients.person as recipient>
-To: ${recipient.name}
-${recipient.address}
-
-Dear ${recipient.name},
-
-Thank you for your interest in our products. We will be sending you a catalog shortly.
-To take advantage of our free gift offer, please fill in the survey attached to this
-letter and return it to the address on the reverse. Only one participant is allowed for
-each household.
-
-Sincere salutations,
-
-
-D. H.
-
----------------------------------------------------------------------------------------
-</#list>
+Or run the examples
```
-
-which generates the following output
-
-```text
-To: John Smith
-3033 Long Drive, Houston, TX
-
-Dear John Smith,
-
-Thank you for your interest in our products. We will be sending you a catalog shortly.
-To take advantage of our free gift offer, please fill in the survey attached to this
-letter and return it to the address on the reverse. Only one participant is allowed for
-each household.
-
-Sincere salutations,
-
-
-D. H.
-```
-
-## 6.4 Transform JSON To CSV
-
-One day I was asked a to prepare a CSV files containing REST endpoints described by Swagger - technically this is a JSON to CSV transformation. Of course I could create that CSV manually but writing a FTL template doing that was simply more fun and saves time in the future.
-
-```text
-<#ftl output_format="plainText" strip_text="true">
-<#assign json = JsonPathTool.parse(DataSources.get(0))>
-<#assign basePath = json.read("$.basePath")>
-<#assign paths = json.read("$.paths")>
-
-<#compress>
- ENDPOINT;METHOD;CONSUMES;PRODUCES;SUMMARY;DESCRIPTION
- <#list paths as endpoint,metadata>
- <#assign relative_url = basePath + endpoint>
- <#assign methods = metadata?keys>
- <#list methods as method>
- <#assign summary = sanitize(paths[endpoint][method]["summary"]!"")>
- <#assign description = sanitize(paths[endpoint][method]["description"]!"")>
- <#assign consumes = join(paths[endpoint][method]["consumes"]![])>
- <#assign produces = join(paths[endpoint][method]["produces"]![])>
- ${relative_url};${method?upper_case};${consumes};${produces};${summary};${description}
- </#list>
- </#list>
-</#compress>
-${'\n'}
-
-<#function sanitize str>
- <#return (((str?replace(";", ","))?replace("(\\n)+", "",'r')))?truncate(250)>
-</#function>
-
-<#function join list>
- <#if list?has_content>
- <#return list?join(", ")>
- <#else>
- <#return "">
- </#if>
-</#function>
-```
-
-Invoking the FTL template
-
-> ./bin/freemarker-cli -t examples/templates/json/csv/swagger-endpoints.ftl examples/data/json/swagger-spec.json
-
-gives you
-
-```text
-ENDPOINT;METHOD;CONSUMES;PRODUCES;SUMMARY;DESCRIPTION
-/api/pets;GET;;;;Returns all pets from the system that the user has access to
-/api/pets;POST;;;;Creates a new pet in the store. Duplicates are allowed
-/api/pets/{id};GET;;;;Returns a user based on a single ID, if the user does not have access to the pet
-/api/pets/{id};DELETE;;;;Deletes a single pet based on the ID supplied
-```
-
-## 6.5 Transforming Excel Documents
-
-Another day my project management asked me to create a CSV configuration file based on an Excel documents - as usual manual copying was not an option due to required data cleanup and data transformation. So I thought about Apache POI which support XLS and XLSX documents - integration of Apache POI was a breeze but the resulting code was not particularly useful example. So a more generic transformation was provided to show the transformation of Excel documents ...
-
-```text
-> ./bin/freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test.xls
-> ./bin/freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test.xlsx
-> ./bin/freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test-multiple-sheets.xlsx
-> ./bin/freemarker-cli -t templates/excel/md/transform.ftl examples/data/excel/test-multiple-sheets.xlsx
-```
-
-The provided FTL transforms an Excel into a HTML document supporting multiple Excel sheets
-
-```text
-<#ftl output_format="HTML" >
-<#assign dataSource = DataSources.get(0)>
-<#assign name = dataSource.name>
-<#assign workbook = ExcelTool.parse(dataSource)>
-<#assign date = .now?iso_utc>
-<#--------------------------------------------------------------------------->
-<!DOCTYPE html>
-<html>
-<head>
- <title>${name}</title>
- <meta charset="UTF-8">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css">
-</head>
-<body>
-<div class="container-fluid">
- <h1>Excel Test
- <small>${name}, ${date}</small>
- </h1>
- <@writeSheets workbook/>
-</div>
-</body>
-</html>
-
-<#--------------------------------------------------------------------------->
-<#-- writeSheets -->
-<#--------------------------------------------------------------------------->
-<#macro writeSheets workbook>
- <#assign sheets = ExcelTool.getSheets(workbook)>
- <#list sheets as sheet>
- <@writeSheet sheet/>
- </#list>
-</#macro>
-
-<#--------------------------------------------------------------------------->
-<#-- writeSheet -->
-<#--------------------------------------------------------------------------->
-<#macro writeSheet sheet>
- <#assign rows = ExcelTool.toTable(sheet)>
- <h2>${sheet.getSheetName()}</h2>
- <@writeRows rows/>
-</#macro>
-
-<#--------------------------------------------------------------------------->
-<#-- writeRows -->
-<#--------------------------------------------------------------------------->
-<#macro writeRows rows>
- <table class="table table-striped">
- <#list rows as row>
- <#if row?is_first>
- <tr>
- <th>#</th>
- <#list row as column>
- <th>${column}</th>
- </#list>
- </tr>
- <#else>
- <tr>
- <td>${row?index}</td>
- <#list row as column>
- <td>${column}</td>
- </#list>
- </tr>
- </#if>
- </#list>
- </table>
-</#macro>
-```
-
-but the result looks reasonable
-
-
-
-## 6.6 Transform Property Files To CSV
-
-In this sample we transform all property files found in a directory (recursive search using include pattern) to a CSV file
-
-```text
-> ./bin/freemarker-cli --data-source-include *.properties -t examples/templates/properties/csv/locker-test-users.ftl examples/data/properties
-TENANT,SITE,USER_ID,DISPOSER_ID,PASSWORD,SMS_OTP,NAME,DESCRIPTION
-TENANT_A,fat,user_0004,user_0004,password_0004,,,
-TENANT_B,fat,user_0001,user_0001,password_0001,,,
-TENANT_B,uat,user_0003,user_0003,password_0003,,,
-TENANT_C,fat,user_0002,user_0002,password_0004,000000,,Many products
-```
-
-The FTL uses a couple of interesting features
-
-* We process a list of property files
-* The `strip_text` and `compress` strips any white-spaces and line-breaks from the output so we can create a proper CSV file
-* We use FTL functions to extract the `tenant` and `site`, e.g. `extractTenant`
-* We add a manual line break using ```${'\n'}```
-
-```text
-<#ftl output_format="plainText" strip_text="true">
-<#compress>
- TENANT,SITE,USER_ID,DISPOSER_ID,PASSWORD,SMS_OTP,NAME,DESCRIPTION
- <#list DataSources.list as dataSource>
- <#assign properties = PropertiesTool.parse(dataSource)>
- <#assign environments = properties["ENVIRONMENTS"]!"">
- <#assign tenant = extractTenant(environments)>
- <#assign site = extractSite(environments)>
- <#assign userId = properties["USER_ID"]!"">
- <#assign disposerId = properties["USER_ID"]!"">
- <#assign password = properties["PASSWORD"]!"">
- <#assign smsOtp = properties["SMS_OTP"]!"">
- <#assign name = properties["NAME"]!"">
- <#assign description = properties["DESCRIPTION"]!"">
- ${tenant},${site},${userId},${disposerId},${password},${smsOtp},${name},${description}
- </#list>
-</#compress>
-${'\n'}
-
-<#function extractSite environments>
-</#function>
-
-<#function extractTenant environments>
-</#function>
-
-```
-
-## 6.7 Transform CSV To XML-FO
-
-For a POC (proof of concept) I created a sample transformation from CSV to XML-FO in order to create a PDF document using [Apache FOP](https://xmlgraphics.apache.org/fop) using the following template file
-
-```text
-<#ftl output_format="XML" >
-<#assign dataSource = DataSources.get(0)>
-<#assign name = dataSource.name>
-<#assign cvsFormat = CSVTool.formats.DEFAULT.withDelimiter('\t').withHeader()>
-<#assign csvParser = CSVTool.parse(dataSource, cvsFormat)>
-<#assign csvHeaders = csvParser.getHeaderMap()?keys>
-<#assign csvRecords = csvParser.records>
-<#--------------------------------------------------------------------------->
-<?xml version="1.0" encoding="UTF-8"?>
-
-<fo:root xmlns:fo="http://www.w3.org/1999/XSL/Format">
- <fo:layout-master-set>
- <fo:simple-page-master master-name="first"
- page-height="21cm"
- page-width="29.7cm"
- margin-top="1cm"
- margin-bottom="1cm"
- margin-left="1cm"
- margin-right="1cm">
- <fo:region-body margin-top="1cm"/>
- <fo:region-before extent="1cm"/>
- <fo:region-after extent="-1.2cm"/>
- </fo:simple-page-master>
- </fo:layout-master-set>
- <fo:page-sequence master-reference="first">
- <fo:static-content flow-name="xsl-region-before">
- <fo:block line-height="10pt" font-size="8pt" text-align="left">Transaction Export - ${.now}</fo:block>
- </fo:static-content>
- <fo:static-content flow-name="xsl-region-after">
- <fo:block line-height="6pt" font-size="6pt" text-align="end">Page <fo:page-number/></fo:block>
- </fo:static-content>
- <fo:flow flow-name="xsl-region-body">
- <fo:table table-layout="fixed" width="100%" border-collapse="separate">
- <fo:table-column column-width="8%"/>
- <fo:table-column column-width="10%"/>
- <fo:table-column column-width="12%"/>
- <fo:table-column column-width="8%"/>
- <fo:table-column column-width="7%"/>
- <fo:table-column column-width="5%"/>
- <fo:table-column column-width="5%"/>
- <fo:table-column column-width="5%"/>
- <fo:table-column column-width="35%"/>
- <fo:table-column column-width="5%"/>
- <@writeTableHeader headers=csvHeaders/>
- <@writeTableBody columns=csvRecords/>
- </fo:table>
- </fo:flow>
- </fo:page-sequence>
-</fo:root>
-
-<#--------------------------------------------------------------------------->
-<#macro writeTableHeader headers>
- <fo:table-header>
- <fo:table-row>
- <#list headers as header>
- <fo:table-cell border-style="solid" border-width="0.1pt" padding-left="1.0px" padding-right="1.0px">
- <fo:block font-size="6pt" font-weight="bold">${header}</fo:block>
- </fo:table-cell>
- </#list>
- </fo:table-row>
- </fo:table-header>
-</#macro>
-
-<#--------------------------------------------------------------------------->
-<#macro writeTableBody columns>
- <fo:table-body>
- <#list columns as column>
- <fo:table-row>
- <#list column.iterator() as field>
- <fo:table-cell border-style="solid" border-width="0.1pt" padding-left="1.0px" padding-right="1.0px">
- <fo:block font-size="6pt">${field}</fo:block>
- </fo:table-cell>
- </#list>
- </fo:table-row>
- </#list>
- </fo:table-body>
-</#macro>
-
-```
-
-In order to create the PDF you need to execute the following commands (assuming that you have Apache FOP installed)
-
-```text
-> ./bin/freemarker-cli -t examples/templates/csv/fo/transform.ftl examples/data/csv/locker-test-users.csv > sample.fo
-> fop -fo sample.fo sample.pdf
-Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent
-WARNING: Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400".
-Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent
-WARNING: Font "ZapfDingbats,normal,700" not found. Substituting with "ZapfDingbats,normal,400".
-Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent
-INFO: Rendered page #1.
-```
-
-The result does not look very impressive but it is a PDF :-)
-
-
-
-Further along the line of the POC we converted a transaction export from CSV to PDF using Apache FOP
-
-```text
-> ./bin/freemarker-cli -t examples/templates/csv/fo/transactions.ftl examples/data/csv/transactions.csv > transactions.fo
-> fop -fo transactions.fo transactions.pdf
-Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent
-WARNING: Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400".
-Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent
-WARNING: Font "ZapfDingbats,normal,700" not found. Substituting with "ZapfDingbats,normal,400".
-Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent
-WARNING: The contents of fo:block line 1 exceed the available area in the inline-progression direction by 11027 millipoints. (See position 1519:51)
-Jan 16, 2019 11:15:22 PM org.apache.fop.events.LoggingEventListener processEvent
-INFO: Rendered page #1.
-Jan 16, 2019 11:15:22 PM org.apache.fop.events.LoggingEventListener processEvent
-INFO: Rendered page #2.
-```
-
-
-
-## 6.8 Transforming HTML To CSV
-
-Recently I got the rather unusual question how to determine the list of dependencies of an application - one easy way is the Maven "dependencies.html" but this is unstructured data. Having said that the Jsoup library is perfectly able to parse most real-life HTML and provides a DOM model
-
-```text
-<#ftl output_format="plainText" strip_text="true">
-<#assign dataSource = DataSources.get(0)>
-<#assign html = JsoupTool.parse(dataSource)>
-
-<#compress>
- <@writeHeader/>
- <@writeDependencies "Project_Dependencies_compile"/>
- <@writeDependencies "Project_Transitive_Dependencies_compile"/>
- <@writeDependencies "Project_Transitive_Dependencies_runtime"/>
- <@writeDependencies "Project_Transitive_Dependencies_provided"/>
-</#compress>
-
-<#macro writeHeader>
- GroupId,ArtifactId,Version,Type,Licenses
-</#macro>
-
-<#macro writeDependencies section>
- <#assign selection = html.select("a[name=${section}]")>
- <#if selection?has_content>
- <#assign table = selection[0].nextElementSibling().child(2).child(0)>
- <#assign rows = table.children()>
- <#list rows as row>
- <#if !row?is_first>
- <#assign groupId = row.child(0).text()>
- <#assign artificatId = row.child(1).text()>
- <#assign version = row.child(2).text()>
- <#assign type = row.child(3).text()>
- <#assign licences = row.child(4).text()?replace(",", "")>
- ${groupId},${artificatId},${version},${type},${licences}
- </#if>
- </#list>
- </#if>
-</#macro>
-
-
-```
-
-Your dependencies as CSV can be generated as shown below
-
-```text
-> ./bin/freemarker-cli -t examples/templates/html/csv/dependencies.ftl examples/data/html/dependencies.html
-GroupId,ArtifactId,Version,Type,Licenses
-com.jayway.jsonpath,json-path,2.4.0,jar,The Apache Software License Version 2.0
-commons-cli,commons-cli,1.4,jar,Apache License Version 2.0
-org.apache.commons,commons-csv,1.5,jar,Apache License Version 2.0
-org.apache.poi,poi,4.0.1,jar,The Apache Software License Version 2.0
-org.apache.poi,poi-ooxml,3.17,jar,The Apache Software License Version 2.0
-org.apache.poi,poi-ooxml-schemas,3.17,jar,The Apache Software License Version 2.0
-org.freemarker,freemarker,2.3.28,jar,Apache License Version 2.0
-org.jsoup,jsoup,1.11.3,jar,The MIT License
-org.slf4j,slf4j-api,1.7.21,jar,MIT License
-org.slf4j,slf4j-log4j12,1.7.21,jar,MIT License
-com.github.virtuald,curvesapi,1.04,jar,BSD License
-commons-codec,commons-codec,1.11,jar,Apache License Version 2.0
-log4j,log4j,1.2.17,jar,The Apache Software License Version 2.0
-net.minidev,accessors-smart,1.2,jar,The Apache Software License Version 2.0
-net.minidev,json-smart,2.3,jar,The Apache Software License Version 2.0
-org.apache.commons,commons-collections4,4.2,jar,Apache License Version 2.0
-org.apache.commons,commons-math3,3.6.1,jar,Apache License Version 2.0
-org.apache.xmlbeans,xmlbeans,2.6.0,jar,The Apache Software License Version 2.0
-org.ow2.asm,asm,5.0.4,jar,BSD
-stax,stax-api,1.0.1,jar,The Apache Software License Version 2.0
-```
-
-## 6.9 Transform CSV To Shell Script
-
-For a customer project we wanted to record REST request / responses using WireMock - really quick and dirty. So we decided to avoid any sophisticated test tool but generate a ready-to-use shell script executing cURL commands. It turned out that handling of dollar signs is a bit tricky
-
-* Using ```noparse``` directive to disable parsing of dollar signs
-* Using ```${r"${MY_BASE_URL}"``` to generate output with dollar signs
-
-and the final FTL is found below
-
-```text
-<#ftl output_format="plainText">
-<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader()>
-<#assign csvParser = CSVTool.parse(DataSources.get(0), cvsFormat)>
-<#assign records = csvParser.records>
-<#assign csvMap = CSVTool.toMap(records, "disposer")>
-<#--------------------------------------------------------------------------->
-#!/bin/sh
-
-<#noparse>
-MY_BASE_URL=${MY_BASE_URL:=https://postman-echo.com}
-</#noparse>
-
-echo "time,user,status,duration,size"
-<#list records as record>
-date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',${record.disposer},%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${r"${MY_BASE_URL}"}/get"
-</#list>
-```
-
-Rendering the FreeMarker template
-
-```
-> ./bin/freemarker-cli -t ./examples/templates/csv/shell/curl.ftl examples/data/csv/user.csv
-```
-
-generates the following shell script
-
-```
-#!/bin/sh
-
-MY_BASE_URL=${MY_BASE_URL:=https://postman-echo.com}
-
-echo "time,user,status,duration,size"
-date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',AAAAAAA,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
-date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',BBBBBBB,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
-date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',CCCCCCC,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
-date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',DDDDDDD,%{http_code},%{time_total},%{size_download}\n' --silent --show-error --output /dev/null "${MY_BASE_URL}/get"
-```
-
-Looks a bit complicated but lets dissect the things
-
-* `date "+%FT%H:%M:%S" | tr -d '\n'` creates a timestamp and removes the line feed
-* `curl --write-out` allows to print runtime data (see [https://ec.haxx.se/usingcurl-writeout.html](https://ec.haxx.se/usingcurl-writeout.html))
-
-Executing the result shell script creates the following output (which is a nice CSV for further processing)
-
-```
-time,user,status,duration,size
-2019-09-27T21:02:52,AAAAAAA,200,0.522473,206
-2019-09-27T21:02:53,BBBBBBB,200,0.498093,206
-2019-09-27T21:02:54,CCCCCCC,200,0.529013,206
-2019-09-27T21:02:54,DDDDDDD,200,0.528268,206
-```
-
-## 6.10 Unleashing The Power Of Grok
-
-Think of `Grok` as modular regular expressions with a pre-defined functionality to parse access logs or any other data where you can't comprehend the regular expression any longer, one very simple example is `QUOTEDSTRING`
-
-```
-QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
-```
-
-And with `Grok` the `QUOTEDSTRING` is just a building block for an even more complex regular expession such as `COMBINEDAPACHELOG`
-
-> bin/freemarker-cli -t examples/templates/accesslog/combined-access.ftl examples/data/accesslog/combined-access.log
-
-which gives you the following output
-
-```
-TIMESTAMP;VERB;REQUEST;HTTPVERSION
-19/Jun/2005:06:44:17 +0200;GET;/wximages/wxwidgets02-small.png;1.1
-19/Jun/2005:06:46:05 +0200;GET;/wximages/wxwidgets02-small.png;1.1
-19/Jun/2005:06:47:37 +0200;GET;/wximages/wxwidgets02-small.png;1.1
-19/Jun/2005:06:48:40 +0200;GET;/wiki.pl?WxWidgets_Bounties;1.1
-19/Jun/2005:06:50:49 +0200;GET;/wiki.pl?WxWidgets_Compared_To_Other_Toolkits;1.1
-19/Jun/2005:06:50:49 +0200;GET;/wxwiki.css;1.1
-19/Jun/2005:06:50:49 +0200;GET;/wximages/wxwidgets02-small.png;1.1
-19/Jun/2005:06:50:50 +0200;GET;/favicon.ico;1.1
-19/Jun/2005:06:52:36 +0200;GET;/wximages/wxwidgets02-small.png;1.1
-19/Jun/2005:06:53:14 +0200;GET;/;1.0
-```
-
-using the following FreeMarker template
-
-```text
-<#ftl output_format="plainText" strip_whitespace=true>
-<#assign grok = GrokTool.compile("%{COMBINEDAPACHELOG}")>
-<#assign dataSource = DataSources.get(0)>
-<#assign lines = dataSource.getLineIterator()>
-
-<#compress>
- TIMESTAMP;VERB;REQUEST;HTTPVERSION
- <#list lines as line>
- <#assign parts = grok.match(line)>
- <#assign timestamp = parts["timestamp"]>
- <#assign verb = parts["verb"]>
- <#assign request = parts["request"]>
- <#assign httpversion = parts["httpversion"]>
- ${timestamp};${verb};${request};${httpversion}
- </#list>
-</#compress>
-```
-
-While this looks small and tidy there are some nifty features
-
-* `GrokTool.compile("%{COMBINEDAPACHELOG}")` builds the `Grok` instance to parse access logs in `Combined Format`
-* The data source is streamed line by line and not loaded into memory in one piece
-* This also works for using `stdin` so are able to parse GB of access log or other files
-
-## 6.11 CSV Transformation
-
-Sometimes you have a CSV file which is not quite right - you need to change the format. Lets have a look how `freemarker-cli` can help
-
-> bin/freemarker-cli -PCVS_IN_DELIMITER=COMMA -PCSV_TARGET_DELIMITER=PIPE -t templates/csv/csv/transform.ftl ./examples/data/csv/contract.csv
-
-renders the following template
-
-```text
-<#ftl output_format="plainText" strip_text="true">
-<#assign csvParser = CSVTool.parse(DataSources.get(0))>
-<#assign csvPrinter = CSVTool.printer(SystemTool.writer)>
-<#--
- Print each record directly to the underyling writer without materializing the CSV in memory.
- FreeMarker and CSV output are out of sync but millions of records can processed without
- running out of memory.
--->
-<#compress>
- <#list csvParser.iterator() as record>
- ${csvPrinter.printRecord(record)}
- </#list>
-</#compress>
-```
-
-and generates the following output
-
-```text
-contract_id|seller_company_name|customer_company_name|customer_duns_number|contract_affiliate|FERC_tariff_reference|contract_service_agreement_id|contract_execution_date|contract_commencement_date|contract_termination_date|actual_termination_date|extension_provision_description|class_name|term_name|increment_name|increment_peaking_name|product_type_name|product_name|quantity|units_for_contract|rate|rate_minimum|rate_maximum|rate_description|units_for_rate|point_of_receipt_control_area|po [...]
-C71|The Electric Company|The Power Company|456543333|N|FERC Electric Tariff Original Volume No. 10|2|2/15/2001|2/15/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||||||ES
-C72|The Electric Company|Utility A|38495837|n|FERC Electric Tariff Original Volume No. 10|15|7/25/2001|8/1/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||||||ES
-C73|The Electric Company|Utility B|493758794|N|FERC Electric Tariff Original Volume No. 10|7|6/8/2001|7/6/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||" "|" "|||ep
-C74|The Electric Company|Utility C|594739573|n|FERC Electric Tariff Original Volume No. 10|25|6/8/2001|7/6/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||" "|" "|||ep
-```
-
-Some useful hints
-
-* For available CSV formats please see [Apache Commons CSV User Guide](http://commons.apache.org/proper/commons-csv/user-guide.html)
-* Stripping the Excel BOM (Byte Order Mark) works out-of-box
-
-## 6.12 Executing Arbitrary Commands
-
-Using Apache Commons Exec allows to execute arbitrary commands - nice but dangerous. It was recently quite useful to to invoke AWS CLI to generate a Confluence page about the overall setup of our AWS accounts.
-
-A few snippets to illustrate the points
-
-```text
-<#ftl output_format="plainText" strip_whitespace="true">
-<#assign profile = SystemTool.getProperty("profile", "default")>
-<#assign ec2Instances = ec2Instances()/>
-
-h3. AWS EC2 Instance
-<@printEc2Instances ec2Instances/>
-
-<#function ec2Instances>
- <#local json = awsCliToJson("aws ec2 describe-instances --profile ${profile}")>
- <#local instances = json.read("$.Reservations[*].Instances[*]")>
- <#return instances?sort_by(['InstanceType'])>
-</#function>
-
-<#function awsCliToJson line>
- <#local output = ExecTool.execute(line)>
- <#return JsonPathTool.parse(output)>
-</#function>
-
-<#function getAwsEc2InstanceTag tags name>
- <#return tags?filter(x -> x["Key"] == name)?first["Value"]!"">
-</#function>
-
-<#macro printEc2Instances ec2Instances>
- <#compress>
- || NAME || INSTANCE_TYPE || VCPUS || STATE || PRIVATE_IP_ADDRESS ||
- <#list ec2Instances as ec2Instance>
- <#assign instanceType = ec2Instance["InstanceType"]>
- <#assign arn = ec2Instance["IamInstanceProfile"]["Arn"]>
- <#assign privateIpAddress = ec2Instance["PrivateIpAddress"]>
- <#assign state = ec2Instance["State"]["Name"]>
- <#assign launchTime = ec2Instance["LaunchTime"]>
-
- <#assign coreCount = ec2Instance["CpuOptions"]["CoreCount"]?number>
- <#assign threadsPerCore = ec2Instance["CpuOptions"]["ThreadsPerCore"]?number>
- <#assign nrOfVirtualCpus = coreCount * threadsPerCore>
-
- <#assign tags = ec2Instance["Tags"]/>
- <#assign awsCloudFormationStackId = getAwsEc2InstanceTag(tags, "aws:cloudformation:stack-id")>
- <#assign awsCloudFormationStackName = getAwsEc2InstanceTag(tags, "aws:cloudformation:stack-name")>
- <#assign name = getAwsEc2InstanceTag(tags, "Name")>
- <#assign country = getAwsEc2InstanceTag(tags, "Country")>
- <#assign environment = getAwsEc2InstanceTag(tags, "Environment")>
-
- | ${name} | ${instanceType} | ${nrOfVirtualCpus} | ${state} | ${privateIpAddress} |
- </#list>
- </#compress>
-</#macro>
-```
-
-## 6.13 Interactive Templates
-
-Sometime you need to apply a CSS, JSON or XPath query in ad ad-hoc way without installing `xmllint`, `jq` or `pup` - in this case you can pass a FreeMarker template in an interactive fashion
-
-```text
-> bin/freemarker-cli -i 'Hello ${SystemTool.envs["USER"]}'; echo
-Hello sgoeschl
-
-> bin/freemarker-cli -i '${JsonPathTool.parse(DataSources.first).read("$.info.title")}' examples/data/json/swagger-spec.json; echo
-Swagger Petstore
-
-> bin/freemarker-cli -i 'Post Title : ${JsonPathTool.parse(DataSources.first).read("$.title")}' https://jsonplaceholder.typicode.com/posts/2; echo
-Post Title : qui est esse
-
-> bin/freemarker-cli -i '${XmlTool.parse(DataSources.first)["recipients/person[1]/name"]}' examples/data/xml/recipients.xml; echo
-John Smith
-
-> bin/freemarker-cli -i '${JsoupTool.parse(DataSources.first).select("a")[0]}' examples/data/html/dependencies.html; echo
-<a href="${project.url}" title="FreeMarker CLI">FreeMarker CLI</a>
-
-> ./bin/freemarker-cli -i '<#list SystemTool.envs as name,value>${name} ==> ${value}${"\n"}</#list>'
-TERM ==> xterm-256color
-LANG ==> en_US
-DISPLAY ==> :0.0
-SHELL ==> /bin/bash
-EDITOR ==> vi
-```
-
-## 6.14 Filtering & Transforming CSV
-
-During an integration project we imported large transactions CSV files (500.000+ records) and in case of import failures the developers would be happy to get a nice outline of the transactions causing the problem (the CSV records have 60+ columns) - in essence it is filtering (based on some primary key) and and transforming into a human-readable output format (Markdown).
-
-So lets start the filtering & transformation using the following command line
-
-```text
-> bin/freemarker-cli -e UTF-8 -l de_AT -Pcolumn="Order ID" \
- -Pvalues=226939189,957081544 \
- -t examples/templates/csv/md/filter.ftl examples/data/csv/sales-records.csv
-```
-
-and Apache FreeMarker template
-
-```text
-<#ftl output_format="plainText" strip_text="true">
-<#assign dataSource = DataSources.get(0)>
-<#assign parser = parser(dataSource)>
-<#assign headers = parser.getHeaderNames()>
-<#assign column = SystemTool.getParameter("column")>
-<#assign values = SystemTool.getParameter("values")?split(",")>
-
-<#compress>
- <@writePageHeader dataSource/>
- <#-- Process each line without materializing the whole file in memory -->
- <#list parser.iterator() as record>
- <#if filter(record)>
- <@writeCsvRecord headers record/>
- </#if>
- </#list>
-</#compress>
-
-<#function parser dataSource>
- <#assign format = CSVTool.formats[SystemTool.getParameter("format", "DEFAULT")]>
- <#assign delimiter = CSVTool.toDelimiter(SystemTool.getParameter("delimiter", format.getDelimiter()))>
- <#return CSVTool.parse(dataSource, format.withFirstRecordAsHeader().withDelimiter(delimiter))>
-</#function>
-
-<#function filter record>
- <#return values?seq_contains(record.get(column))>
-</#function>
-
-<#macro writePageHeader dataSource>
- # ${dataSource.name}
-</#macro>
-
-<#macro writeCsvRecord headers record>
- ## Line ${record.getRecordNumber()}
- | Column | Value |
- | --------- | --------------------------- |
- <#list headers as header>
- | ${header} | ${record.get(header)} |
- </#list>
-</#macro>
-```
-
-yields
-
-```
-# sales-records.csv
-## Line 1
-| Column | Value |
-| --------- | --------------------------- |
-| Region | Central America and the Caribbean |
-| Country | Antigua and Barbuda |
-| Item Type | Baby Food |
-| Sales Channel | Online |
-| Order Priority | M |
-| Order Date | 12/20/2013 |
-| Order ID | 957081544 |
-| Ship Date | 1/11/2014 |
-| Units Sold | 552 |
-| Unit Price | 255.28 |
-| Unit Cost | 159.42 |
-| Total Revenue | 140914.56 |
-| Total Cost | 87999.84 |
-| Total Profit | 52914.72 |
-## Line 4.998
-| Column | Value |
-| --------- | --------------------------- |
-| Region | Asia |
-| Country | Myanmar |
-| Item Type | Baby Food |
-| Sales Channel | Offline |
-| Order Priority | H |
-| Order Date | 11/23/2016 |
-| Order ID | 226939189 |
-| Ship Date | 12/10/2016 |
-| Units Sold | 5204 |
-| Unit Price | 255.28 |
-| Unit Cost | 159.42 |
-| Total Revenue | 1328477.12 |
-| Total Cost | 829621.68 |
-| Total Profit | 498855.44 |
-```
-
-## 6.15 Converting Between JSON And YAML
-
-Sometimes we simply need to transform a JSON into an equivalent YAML or the other way around
-
-```
-> ./bin/freemarker-cli -t templates/yaml/json/transform.ftl examples/data/yaml/swagger-spec.yaml
-> ./bin/freemarker-cli -i '${GsonTool.toJson(YamlTool.parse(DataSources.get(0)))}' examples/data/yaml/swagger-spec.yaml
-> ./bin/freemarker-cli -i '${GsonTool.toJson(yaml)}' -m yaml=examples/data/yaml/swagger-spec.yaml
-
-> ./bin/freemarker-cli -t templates/json/yaml/transform.ftl examples/data/json/swagger-spec.json
-> ./bin/freemarker-cli -i '${YamlTool.toYaml(GsonTool.parse(DataSources.get(0)))}' examples/data/json/swagger-spec.json
-> ./bin/freemarker-cli -i '${YamlTool.toYaml(json)}' -m json=examples/data/json/swagger-spec.json
-```
-
-## 6.16 Using DataFrames
-
-The `DataFrameTool` uses [nRo/DataFrame](https://github.com/nRo/DataFrame) to convert tabular data into a `DataFrame`.
-
-A `DataFrame` allows declartive filtering and transformation of tabular data, i.e. less code to write.
-
-Currently the following sources are supported
-
-* Apache Commons CSV Parser
-* JSON arrays represented as collection of maps
-* Excel sheets represented as rows
-
-### CSV Examples
-
-[nRo/DataFrame]("https://raw.githubusercontent.com/nRo/DataFrame/master/src/test/resources/users.csv") provides the following CSV file
-
-```
-name;age;country
-Schmitt;24;Germany
-Parker;45;USA
-Meier;20;Germany
-Schmitt;30;France
-Peter;44;Germany
-Meier;24;Germany
-Green;33;UK
-Schmitt;30;Germany
-Meier;30;Germany
-```
-
-and create a `DateFrame` using the following code
-
-```
-<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader().withDelimiter(';')>
-<#assign csvParser = CSVTool.parse(DataSources.get(0), cvsFormat)>
-<#assign users = DataFrameTool.toDataFrame(csvParser)>
-```
-
-#### Select By Age
-
-```
-${DataFrameTool.print(users.select("(age > 40)"))}
-```
-
-which shows
-
-```
-┌────────────┬────────────┬────────────┐
-│#name │#age │#country │
-├────────────┼────────────┼────────────┤
-│Parker │45 │USA │
-├────────────┼────────────┼────────────┤
-│Peter │44 │Germany │
-└────────────┴────────────┴────────────┘
-```
-
-#### Complex Select & Sort
-
-Now we want to create a new `DataFrame` by selecting `name` and `country`
-
-```
-<#assign country = "Germany">
-${DataFrameTool.print(users
- .select("(name == 'Schmitt' || name == 'Meier') && country == '${country}'")
- .sort("name", DataFrameTool.sortOrder["ASCENDING"]))}
-```
-
-which shows
-
-```
-┌────────────┬────────────┬────────────┐
-│#name │#age │#country │
-├────────────┼────────────┼────────────┤
-│Meier │20 │Germany │
-├────────────┼────────────┼────────────┤
-│Meier │24 │Germany │
-├────────────┼────────────┼────────────┤
-│Meier │30 │Germany │
-├────────────┼────────────┼────────────┤
-│Schmitt │24 │Germany │
-├────────────┼────────────┼────────────┤
-│Schmitt │30 │Germany │
-└────────────┴────────────┴────────────┘
-```
-
-#### Count Column Values
-
-Let's assume we want to count the records for each `country`
-
-```
-${DataFrameTool.print(users.getColumn("country").transform(DataFrameTool.transformer["COUNT"]))}
-```
-
-returns the following `DataFrame`
-
-```
-┌────────────┬────────────┐
-│#country │#counts │
-├────────────┼────────────┤
-│Germany │6 │
-├────────────┼────────────┤
-│USA │1 │
-├────────────┼────────────┤
-│France │1 │
-├────────────┼────────────┤
-│UK │1 │
-└────────────┴────────────┘
-```
-
-#### Group By Age And Country
-
-Let's assume that we want to group the `DataFrame` by `age` and `country`
-
-```
-${DataFrameTool.print(users.groupBy("age", "country").sort("age"))}
-```
-
-which results in
-
-```
-┌────────────┬────────────┐
-│#age │#country │
-├────────────┼────────────┤
-│20 │Germany │
-├────────────┼────────────┤
-│24 │Germany │
-├────────────┼────────────┤
-│30 │France │
-├────────────┼────────────┤
-│30 │Germany │
-├────────────┼────────────┤
-│33 │UK │
-├────────────┼────────────┤
-│44 │Germany │
-├────────────┼────────────┤
-│45 │USA │
-└────────────┴────────────┘
-```
-
-### JSON Examples
-
-Here we load a `examples/data/json/github-users.json` which represents a tabular data be
-being parsed as a list of maps and print the JSOB as dataframe
-
-```
-./bin/freemarker-cli -i '${DataFrameTool.print(DataFrameTool.fromMaps(GsonTool.parse(DataSources.get(0))))}' examples/data/json/github-users.json
-
-┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
-│#login │#id │#avatar_ur │#gravatar_ │#url │#html_url │#followers │#following │#gists_url │#starred_u │#subscript │#organizat │#repos_url │#events_ur │#received_ │#type │#site_admi │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│mojombo │1.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│defunkt │2.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │true │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│pjhyett │3.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │true │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│wycats │4.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│ezmobius │5.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│ivey │6.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│evanphx │7.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
-```
-
-### Excel Examples
-
-Let's transform an Excel Sheet to a `DataFrame` being printed using the following template
-
-```
-<#assign dataSource = DataSources.get(0)>
-<#assign workbook = ExcelTool.parse(dataSource)>
-<#list ExcelTool.getSheets(workbook) as sheet>
- <#assign table = ExcelTool.toTable(sheet)>
- <#assign df = DataFrameTool.fromRows(table, true)>
- ${DataFrameTool.print(df)}<#t>
-</#list>
-```
-
-which is rendered by the following command line invocation
-
-```
-./bin/freemarker-cli -t examples/templates/excel/dataframe/transform.ftl examples/data/excel/test.xls
-
-┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
-│#Text │#Date │#Number │#Currency │#Time │#Percentag │#Forumula │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│Row 1 │01/01/17 │100.00 │€100.00 │10:00 │50.00% │C2*F2 │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│Row 2 │01/01/17 │100.00 │€100.00 │10:00 │50.00% │C3*F3 │
-└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
-```
-
-## 6.17 Using Advanced FreeMarker Features
-
-There is a `demo.ftl` which shows some advanced FreeMarker functionality
-
-* Invoking a Java constructor
-* Invoke a static method of non-instantiable class
-* Work with Java enumerations
-* Access System properties
-* Access Environment variables
-
-Running
-
-> ./bin/freemarker-cli -t examples/templates/demo.ftl
-
-gives you
-
-```text
-1) FreeMarker Special Variables
----------------------------------------------------------------------------
-FreeMarker version : 2.3.30
-Template name : demo.ftl
-Language : en
-Locale : en_US
-Timestamp : Jun 18, 2020 11:13:51 AM
-Output encoding : UTF-8
-Output format : plainText
-
-2) Invoke a constructor of a Java class
----------------------------------------------------------------------------
-new java.utilDate(1000 * 3600 * 24): Jan 2, 1970 1:00:00 AM
-
-3) Invoke a static method of an non-constructor class
----------------------------------------------------------------------------
-Random UUID : ac6c8bb2-a8cf-423e-ad6b-a4eff09b1fb9
-System.currentTimeMillis : 1,592,471,631,655
-
-4) Access an Enumeration
----------------------------------------------------------------------------
-java.math.RoundingMode#UP: UP
-
-5) Loop Over The Values Of An Enumeration
----------------------------------------------------------------------------
-- java.math.RoundingMode.UP
-- java.math.RoundingMode.DOWN
-- java.math.RoundingMode.CEILING
-- java.math.RoundingMode.FLOOR
-- java.math.RoundingMode.HALF_UP
-- java.math.RoundingMode.HALF_DOWN
-- java.math.RoundingMode.HALF_EVEN
-- java.math.RoundingMode.UNNECESSARY
-
-6) Display list of data sources
----------------------------------------------------------------------------
-List all data sources:
-
-7) SystemTool
----------------------------------------------------------------------------
-Host name : W0GL5179.home
-Command line : -t, examples/templates/demo.ftl
-System property : sgoeschl
-Timestamp : 1592471631667
-Environment var : sgoeschl
-
-8) Access System Properties
----------------------------------------------------------------------------
-app.dir :
-app.home : /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler
-app.pid : 13037
-basedir : /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler
-java.version : 1.8.0_192
-user.name : sgoeschl
-user.dir : /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler
-user.home : /Users/sgoeschl
-
-9) List Environment Variables
----------------------------------------------------------------------------
-- PATH ==> /Users/sgoeschl/bin:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin:/usr/local/opt/ruby/bin:HOME/.gem/ruby/2.7.0/bin:/usr/local/Cellar/git/2.19.1/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Applications/Java/apache-fop-2.3:/Applications/Java/freemarker-cli-2.0.0-BETA-2/bin:
-- GIT_HOME ==> /usr/local/Cellar/git/2.19.1
-- JAVA_8_HOME ==> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home
-- JAVA_HOME ==> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home
-- FOP_HOME ==> /Applications/Java/apache-fop-2.3
-- TERM ==> xterm-256color
-- LANG ==> en_US
-- MAVEN_OPTS ==> -Xmx2048m
-- DISPLAY ==> :0.0
-- JAVA_11_HOME ==> /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
-- BEEONE_NEXUS_CREDENTIALS ==> H50N0OB:fRidnevo0420!
-- JAVA_9_HOME ==> /Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home
-- LOGNAME ==> sgoeschl
-- XPC_SERVICE_NAME ==> 0
-- PWD ==> /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler
-- TERM_PROGRAM_VERSION ==> 433
-- RUBY_HOME ==> /usr/local/opt/ruby
-- JAVA_MAIN_CLASS_13037 ==> org.apache.freemarker.generator.cli.Main
-- SHELL ==> /bin/bash
-- PROFILE_TYPE ==> development
-- TERM_PROGRAM ==> Apple_Terminal
-- LSCOLORS ==> ExFxCxDxBxegedabagacad
-- PROFILE_ENV ==> default
-- SECURITYSESSIONID ==> 186a8
-- USER ==> sgoeschl
-- CLICOLOR ==> 1
-- GATLING_HOME ==> /Applications/Java/gatling-3.1.2
-- LaunchInstanceID ==> E8DE55F6-626E-4D15-B3B4-FFE4704CEF4E
-- TMPDIR ==> /var/folders/cd/jbgc9cg14ld7dlsqk44tpmrw0000gn/T/
-- SSH_AUTH_SOCK ==> /private/tmp/com.apple.launchd.DTdlBj20Ka/Listeners
-- EDITOR ==> vi
-- XPC_FLAGS ==> 0x0
-- FREEMARKER_CLI_HOME ==> /Applications/Java/freemarker-cli-2.0.0-BETA-2
-- TERM_SESSION_ID ==> 2745D6A3-543A-4DE3-830B-945678460311
-- LC_ALL ==> en_US.utf-8
-- __CF_USER_TEXT_ENCODING ==> 0x1F5:0x0:0x0
-- LC_CTYPE ==> UTF-8
-- HOME ==> /Users/sgoeschl
-- SHLVL ==> 1
-
-10) List System Properties
----------------------------------------------------------------------------
-- java.runtime.name ==> Java(TM) SE Runtime Environment
-- java.vm.version ==> 25.192-b12
-- java.vm.vendor ==> Oracle Corporation
-- java.vendor.url ==> http://java.oracle.com/
-- path.separator ==> :
-- java.vm.name ==> Java HotSpot(TM) 64-Bit Server VM
-- file.encoding.pkg ==> sun.io
-- user.country ==> AT
-
-11) Access DataSources
----------------------------------------------------------------------------
-Get the number of documents:
-- 0
-List all files containing "README" in the name
-List all files having "md" extension
-Get all documents
-
-12) FreeMarker CLI Tools
----------------------------------------------------------------------------
-- CSVTool : Process CSV files using Apache Commons CSV (see https://commons.apache.org/proper/commons-csv/)
-- DataFrameTool : Bridge to nRo/DataFrame (see https://github.com/nRo/DataFrame)
-- ExcelTool : Process Excels files (XLS, XLSX) using Apache POI (see https://poi.apache.org)
-- ExecTool : Execute command line tools using Apache Commons Exec (see https://commons.apache.org/proper/commons-exec/)
-- FreeMarkerTool : Expose useful Apache FreeMarker classes
-- GrokTool : Process text files using Grok expressions (see https://github.com/thekrakken/java-grok)
-- GsonTool : Process JSON files using GSON (see https://github.com/google/gson)
-- JsonPathTool : Process JSON files using Java JSON Path (see https://github.com/json-path/JsonPath)
-- JsoupTool : Process HTML files using Jsoup (see https://jsoup.org)
-- PropertiesTool : Process JDK properties files
-- SystemTool : Expose System-related utility methods
-- UUIDTool : Create UUIDs
-- XmlTool : Process XML files using Apache FreeMarker (see https://freemarker.apache.org/docs/xgui.html)
-- YamlTool : Process YAML files using SnakeYAML(see https://bitbucket.org/asomov/snakeyaml/wiki/Home)
-
-13) Document Data Model
----------------------------------------------------------------------------
-- CSVTool
-- DataFrameTool
-- DataSources
-- ExcelTool
-- ExecTool
-- FreeMarkerTool
-- GrokTool
-- GsonTool
-- JsonPathTool
-- JsoupTool
-- PropertiesTool
-- SystemTool
-- UUIDTool
-- XmlTool
-- YamlTool
-
-14) Create a UUID
----------------------------------------------------------------------------
-UUIDTool Random UUID : 832a20fb-ac22-460a-b5d0-33c6fcc126d2
-UUIDTool Named UUID : 298415f9-e888-3d98-90e7-6c0d63ad14dc
-
-15) Printing Special Characters
----------------------------------------------------------------------------
-German Special Characters: äöüßÄÖÜ
-
-16) Locale-specific output
----------------------------------------------------------------------------
-Small Number : 1.23
-Large Number : 12,345,678.90
-Date : Jun 18, 2020
-Time : 11:13:51 AM
-
-17) Execute a program
----------------------------------------------------------------------------
-> date
-Thu Jun 18 11:13:51 CEST 2020
-```
-
-# 7. Design Considerations
-
-## 7.1 How It Works
-
-* The user-supplied files are loaded or input is read from `stdin`
-* The FreeMarker data model containing the data sources and tools is created and passed to the template
-* The generated output is written to the user-supplied file or to `stdout`
-
-## 7.2 FreeMarker Data Model
-
-Within the script a FreeMarker data model is set up and passed to the template - it contains the data sources to be processed and the following tools
-
-| Entry | Description |
-|-----------------------|-----------------------------------------------------------------------------------------------------------|
-| CSVTool | Process CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/) |
-| DataFrameTool | Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame) |
-| ExecTool | Execute command line tools using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/) |
-| ExcelTool | Process Excels files (XLS, XLSX) using [Apache POI](https://poi.apache.org) |
-| DataSources | Helper class to find data sources, e.g. by name, extension or index |
-| FreeMarkerTool | Expose useful FreeMarker classes |
-| GrokTool | Process text files using [Grok](https://github.com/thekrakken/java-grok) instead of regular expressions |
-| GsonTool | Process JSON files using [GSON](https://github.com/google/gson) |
-| JsonPathTool | Process JSON file using [Java JSON Path](https://github.com/json-path/JsonPath) |
-| JsoupTool | Processing HTML files using [Jsoup](https://jsoup.org) |
-| PropertiesTool | Process JDK properties files |
-| SystemTool | System-related utility methods |
-| UUIDTool | Create UUIDs |
-| XmlTool | Process XML files using [Apache FreeMarker](https://freemarker.apache.org/docs/xgui.html) |
-| YamlTool | Process YAML files using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home) |
-
-
-# 8. Tips & Tricks
-
-## 8.1 Template Base Directory
-
-When doing some ad-hoc scripting it is useful to rely on a base directory to resolve the FTL templates
-
-* As a default the FTL templates are resolved relative to the script directory
-* The caller can provide a `-b` or `--basedir` command line parameter
-
-```text
-./bin/freemarker-cli -b examples/templates -t json/md/github-users.ftl examples/data/json/github-users.json; echo
+./run-examples.sh
+templates/info.ftl
+examples/templates/demo.ftl
+templates/csv/html/transform.ftl
+templates/csv/md/transform.ftl
+examples/templates/csv/shell/curl.ftl
+examples/templates/csv/md/filter.ftl
+examples/templates/csv/fo/transform.ftl
+fop -fo target/out/locker-test-users.fo target/out/locker-test-users.pdf
+examples/templates/csv/fo/transactions.ftl
+fop -fo target/out/transactions.fo target/out/transactions-fo.pdf
+templates/csv/html/transform.ftl
+wkhtmltopdf -O landscape target/out/transactions.html target/out/transactions-html.pdf
+examples/templates/dataframe/example.ftl
+examples/templates/accesslog/combined-access.ftl
+examples/templates/excel/dataframe/transform.ftl
+templates/excel/html/transform.ftl
+templates/excel/md/transform.ftl
+templates/excel/csv/transform.ftl
+examples/templates/excel/csv/custom.ftl
+examples/templates/html/csv/dependencies.ftl
+examples/templates/json/csv/swagger-endpoints.ftl
+templates/json/yaml/transform.ftl
+examples/templates/json/md/github-users.ftl
+examples/templates/properties/csv/locker-test-users.ftl
+examples/data/template
+examples/templates/yaml/txt/transform.ftl
+templates/yaml/json/transform.ftl
+examples/templates/xml/txt/recipients.ftl
+Created the following sample files in ./target/out
+total 1464
+-rw-r--r-- 1 sgoeschl staff 646 Jun 28 08:21 combined-access.log.txt
+-rw-r--r-- 1 sgoeschl staff 25676 Jun 28 08:20 contract.html
+-rw-r--r-- 1 sgoeschl staff 7933 Jun 28 08:20 contract.md
+-rw-r--r-- 1 sgoeschl staff 784 Jun 28 08:20 curl.sh
+-rw-r--r-- 1 sgoeschl staff 232 Jun 28 08:21 customer.txt
+-rw-r--r-- 1 sgoeschl staff 6486 Jun 28 08:21 dataframe.txt
+-rw-r--r-- 1 sgoeschl staff 15563 Jun 28 08:20 demo.txt
+-rw-r--r-- 1 sgoeschl staff 1310 Jun 28 08:21 dependencies.csv
+-rw-r--r-- 1 sgoeschl staff 2029 Jun 28 08:21 github-users-curl.md
+-rw-r--r-- 1 sgoeschl staff 2624 Jun 28 08:20 info.txt
+-rw-r--r-- 1 sgoeschl staff 8075 Jun 28 08:20 interactive-dataframe.txt
+-rw-r--r-- 1 sgoeschl staff 66 Jun 28 08:20 interactive-html.txt
+-rw-r--r-- 1 sgoeschl staff 16 Jun 28 08:20 interactive-json.txt
+-rw-r--r-- 1 sgoeschl staff 25090 Jun 28 08:20 interactive-swagger.json
+-rw-r--r-- 1 sgoeschl staff 16870 Jun 28 08:20 interactive-swagger.yaml
+-rw-r--r-- 1 sgoeschl staff 10 Jun 28 08:20 interactive-xml.txt
+-rw-r--r-- 1 sgoeschl staff 285 Jun 28 08:21 locker-test-users.csv
+-rw-r--r-- 1 sgoeschl staff 6341 Jun 28 08:20 locker-test-users.fo
+-rw-r--r-- 1 sgoeschl staff 5526 Jun 28 08:20 locker-test-users.pdf
+-rw-r--r-- 1 sgoeschl staff 921 Jun 28 08:21 recipients.txt
+-rw-r--r-- 1 sgoeschl staff 910 Jun 28 08:20 sales-records.md
+-rw-r--r-- 1 sgoeschl staff 2453 Jun 28 08:21 swagger-spec.csv
+-rw-r--r-- 1 sgoeschl staff 25090 Jun 28 08:21 swagger-spec.json
+-rw-r--r-- 1 sgoeschl staff 16870 Jun 28 08:21 swagger-spec.yaml
+drwxr-xr-x 4 sgoeschl staff 128 Jun 28 08:21 template
+-rw-r--r-- 1 sgoeschl staff 154 Jun 28 08:21 test-multiple-sheets.xlsx.csv
+-rw-r--r-- 1 sgoeschl staff 1917 Jun 28 08:21 test-multiple-sheets.xlsx.html
+-rw-r--r-- 1 sgoeschl staff 389 Jun 28 08:21 test-multiple-sheets.xlsx.md
+-rw-r--r-- 1 sgoeschl staff 155 Jun 28 08:21 test-transform-xls.csv
+-rw-r--r-- 1 sgoeschl staff 1439 Jun 28 08:21 test.xls.dataframe.txt
+-rw-r--r-- 1 sgoeschl staff 1556 Jun 28 08:21 test.xls.html
+-rw-r--r-- 1 sgoeschl staff 1558 Jun 28 08:21 test.xslx.html
+-rw-r--r-- 1 sgoeschl staff 25758 Jun 28 08:20 transactions-fo.pdf
+-rw-r--r-- 1 sgoeschl staff 66016 Jun 28 08:21 transactions-html.pdf
+-rw-r--r-- 1 sgoeschl staff 330128 Jun 28 08:20 transactions.fo
+-rw-r--r-- 1 sgoeschl staff 51008 Jun 28 08:21 transactions.html
```
-## 8.2 Using Pipes
-
-When doing ad-hoc scripting it useful to pipe the output of one command directly into "freemarker-cli"
-```text
-cat examples/data/json/github-users.json | ./bin/freemarker-cli -t examples/templates/json/md/github-users.ftl --stdin; echo
-```
diff --git a/freemarker-generator-cli/examples/templates/dataframe/example.ftl b/freemarker-generator-cli/examples/templates/dataframe/example.ftl
index d1eef7b..9929d21 100644
--- a/freemarker-generator-cli/examples/templates/dataframe/example.ftl
+++ b/freemarker-generator-cli/examples/templates/dataframe/example.ftl
@@ -1,4 +1,3 @@
-<#ftl output_format="plainText">
<#--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -16,7 +15,7 @@
under the License.
-->
<#assign dataSource = DataSources.get(0)>
-<#assign csvParser = CSVTool.parse(dataSource, csvInFormat())>
+<#assign csvParser = CSVTool.parse(dataSource, CSVTool.formats["DATAFRAME"])>
<#assign users = DataFrameTool.fromCSVParser(csvParser)>
Original Data
@@ -45,14 +44,3 @@ ${DataFrameTool.print(users.getColumn("country").transform(DataFrameTool.transfo
Group By Age & Country
=============================================================================
${DataFrameTool.print(users.groupBy("country", "age").sort("country"))}
-<#--------------------------------------------------------------------------->
-<#function csvInFormat>
- <#assign format = CSVTool.formats[CSV_SOURCE_FORMAT!"DEFAULT"]>
- <#assign delimiter = CSVTool.toDelimiter(CSV_SOURCE_DELIMITER!format.getDelimiter())>
- <#assign withHeader = CSV_SOURCE_WITH_HEADER!"false">
- <#assign format = format.withDelimiter(delimiter)>
- <#if withHeader?boolean>
- <#assign format = format.withHeader()>
- </#if>
- <#return format>
-</#function>
diff --git a/freemarker-generator-cli/examples/templates/info.ftl b/freemarker-generator-cli/examples/templates/info.ftl
deleted file mode 100644
index 565e2ec..0000000
--- a/freemarker-generator-cli/examples/templates/info.ftl
+++ /dev/null
@@ -1,67 +0,0 @@
-<#ftl output_format="plainText" strip_whitespace=true>
-<#--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-FreeMarker CLI Information
-------------------------------------------------------------------------------
-FreeMarker version : ${.version}
-Template name : ${.current_template_name}
-Language : ${.lang}
-Locale : ${.locale}
-Timestamp : ${.now}
-Output encoding : ${.output_encoding}
-Output format : ${.output_format}
-
-FreeMarker CLI Template Directories
-------------------------------------------------------------------------------
-<#list SystemTool.getTemplateDirectories() as directory>
-[#${directory?counter}] ${directory}
-</#list>
-
-FreeMarker CLI Tools
-------------------------------------------------------------------------------
-<#list .data_model?keys?sort as key>
-<#if key?ends_with("Tool")>
-- ${key?right_pad(20)} : ${.data_model[key]}
-</#if>
-</#list>
-
-FreeMarker CLI Data Model
----------------------------------------------------------------------------
-<#list .data_model?keys?sort as key>
-- ${key}<#lt>
-</#list>
-
-<#if DataSources.list?has_content>
-FreeMarker CLI DataSources
-------------------------------------------------------------------------------
-<#list DataSources.list as dataSource>
- [#${dataSource?counter}], name=${dataSource.name}, group=${dataSource.group}, contentType=${dataSource.contentType}, charset=${dataSource.charset}, length=${dataSource.length} Bytes
- URI : ${dataSource.uri}
-</#list>
-</#if>
-
-<#if SystemTool.parameters?has_content>
-FreeMarker CLI Parameters
-------------------------------------------------------------------------------
-<#list SystemTool.parameters as key,value>
-<#if value?is_hash>
-- ${key} ==> { <#list value as name,value>${name}=${value} </#list>}
-<#else>
-- ${key} ==> ${value}
-</#if>
-</#list>
-</#if>
diff --git a/freemarker-generator-cli/examples/templates/json/dataframe/github-users.ftl b/freemarker-generator-cli/examples/templates/json/dataframe/github-users.ftl
new file mode 100644
index 0000000..a6547fc
--- /dev/null
+++ b/freemarker-generator-cli/examples/templates/json/dataframe/github-users.ftl
@@ -0,0 +1,20 @@
+<#ftl output_format="plainText" >
+<#--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<#assign json = GsonTool.parse(DataSources.get(0))>
+<#assign dataframe = DataFrameTool.fromMaps(json)>
+${DataFrameTool.print(dataframe)}
diff --git a/freemarker-generator-cli/pom.xml b/freemarker-generator-cli/pom.xml
index 47756a9..81ee0ac 100644
--- a/freemarker-generator-cli/pom.xml
+++ b/freemarker-generator-cli/pom.xml
@@ -67,14 +67,13 @@
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
+ <id>package</id>
<phase>package</phase>
<configuration>
<target>
+ <copy file="CHANGELOG.md" todir="./target/appassembler" />
<copy file="README.md" todir="./target/appassembler" />
<copy file="./src/main/scripts/run-examples.sh" todir="./target/appassembler" />
- <copy todir="./target/appassembler/site">
- <fileset dir="site" />
- </copy>
<copy todir="./target/appassembler/templates">
<fileset dir="templates" />
</copy>
@@ -99,7 +98,6 @@
<configuration>
<tarLongFileMode>gnu</tarLongFileMode>
<descriptors>
- <descriptor>${basedir}/src/main/assembly/dist.xml</descriptor>
<descriptor>${basedir}/src/main/assembly/app.xml</descriptor>
</descriptors>
</configuration>
@@ -126,6 +124,20 @@
</goals>
</execution>
</executions>
+ <configuration>
+ <dotGitDirectory>${project.basedir}/.git</dotGitDirectory>
+ <prefix>git</prefix>
+ <verbose>false</verbose>
+ <generateGitPropertiesFile>true</generateGitPropertiesFile>
+ <generateGitPropertiesFilename>${project.build.outputDirectory}/git.properties
+ </generateGitPropertiesFilename>
+ <format>properties</format>
+ <gitDescribe>
+ <skip>false</skip>
+ <always>false</always>
+ <dirty>-dirty</dirty>
+ </gitDescribe>
+ </configuration>
</plugin>
</plugins>
</build>
@@ -177,7 +189,7 @@
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
+ <artifactId>slf4j-nop</artifactId>
</dependency>
<!-- Testing -->
<dependency>
diff --git a/freemarker-generator-cli/src/main/assembly/dist.xml b/freemarker-generator-cli/src/main/assembly/dist.xml
deleted file mode 100644
index 80dc5b4..0000000
--- a/freemarker-generator-cli/src/main/assembly/dist.xml
+++ /dev/null
@@ -1,41 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
- <id>dist</id>
- <formats>
- <format>tar.gz</format>
- </formats>
- <fileSets>
- <fileSet>
- <directory>${project.basedir}</directory>
- <outputDirectory>${file.separator}</outputDirectory>
- <useDefaultExcludes>true</useDefaultExcludes>
- <excludes>
- <exclude>./src/**</exclude>
- <exclude>./target/**</exclude>
- <exclude>./.idea/**</exclude>
- <exclude>./*.iml</exclude>
- <exclude>./pom.xml</exclude>
- <exclude>./pom.xml</exclude>
- </excludes>
- </fileSet>
- </fileSets>
-</assembly>
\ No newline at end of file
diff --git a/freemarker-generator-cli/src/main/java/org/apache/freemarker/generator/cli/Main.java b/freemarker-generator-cli/src/main/java/org/apache/freemarker/generator/cli/Main.java
index 6315aa0..7e86f85 100644
--- a/freemarker-generator-cli/src/main/java/org/apache/freemarker/generator/cli/Main.java
+++ b/freemarker-generator-cli/src/main/java/org/apache/freemarker/generator/cli/Main.java
@@ -66,7 +66,7 @@ public class Main implements Callable<Integer> {
public String interactiveTemplate;
}
- @Option(names = { "-b", "--basedir" }, description = "optional template base directory")
+ @Option(names = { "-b", "--basedir" }, description = "additional template base directory")
String baseDir;
@Option(names = { "-D", "--system-property" }, description = "set system property")
@@ -87,7 +87,7 @@ public class Main implements Callable<Integer> {
@Option(names = { "-P", "--param" }, description = "set parameter")
Map<String, String> parameters;
- @Option(names = { "-s", "--data-source" }, description = "data source used for redering")
+ @Option(names = { "-s", "--data-source" }, description = "data source used for rendering")
List<String> dataSources;
@Option(names = { "--config" }, defaultValue = FREEMARKER_CLI_PROPERTY_FILE, description = "FreeMarker CLI configuration file")
diff --git a/freemarker-generator-cli/src/site/markdown/cli/advanced/cli-configuration.md b/freemarker-generator-cli/src/site/markdown/cli/advanced/cli-configuration.md
new file mode 100644
index 0000000..cde2cd1
--- /dev/null
+++ b/freemarker-generator-cli/src/site/markdown/cli/advanced/cli-configuration.md
@@ -0,0 +1,77 @@
+## CLI Configuration
+
+The `Apache FreeMarker CLI` configuration can be tweaked to
+
+* Change the underlying `Apache FreeMarker Configuration`
+* Instantiate custom tools
+* Storing user-specific templates
+
+### Apache FreeMarker CLI Properties File
+
+`Apache FreeMarker CLI` reads the `conf/freemarker-cli.properties`
+
+```
+#############################################################################
+# General FreeMarker Configuration
+# See https://freemarker.apache.org/docs/api/freemarker/template/Configuration.html#setSetting-java.lang.String-java.lang.String-
+#############################################################################
+# freemarker.configuration.setting.locale=JVM default
+
+#############################################################################
+# Configure FreeMarker Tools (name -> implementation class)
+#############################################################################
+freemarker.tools.CSVTool=org.apache.freemarker.generator.tools.commonscsv.CommonsCSVTool
+freemarker.tools.DataFrameTool=org.apache.freemarker.generator.tools.dataframe.DataFrameTool
+freemarker.tools.ExcelTool=org.apache.freemarker.generator.tools.excel.ExcelTool
+freemarker.tools.ExecTool=org.apache.freemarker.generator.tools.commonsexec.CommonsExecTool
+freemarker.tools.FreeMarkerTool=org.apache.freemarker.generator.tools.freemarker.FreeMarkerTool
+freemarker.tools.GrokTool=org.apache.freemarker.generator.tools.grok.GrokTool
+freemarker.tools.GsonTool=org.apache.freemarker.generator.tools.gson.GsonTool
+freemarker.tools.JsonPathTool=org.apache.freemarker.generator.tools.jsonpath.JsonPathTool
+freemarker.tools.JsoupTool=org.apache.freemarker.generator.tools.jsoup.JsoupTool
+freemarker.tools.PropertiesTool=org.apache.freemarker.generator.tools.properties.PropertiesTool
+freemarker.tools.SystemTool=org.apache.freemarker.generator.tools.system.SystemTool
+freemarker.tools.UUIDTool=org.apache.freemarker.generator.tools.uuid.UUIDTool
+freemarker.tools.XmlTool=org.apache.freemarker.generator.tools.xml.XmlTool
+freemarker.tools.YamlTool=org.apache.freemarker.generator.tools.snakeyaml.SnakeYamlTool
+```
+
+Changing this file allows to tweak the underlying `Apache FreeMarker Configuration` and add custom tools.
+
+### Storing User-Specific Templates
+
+Over the time you will accumalate more and more `Apache FreeMarker` templates - some of them are stored within a project but some of the more general might be free-floating and you don't want to store them in the installation directory.
+
+To give those free-floating templates a home `Apache FreeMarker CLI` tries to read templates from `~/freemarker-cli`, e.g.
+
+```
+tree ~/.freemarker-cli/
+/Users/sgoeschl/.freemarker-cli/
+`-- templates
+ `-- json
+ |-- confluence
+ | |-- aws
+ | | `-- describe.ftl
+ |-- csv
+ | `-- swagger-endpoints.ftl
+ |-- ftl
+ | `-- customer-user-products.ftl
+ |-- html
+ | `-- customer-user-products.ftl
+ `-- md
+ `-- customer-user-products.ftl
+```
+
+If a `~/freemarker-cli` is found it will be automatically added to the `FreeMarker Template Loader` (for more information see https://freemarker.apache.org/docs/pgui_config_templateloading.html)
+
+You can easily check this, e.g.
+
+```
+> freemarker-cli -t templates/info.ftl
+
+FreeMarker CLI Template Loader Directories
+------------------------------------------------------------------------------
+[#1] /Users/sgoeschl
+[#2] /Users/sgoeschl/.freemarker-cli
+[#3] /Applications/Java/freemarker-cli-2.0.0
+```
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/data-models.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/data-models.md
index c05323c..25bb3f9 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/concepts/data-models.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/data-models.md
@@ -9,21 +9,44 @@ A `DataModel` is an eagerly loaded `DataSource` available in Apache FreeMarker's
Expose the fields of the JSON data source in FreeMarker's model
```
-bin/freemarker-cli --data-model https://xkcd.com/info.0.json -i '<a href="${img}">${title}</a>'
-<a href="https://imgs.xkcd.com/comics/scenario_4.png">Scenario 4</a>
+> curl -s https://xkcd.com/info.0.json | python -m json.tool
+{
+ "alt": "The git vehicle fleet eventually pivoted to selling ice cream, but some holdovers remain. If you flag down an ice cream truck and hand the driver a floppy disk, a few hours later you'll get an invite to a git repo.",
+ "day": "24",
+ "img": "https://imgs.xkcd.com/comics/old_days_2.png",
+ "link": "",
+ "month": "6",
+ "news": "",
+ "num": 2324,
+ "safe_title": "Old Days 2",
+ "title": "Old Days 2",
+ "transcript": "",
+ "year": "2020"
+}
+
+> freemarker-cli --data-model https://xkcd.com/info.0.json -i '<a href="${img}">${title}</a>'; echo
+<a href="https://imgs.xkcd.com/comics/old_days_2.png">Old Days 2</a>
```
Exposed the JSON data source as variable `post` in FreeMarker's model
```
-bin/freemarker-cli --data-model post=https://jsonplaceholder.typicode.com/posts/2 -i 'post title is: ${post.title}'
+> curl -s https://jsonplaceholder.typicode.com/posts/2 | python -m json.tool
+{
+ "body": "est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla",
+ "id": 2,
+ "title": "qui est esse",
+ "userId": 1
+}
+
+> freemarker-cli --data-model post=https://jsonplaceholder.typicode.com/posts/2 -i 'post title is: ${post.title}'; echo
post title is: qui est esse
```
Expose all environment variables as `env` in theFreeMarker model
```
-bin/freemarker-cli --data-model env=env:/// -i '<#list env as name,value>${name}=${value}${"\n"}</#list>'
+> freemarker-cli --data-model env=env:/// -i '<#list env as name,value>${name}=${value}${"\n"}</#list>'
HOME=/Users/sgoeschl
USER=sgoeschl
```
@@ -31,14 +54,14 @@ USER=sgoeschl
Expose a single envionment variable in theFreeMarker model
```
-bin/freemarker-cli --data-model NAME=env:///USER -i 'Hello ${NAME}'
+> freemarker-cli --data-model NAME=env:///USER -i 'Hello ${NAME}'; echo
Hello sgoeschl
```
Alternatively use the short command line options, e.g.
```
-bin/freemarker-cli -m NAME=env:///USER -i 'Hello ${NAME}!'
+> freemarker-cli -m NAME=env:///USER -i 'Hello ${NAME}!'; echo
Hello sgoeschl!
```
@@ -51,7 +74,7 @@ The following snippet shows a more advanced example
> export DB_CONFIG='{"db_default_user":"scott","db_default_password":"tiger"}'
> echo $DB_CONFIG
{"db_default_user":"scott","db_default_password":"tiger"}
-> bin/freemarker-cli -m config=env:///DB_CONFIG#mimetype=application/json -i '<#list config as name,value>${name}=${value}${"\n"}</#list>'
+> freemarker-cli -m config=env:///DB_CONFIG#mimetype=application/json -i '<#list config as name,value>${name}=${value}${"\n"}</#list>'
db_default_user=scott
db_default_password=tiger
```
\ No newline at end of file
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/data-sources.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/data-sources.md
index 548a65b..1a921fc 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/concepts/data-sources.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/data-sources.md
@@ -13,7 +13,7 @@ A `DataSource` consists of lazy-loaded data available in Apache FreeMarker's mod
A `DataSource` can be loaded from the file system, e.g. as positional command line argument
```
-bin/freemarker-cli -t templates/info.ftl README.md
+freemarker-cli -t templates/info.ftl README.md
FreeMarker CLI DataSources
------------------------------------------------------------------------------
@@ -24,7 +24,7 @@ FreeMarker CLI DataSources
from an URL
```
-bin/freemarker-cli --data-source xkcd=https://xkcd.com/info.0.json -t templates/info.ftl
+freemarker-cli --data-source xkcd=https://xkcd.com/info.0.json -t templates/info.ftl
FreeMarker CLI DataSources
------------------------------------------------------------------------------
@@ -36,7 +36,7 @@ or from an environment variable, e.g. `NGINX_CONF` having a JSON payload
```
export NGINX_CONF='{"NGINX_PORT":"8443","NGINX_HOSTNAME":"localhost"}'
-bin/freemarker-cli -t templates/info.ftl -s conf=env:///NGINX_CONF#mimetype=application/json
+freemarker-cli -t templates/info.ftl -s conf=env:///NGINX_CONF#mimetype=application/json
FreeMarker CLI DataSources
------------------------------------------------------------------------------
@@ -47,7 +47,7 @@ FreeMarker CLI DataSources
Of course you can load multiple `DataSources` directly
```
-bin/freemarker-cli -t templates/info.ftl README.md xkcd=https://xkcd.com/info.0.json
+freemarker-cli -t templates/info.ftl README.md xkcd=https://xkcd.com/info.0.json
FreeMarker CLI DataSources
------------------------------------------------------------------------------
@@ -60,7 +60,7 @@ FreeMarker CLI DataSources
or load them from a directory
```
-bin/freemarker-cli -t templates/info.ftl -s examples/data/
+freemarker-cli -t templates/info.ftl -s examples/data
FreeMarker CLI DataSources
------------------------------------------------------------------------------
[#1], name=combined-access.log, group=default, contentType=text/plain, charset=UTF-8, length=2,068 Bytes
@@ -74,7 +74,7 @@ FreeMarker CLI DataSources
which can be combined with `include` and `exclude` filters
```
-bin/freemarker-cli -t templates/info.ftl -s examples/data --data-source-include=*.json
+freemarker-cli -t templates/info.ftl -s examples/data --data-source-include=*.json
FreeMarker CLI DataSources
------------------------------------------------------------------------------
@@ -82,5 +82,30 @@ FreeMarker CLI DataSources
URI : file:/Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler/examples/data/json/github-users.json
[#2], name=swagger-spec.json, group=default, contentType=application/json, charset=UTF-8, length=24,948 Bytes
URI : file:/Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler/examples/data/json/swagger-spec.json
+```
+
+### Selecting A DataSource
+
+After loading one or more `DataSource` it needs to be selected for template processing - the `DataSources` instance
+exposed in the data model provides
+
+* Selecting by index
+* Selecting by name
+* Filter by the globbing pattern (see [Apache Commons IO](https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/filefilter/WildcardFileFilter.html))
+
+A few FTL examples
```
+<#assign dataSource = DataSources.get(0)>
+
+<#assign dataSource = DataSources.get("user.csv)>
+
+<#list DataSources.find("*.md") as dataSource>
+- ${dataSource.name}
+</#list>
+```
+
+
+
+
+
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/design-goals.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/design-goals.md
new file mode 100644
index 0000000..0d2e07c
--- /dev/null
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/design-goals.md
@@ -0,0 +1,17 @@
+## Design Goals
+
+* Create a proper command-line tool which has Unix look & feel
+* Handle arbitrary large input and output data
+* Support multiple source files/directories for a single transformation
+* Support transformation of Property files using plain-vanilla JDK
+* Support transformation of CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/)
+* Support transformation of JSON using [Jayway's JSONPath](https://github.com/jayway/JsonPath) and [GSON](https://github.com/google/gson)
+* Support transformation of Excel using [Apache POI](https://poi.apache.org)
+* Support transformation of YAML using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home)
+* Support transformation of HTML using [JSoup](https://jsoup.org)
+* Support transformation of structured logfiles using [Grok](https://github.com/thekrakken/java-grok)
+* XML & XPath is supported by FreeMarker [out-of-the-box](http://freemarker.org/docs/xgui.html)
+* Support for reading a data source content from STDIN to integrate with command line tools
+* Support execution of arbitrary commands using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/)
+* Add some commonly useful information such as `System Properties`, `Enviroment Variables`
+* Support embedding the code in existing applications
\ No newline at end of file
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/named-uris.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/named-uris.md
index 1c99b27..58f9faa 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/concepts/named-uris.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/named-uris.md
@@ -1,4 +1,4 @@
-# Named URIs
+## Named URIs
Named URIs allow to identify `DataSources` and pass additional information
@@ -29,7 +29,7 @@ For our purposes, the scheme and the path components are especially important, t
The following Named URI loads a "user.csv" and the data source is available as `my_users`
```
-bin/freemarker-cli -t templates/info.ftl my_users=examples/data/csv/user.csv
+freemarker-cli -t templates/info.ftl my_users=examples/data/csv/user.csv
[#1], name=my_users, group=default, contentType=text/csv, charset=UTF-8, length=376 Bytes
URI : file:examples/data/csv/user.csv
```
@@ -37,7 +37,7 @@ URI : file:examples/data/csv/user.csv
A Named URI allows to pass additional information as part of the fragment, e.g. the charset of the text file
```
-bin/freemarker-cli -t templates/info.ftl my_users=examples/data/csv/user.csv#charset=UTF-16
+freemarker-cli -t templates/info.ftl my_users=examples/data/csv/user.csv#charset=UTF-16
[#1], name=my_users, group=default, contentType=text/csv, charset=UTF-16, length=376 Bytes
URI : file:examples/data/csv/user.csv
```
@@ -45,7 +45,7 @@ URI : file:examples/data/csv/user.csv
In addition to the simplified file syntax full URIs can be used
```
-bin/freemarker-cli -t templates/info.ftl http://google.com?foo=bar
+freemarker-cli -t templates/info.ftl http://google.com?foo=bar
[#1], name=google.com, group=default, contentType=text/html, charset=ISO-8859-1, length=-1 Bytes
URI : http://google.com?foo=bar
```
@@ -53,7 +53,7 @@ URI : http://google.com?foo=bar
and also combined with a name
```
-bin/freemarker-cli -t templates/info.ftl page=http://google.com?foo=bar
+freemarker-cli -t templates/info.ftl page=http://google.com?foo=bar
[#1], name=page, group=default, contentType=text/html, charset=ISO-8859-1, length=-1 Bytes
URI : http://google.com?foo=bar
```
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/user-parameters.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/passing-data.md
similarity index 58%
rename from freemarker-generator-cli/src/site/markdown/cli/concepts/user-parameters.md
rename to freemarker-generator-cli/src/site/markdown/cli/concepts/passing-data.md
index 46f2d2e..eb6110e 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/concepts/user-parameters.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/passing-data.md
@@ -1,4 +1,19 @@
-# User-Supplied Parameters
+## Passing Configuration Data
+
+`Apache FreeMarker CLI` provides multiple ways to pass configuration data used in in templates
+
+* System properties
+* Parameters
+
+### User-Supplied System Poperties
+
+User-supplied system properties are added to the JVM's system properties
+
+```
+> freemarker-cli -Dfoo1=foo1 -D foo2=foo2 -t templates/info.ftl
+```
+
+### User-Supplied Parameters
User-supplied parameters allow to pass additional information to an Apache FreeMarker template
@@ -8,7 +23,7 @@ User-supplied parameters allow to pass additional information to an Apache FreeM
Pass a simple name/value pair on the command line
```
-> bin/freemarker-cli -t templates/info.ftl -P key=value
+> freemarker-cli -t templates/info.ftl -P key=value
User Supplied Parameters
------------------------------------------------------------------------------
@@ -18,7 +33,7 @@ User Supplied Parameters
By providing a `group` you can create nested maps
```
-> bin/freemarker-cli -t templates/info.ftl -P foo1:group=bar1 -P foo2:group=bar2
+> freemarker-cli -t templates/info.ftl -P foo1:group=bar1 -P foo2:group=bar2
User Supplied Parameters
------------------------------------------------------------------------------
@@ -28,7 +43,7 @@ User Supplied Parameters
It is also possible to mix and match the two approaches
```
-> bin/freemarker-cli -t templates/info.ftl -P foo1:group=bar1 -P foo2:group=bar2 -P key=value
+> freemarker-cli -t templates/info.ftl -P foo1:group=bar1 -P foo2:group=bar2 -P key=value
User Supplied Parameters
------------------------------------------------------------------------------
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/template-loading.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/template-loading.md
new file mode 100644
index 0000000..f54228e
--- /dev/null
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/template-loading.md
@@ -0,0 +1,60 @@
+## Template Loading
+
+In order the render a template it needs to be loaded first - there are multiple ways of creating/loading a template
+
+* Using a [FreeMarker Template Loader](https://freemarker.apache.org/docs/pgui_config_templateloading.html) and abstract template paths
+* Load a template without template loader (aka free-style template loading), e.g. absolute template file, a directory or an URL
+* Provide the template directly on the command-line (aka interactive template)
+
+### FreeMarker MultiTemplateLoader
+
+`Apache FreeMarker CLI` uses a `MultiTemplateLoader` searching for templates in the following directories
+
+* Current working directory
+* Optional `~/.freemarker-cli` directory
+* `Apache FreeMarker CLI` installation directory
+
+You can check the currently used template loader directories easily on the command line, e.g.
+
+```
+freemarker-cli -t templates/info.ftl
+
+FreeMarker CLI Template Loader Directories
+------------------------------------------------------------------------------
+[#1] /Users/sgoeschl/work/github/apache/freemarker-generator
+[#2] /Users/sgoeschl/.freemarker-cli
+[#3] /Applications/Java/freemarker-cli-2.0.0
+```
+
+The main benefit of `MultiTemplateLoader` is the use of abstract template paths finding a template in the template loader directories
+
+```
+freemarker-cli -t templates/info.ftl
+```
+
+and [Template Includes](https://freemarker.apache.org/docs/ref_directive_include.html)
+
+```
+<#import "/templates/lib/commons-csv.ftl" as csv />
+```
+
+### Free-Style Template Loading
+
+The previosly described `Template Loaders` do not support absolute template files or arbitraRY URLS - this behaviour
+stems from security aspects when running `Apache FreeMarker` on the server side. For a command-line tool this is mostly
+irrelevant therefore any template file outside of the template loader directories can be loaded
+
+This example loads the `info.ftl` directly from a GitHub URL
+
+```
+freemarker-cli -t https://raw.githubusercontent.com/apache/freemarker-generator/master/freemarker-generator-cli/templates/info.ftl
+```
+
+### Interactive Template Loading
+
+The template can be defined directly on the command line in case of trivial transformations
+
+```
+freemarker-cli -i '${GsonTool.toJson(yaml)}' -m yaml=examples/data/yaml/swagger-spec.yaml
+```
+
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/tools.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/tools.md
new file mode 100644
index 0000000..c07a01a
--- /dev/null
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/tools.md
@@ -0,0 +1,22 @@
+## Tools
+
+A `Apache FreeMarker CLI Tool` is just a POJO (plain old Java object) that is "useful" in a template and is not meant to be rendered in the output.
+
+The following tools are currently implemented
+
+| Tool | Description |
+|-----------------------|-----------------------------------------------------------------------------------------------------------|
+| CSVTool | Process CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/) |
+| DataFrameTool | Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame) |
+| ExecTool | Execute command line tools using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/) |
+| ExcelTool | Process Excels files (XLS, XLSX) using [Apache POI](https://poi.apache.org) |
+| FreeMarkerTool | Expose useful FreeMarker classes |
+| GrokTool | Process text files using [Grok](https://github.com/thekrakken/java-grok) instead of regular expressions |
+| GsonTool | Process JSON files using [GSON](https://github.com/google/gson) |
+| JsonPathTool | Process JSON file using [Java JSON Path](https://github.com/json-path/JsonPath) |
+| JsoupTool | Processing HTML files using [Jsoup](https://jsoup.org) |
+| PropertiesTool | Process JDK properties files |
+| SystemTool | System-related utility methods |
+| UUIDTool | Create UUIDs |
+| XmlTool | Process XML files using [Apache FreeMarker](https://freemarker.apache.org/docs/xgui.html) |
+| YamlTool | Process YAML files using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home) |
diff --git a/freemarker-generator-cli/src/site/markdown/cli/concepts/transformation.md b/freemarker-generator-cli/src/site/markdown/cli/concepts/transformation.md
index 966743b..388a681 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/concepts/transformation.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/concepts/transformation.md
@@ -1,5 +1,7 @@
## Transformation
+The `freemarker-cli` generates text output based on FreeMarker templates and data
+
* A command line invocation requires 1..n `templates` and 0..n `data sources` / `data models`
* A command line invocation is mapped to a series of `transformations`
* The `transformation` consists of exactly one `template`, 0..n `data sources` / `data models` and an `output`
@@ -9,4 +11,4 @@
* an output directory
* When the output is written to a directory
* the structure of the input directory is preserved
- * a "ftl" file externsion is removed
+ * a `ftl` file externsion is removed
diff --git a/freemarker-generator-cli/src/site/markdown/cli/introduction/getting-started.md b/freemarker-generator-cli/src/site/markdown/cli/introduction/getting-started.md
new file mode 100644
index 0000000..a288348
--- /dev/null
+++ b/freemarker-generator-cli/src/site/markdown/cli/introduction/getting-started.md
@@ -0,0 +1,142 @@
+## Getting Started
+
+### Installation
+
+* Requires JDK 1.8+ on Linux, Mac OSX and Windows
+* Download [freemarker-generator-cli-0.1.0-SNAPSHOT-app.tar.gz] or [freemarker-generator-cli-0.1.0-SNAPSHOT-app.zip]
+* Unpack the archive in a directory of your choice
+* Add the `bin/freemarker-cli` or `bin/freemarker-cli.bat` to your `PATH` variable
+
+### Verify Installation
+
+On my local box (Mac OS 10.15.5) I use the following setup
+
+```
+export FREEMARKER_CLI_HOME=/Applications/Java/freemarker-cli-2.0.0
+export PATH=$PATH:$FREEMARKER_CLI_HOME/bin
+```
+
+Afterwards `Apache FreeMarker CLI` can be executed from the command line
+
+```
+> which freemarker-cli
+/Applications/Java/freemarker-cli-2.0.0/bin/freemarker-cli
+```
+
+and check the version of `Apache FreeMarker CLI`
+
+```
+> freemarker-cli -V
+version=0.1.0-SNAPSHOT, time=2020-06-25T21:48:02+0200, commit=b320d00094be8789086ad6153d9d3fcaf4b8c75f
+```
+
+### Command Line Options
+
+`Apache FreeMarker CLI` provides command line help as shown below
+
+```
+> freemarker-cli -h
+Usage: freemarker-cli (-t=<templates> [-t=<templates>]... |
+ -i=<interactiveTemplate>) [-hV] [--stdin] [-b=<baseDir>]
+ [--config=<configFile>] [--data-source-exclude=<exclude>]
+ [--data-source-include=<include>] [-e=<inputEncoding>]
+ [-l=<locale>] [-o=<outputFile>]
+ [--output-encoding=<outputEncoding>] [--times=<times>]
+ [-D=<String=String>]... [-m=<dataModels>]...
+ [-P=<String=String>]... [-s=<dataSources>]...
+ [<sources>...]
+Apache FreeMarker CLI
+ [<sources>...] data source files and/or directories
+ -b, --basedir=<baseDir> optional template base directory
+ --config=<configFile> FreeMarker CLI configuration file
+ -D, --system-property=<String=String>
+ set system property
+ --data-source-exclude=<exclude>
+ file exclude pattern for data sources
+ --data-source-include=<include>
+ file include pattern for data sources
+ -e, --input-encoding=<inputEncoding>
+ encoding of data source
+ -h, --help Show this help message and exit.
+ -i, --interactive=<interactiveTemplate>
+ interactive template to process
+ -l, --locale=<locale> locale being used for the output, e.g. 'en_US'
+ -m, --data-model=<dataModels>
+ data model used for rendering
+ -o, --output=<outputFile> output file or directory
+ --output-encoding=<outputEncoding>
+ encoding of output, e.g. UTF-8
+ -P, --param=<String=String>
+ set parameter
+ -s, --data-source=<dataSources>
+ data source used for redering
+ --stdin read data source from stdin
+ -t, --template=<templates>
+ template to process
+ --times=<times> re-run X times for profiling
+ -V, --version Print version information and exit.
+```
+
+### The Info Template
+
+The distribution ships with a couple of FreeMarker templates and the `templates/info.ftl` is particularly helpful
+to better understand `Apache FreeMarker CLI`
+
+```
+> freemarker-cli -t templates/info.ftl
+FreeMarker CLI Information
+------------------------------------------------------------------------------
+FreeMarker version : 2.3.30
+Template name : templates/info.ftl
+Language : en
+Locale : en_US
+Timestamp : Jun 26, 2020 10:44:15 AM
+Output encoding : UTF-8
+Output format : plainText
+
+FreeMarker CLI Template Loader Directories
+------------------------------------------------------------------------------
+[#1] /Users/sgoeschl/work/github/apache/freemarker-generator
+[#2] /Users/sgoeschl/.freemarker-cli
+[#3] /Applications/Java/freemarker-cli-2.0.0
+
+FreeMarker CLI Tools
+------------------------------------------------------------------------------
+- CSVTool : Process CSV files using Apache Commons CSV (see https://commons.apache.org/proper/commons-csv/)
+- DataFrameTool : Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame)
+- ExcelTool : Process Excels files (XLS, XLSX) using Apache POI (see https://poi.apache.org)
+- ExecTool : Execute command line tools using Apache Commons Exec (see https://commons.apache.org/proper/commons-exec/)
+- FreeMarkerTool : Expose advanced Apache FreeMarker classes
+- GrokTool : Process text files using Grok expressions (see https://github.com/thekrakken/java-grok)
+- GsonTool : Process JSON files using GSON (see https://github.com/google/gson)
+- JsonPathTool : Process JSON files using Java JSON Path (see https://github.com/json-path/JsonPath)
+- JsoupTool : Process HTML files using Jsoup (see https://jsoup.org)
+- PropertiesTool : Process JDK properties files
+- SystemTool : Expose System-related utility methods
+- UUIDTool : Create UUIDs
+- XmlTool : Process XML files using Apache FreeMarker (see https://freemarker.apache.org/docs/xgui.html)
+- YamlTool : Process YAML files using SnakeYAML(see https://bitbucket.org/asomov/snakeyaml/wiki/Home)
+
+FreeMarker CLI Data Model
+---------------------------------------------------------------------------
+- CSVTool
+- DataFrameTool
+- DataSources
+- ExcelTool
+- ExecTool
+- FreeMarkerTool
+- GrokTool
+- GsonTool
+- JsonPathTool
+- JsoupTool
+- PropertiesTool
+- SystemTool
+- UUIDTool
+- XmlTool
+- YamlTool
+```
+
+* The "FreeMarker CLI Information" section provides insights into configuration and currently processed template.
+* The "FreeMarker CLI Template Loader Directories" shows the template directories being searched to resolve a template path
+* The "FreeMarker CLI Tools" section list the available tools
+* The "FreeMarker CLI Data Model" section shows all available entries in the current FreeMarker context
diff --git a/freemarker-generator-cli/README.md b/freemarker-generator-cli/src/site/markdown/cli/usage/running-examples.md
similarity index 51%
copy from freemarker-generator-cli/README.md
copy to freemarker-generator-cli/src/site/markdown/cli/usage/running-examples.md
index 70179f2..9b329a1 100644
--- a/freemarker-generator-cli/README.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/usage/running-examples.md
@@ -1,140 +1,15 @@
-# Apache FreeMarker Generator CLI
+## Running Examples
-# 1. Is This Project For You?
-
-You somehow found this GitHub project and wonder if it solves a problem you might have?!
-
-* You need to transform some structured text document (CSV, HTML, JSON, XML, YAML, Java Property files, access logs) into CSV, HTML, Markdown or Confluence markup?
-* You need to convert an Excel document into CSV, HTML or Markdown?
-* You need to create a nice-looking PDF from some boring-looking CSV or JSON content ?
-
-The goal of `freemarker-cli` is to automate repeated transformation tasks
-
-* Which are too boring to be done manually
-* Which happen not often enough to write a dedicated script or program
-
-# 2. Once Upon A Time
-
-In December 2015 I needed a little bit of test data management for a customer project - to make a long story short (after writing a few more Groovy scripts) it boiled down to transforming one or more JSON files to something human readable.
-
-What are the options?
-
-* The cool kids say 'Node.js' - but they always say 'Node.js'
-* Some fancy Groovy scripts using Groovy's markup builder - but the syntax looks a bit odd
-* Using 'JsonPath' and 'Velocity' to reuse good & old stuff
-
-So I went with 'Apache Groovy', 'JsonPath' and 'Apache Velocity'
-
-* Playing with Groovy over the public holidays
-* Groovy has a built-in package manager which makes distribution a breeze
-* Providing samples to transform JSON to Markdown
-
-Using Velocity actually created some minor issues so I migrated to [Apache FreeMarker](https://freemarker.apache.org) during Christmas 2016
-
-* Velocity 1.7 was released 2010 and only recently there was a new release
-* I was painful to get Velocity Tools working
-* Velocity XML processing support is also painful
-* Spring 4.3 deprecated Velocity support which could affect me in the long run
-* FreeMarker has no additional dependencies and things are just working :-)
-
-While I love Apache Velocity (Apache Turbine anyone?) I decided to give FreeMarker a chance and migrated my [velocity-cli](https://github.com/sgoeschl/velocity-cli) to FreeMarker.
-
-Some years later the not-so-small-any-longer-and-not-having-tests Groovy script was still growing so I decided
-
-* To ditch Groovy and migrate to plain JDK 8
-* Write unit tests since I had no more excuses
-* To ditch Commons CLI and migrate to [Picocli](https://picocli.info)
-
-# 3. Design Goals
-
-* Create a proper command-line tool which has Unix look & feel
-* Handle arbitrary large input and output data
-* Support multiple source files/directories for a single transformation
-* Support transformation of Property files using plain-vanilla JDK
-* Support transformation of CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/)
-* Support transformation of JSON using [Jayway's JSONPath](https://github.com/jayway/JsonPath) and [GSON](https://github.com/google/gson)
-* Support transformation of Excel using [Apache POI](https://poi.apache.org)
-* Support transformation of YAML using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home)
-* Support transformation of HTML using [JSoup](https://jsoup.org)
-* Support transformation of structured logfiles using [Grok](https://github.com/thekrakken/java-grok)
-* XML & XPath is supported by FreeMarker [out-of-the-box](http://freemarker.org/docs/xgui.html)
-* Support for reading a data source content from STDIN to integrate with command line tools
-* Support execution of arbitrary commands using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/)
-* Add some commonly useful information such as `System Properties`, `Enviroment Variables`
-* Support embedding the code in existing applications
-
-# 4. Installation
-
-Download the latest release from [GitHub](https://github.com/sgoeschl/freemarker-cli/releases), e.g. [freemarker-cli-2.0.0-BETA-5-app.tar.gz](https://github.com/sgoeschl/freemarker-cli/releases/download/v2.0.0-BETA-4/freemarker-cli-2.0.0-BETA-5-app.tar.gz) and unpack it into of a directory of your choice, e.g `/Application/Java/freemarker-cli`
-
-It is recommended
-
-* To add `bin/freemarker-cli` or `bin/freemarker-cli.bat` to your executable path
-* To create a `~/.freemarker-cli` directory to store your custom FTL templates
-
-You can test the installation by executing
-
-```text
-> ./bin/freemarker-cli -t templates/info.ftl
-FreeMarker CLI Information
-------------------------------------------------------------------------------
-FreeMarker version : 2.3.30
-Template name : info.ftl
-Language : en
-Locale : en_US
-Timestamp : Jun 18, 2020 10:55:04 AM
-Output encoding : UTF-8
-Output format : plainText
-
-FreeMarker CLI Template Directories
-------------------------------------------------------------------------------
-[#1] /Users/sgoeschl/work/github/apache/freemarker-generator/freemarker-generator-cli/target/appassembler
-[#2] /Users/sgoeschl/.freemarker-cli
-
-FreeMarker CLI Tools
-------------------------------------------------------------------------------
-- CSVTool : Process CSV files using Apache Commons CSV (see https://commons.apache.org/proper/commons-csv/)
-- DataFrameTool : Bridge to nRo/DataFrame (see https://github.com/nRo/DataFrame)
-- ExcelTool : Process Excels files (XLS, XLSX) using Apache POI (see https://poi.apache.org)
-- ExecTool : Execute command line tools using Apache Commons Exec (see https://commons.apache.org/proper/commons-exec/)
-- FreeMarkerTool : Expose useful Apache FreeMarker classes
-- GrokTool : Process text files using Grok expressions (see https://github.com/thekrakken/java-grok)
-- GsonTool : Process JSON files using GSON (see https://github.com/google/gson)
-- JsonPathTool : Process JSON files using Java JSON Path (see https://github.com/json-path/JsonPath)
-- JsoupTool : Process HTML files using Jsoup (see https://jsoup.org)
-- PropertiesTool : Process JDK properties files
-- SystemTool : Expose System-related utility methods
-- UUIDTool : Create UUIDs
-- XmlTool : Process XML files using Apache FreeMarker (see https://freemarker.apache.org/docs/xgui.html)
-- YamlTool : Process YAML files using SnakeYAML(see https://bitbucket.org/asomov/snakeyaml/wiki/Home)
-
-FreeMarker CLI Data Model
----------------------------------------------------------------------------
-- CSVTool
-- DataFrameTool
-- DataSources
-- ExcelTool
-- ExecTool
-- FreeMarkerTool
-- GrokTool
-- GsonTool
-- JsonPathTool
-- JsoupTool
-- PropertiesTool
-- SystemTool
-- UUIDTool
-- XmlTool
-- YamlTool
-```
-
-There a many examples (see below) available you can execute - run `./run-examples.sh` and have a look at the generated output
+There a many examples (see below) available you can execute - The examples were tested with JDK 1.8 on Mac OS X.
+
+Run `run-examples.sh` in the `Apache Freemarker CLI` installation directory and have a look at the generated output.
```text
./run-examples.sh
templates/info.ftl
examples/templates/demo.ftl
templates/csv/html/transform.ftl
-examples/templates/csv/md/transform.ftl
+templates/csv/md/transform.ftl
examples/templates/csv/shell/curl.ftl
examples/templates/csv/md/filter.ftl
examples/templates/csv/fo/transform.ftl
@@ -160,121 +35,60 @@ examples/templates/yaml/txt/transform.ftl
templates/yaml/json/transform.ftl
examples/templates/xml/txt/recipients.ftl
Created the following sample files in ./target/out
-total 1576
--rw-r--r-- 1 sgoeschl staff 646 Jun 18 10:56 combined-access.log.txt
--rw-r--r-- 1 sgoeschl staff 22548 Jun 18 10:56 contract.html
--rw-r--r-- 1 sgoeschl staff 7933 Jun 18 10:56 contract.md
--rw-r--r-- 1 sgoeschl staff 784 Jun 18 10:56 curl.sh
--rw-r--r-- 1 sgoeschl staff 232 Jun 18 10:56 customer.txt
--rw-r--r-- 1 sgoeschl staff 6488 Jun 18 10:56 dataframe.txt
--rw-r--r-- 1 sgoeschl staff 15632 Jun 18 10:56 demo.txt
--rw-r--r-- 1 sgoeschl staff 1310 Jun 18 10:56 dependencies.csv
--rw-r--r-- 1 sgoeschl staff 2029 Jun 18 10:56 github-users-curl.md
--rw-r--r-- 1 sgoeschl staff 2630 Jun 18 10:56 info.txt
--rw-r--r-- 1 sgoeschl staff 8075 Jun 18 10:56 interactive-dataframe.txt
--rw-r--r-- 1 sgoeschl staff 66 Jun 18 10:56 interactive-html.txt
--rw-r--r-- 1 sgoeschl staff 16 Jun 18 10:56 interactive-json.txt
--rw-r--r-- 1 sgoeschl staff 25090 Jun 18 10:56 interactive-swagger.json
--rw-r--r-- 1 sgoeschl staff 16870 Jun 18 10:56 interactive-swagger.yaml
--rw-r--r-- 1 sgoeschl staff 10 Jun 18 10:56 interactive-xml.txt
--rw-r--r-- 1 sgoeschl staff 285 Jun 18 10:56 locker-test-users.csv
--rw-r--r-- 1 sgoeschl staff 6341 Jun 18 10:56 locker-test-users.fo
--rw-r--r-- 1 sgoeschl staff 5526 Jun 18 10:56 locker-test-users.pdf
--rw-r--r-- 1 sgoeschl staff 921 Jun 18 10:56 recipients.txt
--rw-r--r-- 1 sgoeschl staff 910 Jun 18 10:56 sales-records.md
--rw-r--r-- 1 sgoeschl staff 2453 Jun 18 10:56 swagger-spec.csv
--rw-r--r-- 1 sgoeschl staff 25090 Jun 18 10:56 swagger-spec.json
--rw-r--r-- 1 sgoeschl staff 16870 Jun 18 10:56 swagger-spec.yaml
-drwxr-xr-x 4 sgoeschl staff 128 Jun 18 10:49 template
--rw-r--r-- 1 sgoeschl staff 156 Jun 18 10:56 test-multiple-sheets.xlsx.csv
--rw-r--r-- 1 sgoeschl staff 1917 Jun 18 10:56 test-multiple-sheets.xlsx.html
--rw-r--r-- 1 sgoeschl staff 389 Jun 18 10:56 test-multiple-sheets.xlsx.md
--rw-r--r-- 1 sgoeschl staff 157 Jun 18 10:56 test-transform-xls.csv
--rw-r--r-- 1 sgoeschl staff 1439 Jun 18 10:56 test.xls.dataframe.txt
--rw-r--r-- 1 sgoeschl staff 1556 Jun 18 10:56 test.xls.html
--rw-r--r-- 1 sgoeschl staff 1558 Jun 18 10:56 test.xslx.html
--rw-r--r-- 1 sgoeschl staff 25760 Jun 18 10:56 transactions-fo.pdf
--rw-r--r-- 1 sgoeschl staff 66016 Jun 18 10:56 transactions-html.pdf
--rw-r--r-- 1 sgoeschl staff 330129 Jun 18 10:56 transactions.fo
--rw-r--r-- 1 sgoeschl staff 51008 Jun 18 10:56 transactions.html
-
+total 1464
+-rw-r--r-- 1 sgoeschl staff 646 Jun 27 16:38 combined-access.log.txt
+-rw-r--r-- 1 sgoeschl staff 25676 Jun 27 16:38 contract.html
+-rw-r--r-- 1 sgoeschl staff 7933 Jun 27 16:38 contract.md
+-rw-r--r-- 1 sgoeschl staff 784 Jun 27 16:38 curl.sh
+-rw-r--r-- 1 sgoeschl staff 232 Jun 27 16:38 customer.txt
+-rw-r--r-- 1 sgoeschl staff 6486 Jun 27 16:38 dataframe.txt
+-rw-r--r-- 1 sgoeschl staff 15613 Jun 27 16:38 demo.txt
+-rw-r--r-- 1 sgoeschl staff 1310 Jun 27 16:38 dependencies.csv
+-rw-r--r-- 1 sgoeschl staff 2029 Jun 27 16:38 github-users-curl.md
+-rw-r--r-- 1 sgoeschl staff 2627 Jun 27 16:38 info.txt
+-rw-r--r-- 1 sgoeschl staff 8075 Jun 27 16:38 interactive-dataframe.txt
+-rw-r--r-- 1 sgoeschl staff 66 Jun 27 16:38 interactive-html.txt
+-rw-r--r-- 1 sgoeschl staff 16 Jun 27 16:38 interactive-json.txt
+-rw-r--r-- 1 sgoeschl staff 25090 Jun 27 16:38 interactive-swagger.json
+-rw-r--r-- 1 sgoeschl staff 16870 Jun 27 16:38 interactive-swagger.yaml
+-rw-r--r-- 1 sgoeschl staff 10 Jun 27 16:38 interactive-xml.txt
+-rw-r--r-- 1 sgoeschl staff 285 Jun 27 16:38 locker-test-users.csv
+-rw-r--r-- 1 sgoeschl staff 6341 Jun 27 16:38 locker-test-users.fo
+-rw-r--r-- 1 sgoeschl staff 5526 Jun 27 16:38 locker-test-users.pdf
+-rw-r--r-- 1 sgoeschl staff 921 Jun 27 16:38 recipients.txt
+-rw-r--r-- 1 sgoeschl staff 910 Jun 27 16:38 sales-records.md
+-rw-r--r-- 1 sgoeschl staff 2453 Jun 27 16:38 swagger-spec.csv
+-rw-r--r-- 1 sgoeschl staff 25090 Jun 27 16:38 swagger-spec.json
+-rw-r--r-- 1 sgoeschl staff 16870 Jun 27 16:38 swagger-spec.yaml
+drwxr-xr-x 4 sgoeschl staff 128 Jun 27 16:38 template
+-rw-r--r-- 1 sgoeschl staff 154 Jun 27 16:38 test-multiple-sheets.xlsx.csv
+-rw-r--r-- 1 sgoeschl staff 1917 Jun 27 16:38 test-multiple-sheets.xlsx.html
+-rw-r--r-- 1 sgoeschl staff 389 Jun 27 16:38 test-multiple-sheets.xlsx.md
+-rw-r--r-- 1 sgoeschl staff 155 Jun 27 16:38 test-transform-xls.csv
+-rw-r--r-- 1 sgoeschl staff 1439 Jun 27 16:38 test.xls.dataframe.txt
+-rw-r--r-- 1 sgoeschl staff 1556 Jun 27 16:38 test.xls.html
+-rw-r--r-- 1 sgoeschl staff 1558 Jun 27 16:38 test.xslx.html
+-rw-r--r-- 1 sgoeschl staff 25757 Jun 27 16:38 transactions-fo.pdf
+-rw-r--r-- 1 sgoeschl staff 66016 Jun 27 16:38 transactions-html.pdf
+-rw-r--r-- 1 sgoeschl staff 330128 Jun 27 16:38 transactions.fo
+-rw-r--r-- 1 sgoeschl staff 51008 Jun 27 16:38 transactions.html
```
Please note that generated PDF files are very likely not found since they require `wkhtmltopdf` and `Apache FOP` installation.
-# 5. Usage
-
-```text
-> ./bin/freemarker-cli -h
-Usage: freemarker-cli (-t=<templates> [-t=<templates>]... |
- -i=<interactiveTemplate>) [-hV] [--stdin] [-b=<baseDir>]
- [--config=<configFile>] [--data-source-exclude=<exclude>]
- [--data-source-include=<include>] [-e=<inputEncoding>]
- [-l=<locale>] [-o=<outputFile>]
- [--output-encoding=<outputEncoding>] [--times=<times>]
- [-D=<String=String>]... [-m=<dataModels>]...
- [-P=<String=String>]... [-s=<dataSources>]...
- [<sources>...]
-Apache FreeMarker CLI
- [<sources>...] data source files and/or directories
- -b, --basedir=<baseDir> optional template base directory
- --config=<configFile> FreeMarker CLI configuration file
- -D, --system-property=<String=String>
- set system property
- --data-source-exclude=<exclude>
- file exclude pattern for data sources
- --data-source-include=<include>
- file include pattern for data sources
- -e, --input-encoding=<inputEncoding>
- encoding of data source
- -h, --help Show this help message and exit.
- -i, --interactive=<interactiveTemplate>
- interactive template to process
- -l, --locale=<locale> locale being used for the output, e.g. 'en_US'
- -m, --data-model=<dataModels>
- data model used for rendering
- -o, --output=<outputFile> output file or directory
- --output-encoding=<outputEncoding>
- encoding of output, e.g. UTF-8
- -P, --param=<String=String>
- set parameter
- -s, --data-source=<dataSources>
- data source used for redering
- --stdin read data source from stdin
- -t, --template=<templates>
- template to process
- --times=<times> re-run X times for profiling
- -V, --version Print version information and exit.
-
-```
-
-# 6. Examples
-
-The examples were tested with JDK 1.8 on Mac OS X
-
-```text
-> java -version
-java version "1.8.0_192"
-Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
-Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
-```
-It is assumed that you run the examples from the `freemarker-cli` installation directory.
-
-## 6.1 Transforming GitHub JSON To Markdown
+### 1. Transforming GitHub JSON To Markdown
A simple example with real JSON data to be transformed into Markdown
-### Invocation
-
You can either use the existing JSON sample
-> ./bin/freemarker-cli -t examples/templates/json/md/github-users.ftl examples/data/json/github-users.json
+> freemarker-cli -t examples/templates/json/md/github-users.ftl examples/data/json/github-users.json
or pipe a cURL response
-> curl -s https://api.github.com/users | ./bin/freemarker-cli -t examples/templates/json/md/github-users.ftl --stdin
+> curl -s https://api.github.com/users | freemarker-cli -t examples/templates/json/md/github-users.ftl --stdin
-### FreeMarker Template
+Below you see the Apache FreeMarker Template
```text
<#ftl output_format="plainText" >
@@ -300,15 +114,15 @@ Report generated at ${.now?iso_utc}
creates the following output
-
+
-## 6.2 CSV to HTML/Markdown Transformation
+### 2. CSV to HTML/Markdown Transformation
Sometimes you have a CSV file which needs to be translated in Markdown or HTML - there are on-line solutions available such as [CSV To Markdown Table Generator](https://donatstudios.com/CsvToMarkdownTable) but having a local solution gives you more flexibility.
```text
-> ./bin/freemarker-cli -t templates/csv/md/transform.ftl examples/data/csv/contract.csv
-> ./bin/freemarker-cli -t templates/csv/html/transform.ftl examples/data/csv/contract.csv
+> freemarker-cli -t templates/csv/md/transform.ftl examples/data/csv/contract.csv
+> freemarker-cli -t templates/csv/html/transform.ftl examples/data/csv/contract.csv
```
The FreeMarker template is shown below
@@ -339,14 +153,14 @@ The FreeMarker template is shown below
The resulting file actually looks pleasant when compared to raw CSV
-
+
-## 6.3 Transform XML To Plain Text
+### 3. Transform XML To Plain Text
Of course you can also transform a XML document
```text
-> ./bin/freemarker-cli -t examples/templates/xml/txt/recipients.ftl examples/data/xml/recipients.xml
+> freemarker-cli -t examples/templates/xml/txt/recipients.ftl examples/data/xml/recipients.xml
```
using the following template
@@ -394,7 +208,7 @@ Sincere salutations,
D. H.
```
-## 6.4 Transform JSON To CSV
+### 4. Transform JSON To CSV
One day I was asked a to prepare a CSV files containing REST endpoints described by Swagger - technically this is a JSON to CSV transformation. Of course I could create that CSV manually but writing a FTL template doing that was simply more fun and saves time in the future.
@@ -435,7 +249,7 @@ ${'\n'}
Invoking the FTL template
-> ./bin/freemarker-cli -t examples/templates/json/csv/swagger-endpoints.ftl examples/data/json/swagger-spec.json
+> freemarker-cli -t examples/templates/json/csv/swagger-endpoints.ftl examples/data/json/swagger-spec.json
gives you
@@ -447,15 +261,15 @@ ENDPOINT;METHOD;CONSUMES;PRODUCES;SUMMARY;DESCRIPTION
/api/pets/{id};DELETE;;;;Deletes a single pet based on the ID supplied
```
-## 6.5 Transforming Excel Documents
+### 5. Transforming Excel Documents
Another day my project management asked me to create a CSV configuration file based on an Excel documents - as usual manual copying was not an option due to required data cleanup and data transformation. So I thought about Apache POI which support XLS and XLSX documents - integration of Apache POI was a breeze but the resulting code was not particularly useful example. So a more generic transformation was provided to show the transformation of Excel documents ...
```text
-> ./bin/freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test.xls
-> ./bin/freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test.xlsx
-> ./bin/freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test-multiple-sheets.xlsx
-> ./bin/freemarker-cli -t templates/excel/md/transform.ftl examples/data/excel/test-multiple-sheets.xlsx
+> freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test.xls
+> freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test.xlsx
+> freemarker-cli -t templates/excel/html/transform.ftl examples/data/excel/test-multiple-sheets.xlsx
+> freemarker-cli -t templates/excel/md/transform.ftl examples/data/excel/test-multiple-sheets.xlsx
```
The provided FTL transforms an Excel into a HTML document supporting multiple Excel sheets
@@ -532,14 +346,14 @@ The provided FTL transforms an Excel into a HTML document supporting multiple Ex
but the result looks reasonable
-
+
-## 6.6 Transform Property Files To CSV
+### 6. Transform Property Files To CSV
In this sample we transform all property files found in a directory (recursive search using include pattern) to a CSV file
```text
-> ./bin/freemarker-cli --data-source-include *.properties -t examples/templates/properties/csv/locker-test-users.ftl examples/data/properties
+> freemarker-cli --data-source-include *.properties -t examples/templates/properties/csv/locker-test-users.ftl examples/data/properties
TENANT,SITE,USER_ID,DISPOSER_ID,PASSWORD,SMS_OTP,NAME,DESCRIPTION
TENANT_A,fat,user_0004,user_0004,password_0004,,,
TENANT_B,fat,user_0001,user_0001,password_0001,,,
@@ -582,7 +396,7 @@ ${'\n'}
```
-## 6.7 Transform CSV To XML-FO
+### 7. Transform CSV To XML-FO
For a POC (proof of concept) I created a sample transformation from CSV to XML-FO in order to create a PDF document using [Apache FOP](https://xmlgraphics.apache.org/fop) using the following template file
@@ -670,7 +484,7 @@ For a POC (proof of concept) I created a sample transformation from CSV to XML-F
In order to create the PDF you need to execute the following commands (assuming that you have Apache FOP installed)
```text
-> ./bin/freemarker-cli -t examples/templates/csv/fo/transform.ftl examples/data/csv/locker-test-users.csv > sample.fo
+> freemarker-cli -t examples/templates/csv/fo/transform.ftl examples/data/csv/locker-test-users.csv > sample.fo
> fop -fo sample.fo sample.pdf
Dec 29, 2018 10:24:30 PM org.apache.fop.events.LoggingEventListener processEvent
WARNING: Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400".
@@ -682,12 +496,12 @@ INFO: Rendered page #1.
The result does not look very impressive but it is a PDF :-)
-
+
Further along the line of the POC we converted a transaction export from CSV to PDF using Apache FOP
```text
-> ./bin/freemarker-cli -t examples/templates/csv/fo/transactions.ftl examples/data/csv/transactions.csv > transactions.fo
+> freemarker-cli -t examples/templates/csv/fo/transactions.ftl examples/data/csv/transactions.csv > transactions.fo
> fop -fo transactions.fo transactions.pdf
Jan 16, 2019 11:15:21 PM org.apache.fop.events.LoggingEventListener processEvent
WARNING: Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400".
@@ -701,9 +515,9 @@ Jan 16, 2019 11:15:22 PM org.apache.fop.events.LoggingEventListener processEvent
INFO: Rendered page #2.
```
-
+
-## 6.8 Transforming HTML To CSV
+### 8. Transforming HTML To CSV
Recently I got the rather unusual question how to determine the list of dependencies of an application - one easy way is the Maven "dependencies.html" but this is unstructured data. Having said that the Jsoup library is perfectly able to parse most real-life HTML and provides a DOM model
@@ -748,7 +562,7 @@ Recently I got the rather unusual question how to determine the list of dependen
Your dependencies as CSV can be generated as shown below
```text
-> ./bin/freemarker-cli -t examples/templates/html/csv/dependencies.ftl examples/data/html/dependencies.html
+> freemarker-cli -t examples/templates/html/csv/dependencies.ftl examples/data/html/dependencies.html
GroupId,ArtifactId,Version,Type,Licenses
com.jayway.jsonpath,json-path,2.4.0,jar,The Apache Software License Version 2.0
commons-cli,commons-cli,1.4,jar,Apache License Version 2.0
@@ -772,7 +586,7 @@ org.ow2.asm,asm,5.0.4,jar,BSD
stax,stax-api,1.0.1,jar,The Apache Software License Version 2.0
```
-## 6.9 Transform CSV To Shell Script
+### 9. Transform CSV To Shell Script
For a customer project we wanted to record REST request / responses using WireMock - really quick and dirty. So we decided to avoid any sophisticated test tool but generate a ready-to-use shell script executing cURL commands. It turned out that handling of dollar signs is a bit tricky
@@ -803,7 +617,7 @@ date "+%FT%H:%M:%S" | tr -d '\n'; curl --write-out ',${record.disposer},%{http_c
Rendering the FreeMarker template
```
-> ./bin/freemarker-cli -t ./examples/templates/csv/shell/curl.ftl examples/data/csv/user.csv
+> freemarker-cli -t ./examples/templates/csv/shell/curl.ftl examples/data/csv/user.csv
```
generates the following shell script
@@ -835,7 +649,7 @@ time,user,status,duration,size
2019-09-27T21:02:54,DDDDDDD,200,0.528268,206
```
-## 6.10 Unleashing The Power Of Grok
+### 10. Unleashing The Power Of Grok
Think of `Grok` as modular regular expressions with a pre-defined functionality to parse access logs or any other data where you can't comprehend the regular expression any longer, one very simple example is `QUOTEDSTRING`
@@ -890,46 +704,7 @@ While this looks small and tidy there are some nifty features
* The data source is streamed line by line and not loaded into memory in one piece
* This also works for using `stdin` so are able to parse GB of access log or other files
-## 6.11 CSV Transformation
-
-Sometimes you have a CSV file which is not quite right - you need to change the format. Lets have a look how `freemarker-cli` can help
-
-> bin/freemarker-cli -PCVS_IN_DELIMITER=COMMA -PCSV_TARGET_DELIMITER=PIPE -t templates/csv/csv/transform.ftl ./examples/data/csv/contract.csv
-
-renders the following template
-
-```text
-<#ftl output_format="plainText" strip_text="true">
-<#assign csvParser = CSVTool.parse(DataSources.get(0))>
-<#assign csvPrinter = CSVTool.printer(SystemTool.writer)>
-<#--
- Print each record directly to the underyling writer without materializing the CSV in memory.
- FreeMarker and CSV output are out of sync but millions of records can processed without
- running out of memory.
--->
-<#compress>
- <#list csvParser.iterator() as record>
- ${csvPrinter.printRecord(record)}
- </#list>
-</#compress>
-```
-
-and generates the following output
-
-```text
-contract_id|seller_company_name|customer_company_name|customer_duns_number|contract_affiliate|FERC_tariff_reference|contract_service_agreement_id|contract_execution_date|contract_commencement_date|contract_termination_date|actual_termination_date|extension_provision_description|class_name|term_name|increment_name|increment_peaking_name|product_type_name|product_name|quantity|units_for_contract|rate|rate_minimum|rate_maximum|rate_description|units_for_rate|point_of_receipt_control_area|po [...]
-C71|The Electric Company|The Power Company|456543333|N|FERC Electric Tariff Original Volume No. 10|2|2/15/2001|2/15/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||||||ES
-C72|The Electric Company|Utility A|38495837|n|FERC Electric Tariff Original Volume No. 10|15|7/25/2001|8/1/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||||||ES
-C73|The Electric Company|Utility B|493758794|N|FERC Electric Tariff Original Volume No. 10|7|6/8/2001|7/6/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||" "|" "|||ep
-C74|The Electric Company|Utility C|594739573|n|FERC Electric Tariff Original Volume No. 10|25|6/8/2001|7/6/2001|||Evergreen|N/A|N/A|N/A|N/A|MB|ENERGY|0||" "|" "|" "|Market Based||||" "|" "|||ep
-```
-
-Some useful hints
-
-* For available CSV formats please see [Apache Commons CSV User Guide](http://commons.apache.org/proper/commons-csv/user-guide.html)
-* Stripping the Excel BOM (Byte Order Mark) works out-of-box
-
-## 6.12 Executing Arbitrary Commands
+### 11. Executing Arbitrary Commands
Using Apache Commons Exec allows to execute arbitrary commands - nice but dangerous. It was recently quite useful to to invoke AWS CLI to generate a Confluence page about the overall setup of our AWS accounts.
@@ -985,7 +760,7 @@ h3. AWS EC2 Instance
</#macro>
```
-## 6.13 Interactive Templates
+## 12. Interactive Templates
Sometime you need to apply a CSS, JSON or XPath query in ad ad-hoc way without installing `xmllint`, `jq` or `pup` - in this case you can pass a FreeMarker template in an interactive fashion
@@ -1005,7 +780,7 @@ John Smith
> bin/freemarker-cli -i '${JsoupTool.parse(DataSources.first).select("a")[0]}' examples/data/html/dependencies.html; echo
<a href="${project.url}" title="FreeMarker CLI">FreeMarker CLI</a>
-> ./bin/freemarker-cli -i '<#list SystemTool.envs as name,value>${name} ==> ${value}${"\n"}</#list>'
+> freemarker-cli -i '<#list SystemTool.envs as name,value>${name} ==> ${value}${"\n"}</#list>'
TERM ==> xterm-256color
LANG ==> en_US
DISPLAY ==> :0.0
@@ -1013,7 +788,7 @@ SHELL ==> /bin/bash
EDITOR ==> vi
```
-## 6.14 Filtering & Transforming CSV
+## 13. Filtering & Transforming CSV
During an integration project we imported large transactions CSV files (500.000+ records) and in case of import failures the developers would be happy to get a nice outline of the transactions causing the problem (the CSV records have 60+ columns) - in essence it is filtering (based on some primary key) and and transforming into a human-readable output format (Markdown).
@@ -1109,214 +884,21 @@ yields
| Total Profit | 498855.44 |
```
-## 6.15 Converting Between JSON And YAML
+### 14. Converting Between JSON And YAML
Sometimes we simply need to transform a JSON into an equivalent YAML or the other way around
```
-> ./bin/freemarker-cli -t templates/yaml/json/transform.ftl examples/data/yaml/swagger-spec.yaml
-> ./bin/freemarker-cli -i '${GsonTool.toJson(YamlTool.parse(DataSources.get(0)))}' examples/data/yaml/swagger-spec.yaml
-> ./bin/freemarker-cli -i '${GsonTool.toJson(yaml)}' -m yaml=examples/data/yaml/swagger-spec.yaml
-
-> ./bin/freemarker-cli -t templates/json/yaml/transform.ftl examples/data/json/swagger-spec.json
-> ./bin/freemarker-cli -i '${YamlTool.toYaml(GsonTool.parse(DataSources.get(0)))}' examples/data/json/swagger-spec.json
-> ./bin/freemarker-cli -i '${YamlTool.toYaml(json)}' -m json=examples/data/json/swagger-spec.json
-```
-
-## 6.16 Using DataFrames
-
-The `DataFrameTool` uses [nRo/DataFrame](https://github.com/nRo/DataFrame) to convert tabular data into a `DataFrame`.
-
-A `DataFrame` allows declartive filtering and transformation of tabular data, i.e. less code to write.
-
-Currently the following sources are supported
-
-* Apache Commons CSV Parser
-* JSON arrays represented as collection of maps
-* Excel sheets represented as rows
-
-### CSV Examples
-
-[nRo/DataFrame]("https://raw.githubusercontent.com/nRo/DataFrame/master/src/test/resources/users.csv") provides the following CSV file
+> freemarker-cli -t templates/yaml/json/transform.ftl examples/data/yaml/swagger-spec.yaml
+> freemarker-cli -i '${GsonTool.toJson(YamlTool.parse(DataSources.get(0)))}' examples/data/yaml/swagger-spec.yaml
+> freemarker-cli -i '${GsonTool.toJson(yaml)}' -m yaml=examples/data/yaml/swagger-spec.yaml
+> freemarker-cli -t templates/json/yaml/transform.ftl examples/data/json/swagger-spec.json
+> freemarker-cli -i '${YamlTool.toYaml(GsonTool.parse(DataSources.get(0)))}' examples/data/json/swagger-spec.json
+> freemarker-cli -i '${YamlTool.toYaml(json)}' -m json=examples/data/json/swagger-spec.json
```
-name;age;country
-Schmitt;24;Germany
-Parker;45;USA
-Meier;20;Germany
-Schmitt;30;France
-Peter;44;Germany
-Meier;24;Germany
-Green;33;UK
-Schmitt;30;Germany
-Meier;30;Germany
-```
-
-and create a `DateFrame` using the following code
-
-```
-<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader().withDelimiter(';')>
-<#assign csvParser = CSVTool.parse(DataSources.get(0), cvsFormat)>
-<#assign users = DataFrameTool.toDataFrame(csvParser)>
-```
-
-#### Select By Age
-
-```
-${DataFrameTool.print(users.select("(age > 40)"))}
-```
-
-which shows
-
-```
-┌────────────┬────────────┬────────────┐
-│#name │#age │#country │
-├────────────┼────────────┼────────────┤
-│Parker │45 │USA │
-├────────────┼────────────┼────────────┤
-│Peter │44 │Germany │
-└────────────┴────────────┴────────────┘
-```
-
-#### Complex Select & Sort
-
-Now we want to create a new `DataFrame` by selecting `name` and `country`
-
-```
-<#assign country = "Germany">
-${DataFrameTool.print(users
- .select("(name == 'Schmitt' || name == 'Meier') && country == '${country}'")
- .sort("name", DataFrameTool.sortOrder["ASCENDING"]))}
-```
-
-which shows
-
-```
-┌────────────┬────────────┬────────────┐
-│#name │#age │#country │
-├────────────┼────────────┼────────────┤
-│Meier │20 │Germany │
-├────────────┼────────────┼────────────┤
-│Meier │24 │Germany │
-├────────────┼────────────┼────────────┤
-│Meier │30 │Germany │
-├────────────┼────────────┼────────────┤
-│Schmitt │24 │Germany │
-├────────────┼────────────┼────────────┤
-│Schmitt │30 │Germany │
-└────────────┴────────────┴────────────┘
-```
-
-#### Count Column Values
-
-Let's assume we want to count the records for each `country`
-
-```
-${DataFrameTool.print(users.getColumn("country").transform(DataFrameTool.transformer["COUNT"]))}
-```
-
-returns the following `DataFrame`
-
-```
-┌────────────┬────────────┐
-│#country │#counts │
-├────────────┼────────────┤
-│Germany │6 │
-├────────────┼────────────┤
-│USA │1 │
-├────────────┼────────────┤
-│France │1 │
-├────────────┼────────────┤
-│UK │1 │
-└────────────┴────────────┘
-```
-
-#### Group By Age And Country
-Let's assume that we want to group the `DataFrame` by `age` and `country`
-
-```
-${DataFrameTool.print(users.groupBy("age", "country").sort("age"))}
-```
-
-which results in
-
-```
-┌────────────┬────────────┐
-│#age │#country │
-├────────────┼────────────┤
-│20 │Germany │
-├────────────┼────────────┤
-│24 │Germany │
-├────────────┼────────────┤
-│30 │France │
-├────────────┼────────────┤
-│30 │Germany │
-├────────────┼────────────┤
-│33 │UK │
-├────────────┼────────────┤
-│44 │Germany │
-├────────────┼────────────┤
-│45 │USA │
-└────────────┴────────────┘
-```
-
-### JSON Examples
-
-Here we load a `examples/data/json/github-users.json` which represents a tabular data be
-being parsed as a list of maps and print the JSOB as dataframe
-
-```
-./bin/freemarker-cli -i '${DataFrameTool.print(DataFrameTool.fromMaps(GsonTool.parse(DataSources.get(0))))}' examples/data/json/github-users.json
-
-┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
-│#login │#id │#avatar_ur │#gravatar_ │#url │#html_url │#followers │#following │#gists_url │#starred_u │#subscript │#organizat │#repos_url │#events_ur │#received_ │#type │#site_admi │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│mojombo │1.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│defunkt │2.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │true │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│pjhyett │3.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │true │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│wycats │4.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│ezmobius │5.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│ivey │6.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│evanphx │7.00000000 │https:/... │ │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │https:/... │User │false │
-└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
-```
-
-### Excel Examples
-
-Let's transform an Excel Sheet to a `DataFrame` being printed using the following template
-
-```
-<#assign dataSource = DataSources.get(0)>
-<#assign workbook = ExcelTool.parse(dataSource)>
-<#list ExcelTool.getSheets(workbook) as sheet>
- <#assign table = ExcelTool.toTable(sheet)>
- <#assign df = DataFrameTool.fromRows(table, true)>
- ${DataFrameTool.print(df)}<#t>
-</#list>
-```
-
-which is rendered by the following command line invocation
-
-```
-./bin/freemarker-cli -t examples/templates/excel/dataframe/transform.ftl examples/data/excel/test.xls
-
-┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
-│#Text │#Date │#Number │#Currency │#Time │#Percentag │#Forumula │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│Row 1 │01/01/17 │100.00 │€100.00 │10:00 │50.00% │C2*F2 │
-├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
-│Row 2 │01/01/17 │100.00 │€100.00 │10:00 │50.00% │C3*F3 │
-└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
-```
-
-## 6.17 Using Advanced FreeMarker Features
+### 15. Using Advanced FreeMarker Features
There is a `demo.ftl` which shows some advanced FreeMarker functionality
@@ -1328,7 +910,7 @@ There is a `demo.ftl` which shows some advanced FreeMarker functionality
Running
-> ./bin/freemarker-cli -t examples/templates/demo.ftl
+> freemarker-cli -t examples/templates/demo.ftl
gives you
@@ -1506,56 +1088,4 @@ Time : 11:13:51 AM
---------------------------------------------------------------------------
> date
Thu Jun 18 11:13:51 CEST 2020
-```
-
-# 7. Design Considerations
-
-## 7.1 How It Works
-
-* The user-supplied files are loaded or input is read from `stdin`
-* The FreeMarker data model containing the data sources and tools is created and passed to the template
-* The generated output is written to the user-supplied file or to `stdout`
-
-## 7.2 FreeMarker Data Model
-
-Within the script a FreeMarker data model is set up and passed to the template - it contains the data sources to be processed and the following tools
-
-| Entry | Description |
-|-----------------------|-----------------------------------------------------------------------------------------------------------|
-| CSVTool | Process CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/) |
-| DataFrameTool | Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame) |
-| ExecTool | Execute command line tools using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/) |
-| ExcelTool | Process Excels files (XLS, XLSX) using [Apache POI](https://poi.apache.org) |
-| DataSources | Helper class to find data sources, e.g. by name, extension or index |
-| FreeMarkerTool | Expose useful FreeMarker classes |
-| GrokTool | Process text files using [Grok](https://github.com/thekrakken/java-grok) instead of regular expressions |
-| GsonTool | Process JSON files using [GSON](https://github.com/google/gson) |
-| JsonPathTool | Process JSON file using [Java JSON Path](https://github.com/json-path/JsonPath) |
-| JsoupTool | Processing HTML files using [Jsoup](https://jsoup.org) |
-| PropertiesTool | Process JDK properties files |
-| SystemTool | System-related utility methods |
-| UUIDTool | Create UUIDs |
-| XmlTool | Process XML files using [Apache FreeMarker](https://freemarker.apache.org/docs/xgui.html) |
-| YamlTool | Process YAML files using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home) |
-
-
-# 8. Tips & Tricks
-
-## 8.1 Template Base Directory
-
-When doing some ad-hoc scripting it is useful to rely on a base directory to resolve the FTL templates
-
-* As a default the FTL templates are resolved relative to the script directory
-* The caller can provide a `-b` or `--basedir` command line parameter
-
-```text
-./bin/freemarker-cli -b examples/templates -t json/md/github-users.ftl examples/data/json/github-users.json; echo
-```
-
-## 8.2 Using Pipes
-
-When doing ad-hoc scripting it useful to pipe the output of one command directly into "freemarker-cli"
-
-```text
-cat examples/data/json/github-users.json | ./bin/freemarker-cli -t examples/templates/json/md/github-users.ftl --stdin; echo
-```
+```
\ No newline at end of file
diff --git a/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-csv.md b/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-csv.md
new file mode 100644
index 0000000..e6c056d
--- /dev/null
+++ b/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-csv.md
@@ -0,0 +1,80 @@
+## Transforming CSV
+
+A common task is changing the output format of a CSV file therefore `Apache FreeMarker CLI` ships with a ready-to-use
+templates to convert CSVs
+
+* Convert a CSV into a different format
+* Convert a CSV into Markdown or HTML
+
+### Convert CSV To A Different Output Format
+
+Let's assume that a CSV file in [DEFAULT format](https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVFormat.html#DEFAULT)
+should be converted into a [Microsoft Excel format](https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVFormat.html#EXCEL).
+This allows opening the CSV directly in Excel without going to the tedious CSV import dialog.
+
+The following command line prints the converted CSV to `stdout`
+
+```
+freemarker-cli \
+ -PCSV_SOURCE_FORMAT=DEFAULT \
+ -PCSV_TARGET_FORMAT=EXCEL \
+ -PCSV_TARGET_DELIMITER=SEMICOLON \
+ -t templates/csv/csv/transform.ftl \
+ https://raw.githubusercontent.com/apache/freemarker-generator/master/freemarker-generator-cli/examples/data/csv/contract.csv
+```
+
+The command line invocation seems a bit complex at first so let's look at it more closely
+
+* `CSV_SOURCE_FORMAT` defines the CSV source format for reading the CSV
+* `CSV_TARGET_FORMAT` defines the CSV tagrte format for writing the CSV
+* `CSV_TARGET_DELIMITER` explicitely sets the delimiter of the target CSV to a semicolon since this expected by Excel for my current locale
+
+### Convert CSV To Markdown
+
+When doing documentation it is sometimes helpful to convert a CSV into its Markdown representation (AsciiDoc
+actually allows including CSV directly)
+
+The following command line prints the resulting MarkDown to `stdout`
+
+```
+freemarker-cli \
+ -PCSV_SOURCE_FORMAT=DEFAULT \
+ -t templates/csv/md/transform.ftl \
+ https://raw.githubusercontent.com/apache/freemarker-generator/master/freemarker-generator-cli/examples/data/csv/contract.csv
+```
+
+Please note that most MarkDown tools expect a header row (see [Create a table without a header in Markdown](https://stackoverflow.com/questions/17536216/create-a-table-without-a-header-in-markdown))
+
+### Convert CSV To HTML
+
+Of course it is possible to convert a CSV to HTML as well
+
+```
+freemarker-cli \
+ -PCSV_SOURCE_FORMAT=DEFAULT \
+ -t templates/csv/html/transform.ftl \
+ https://raw.githubusercontent.com/apache/freemarker-generator/master/freemarker-generator-cli/examples/data/csv/contract.csv
+```
+
+### CSV Configuration Options
+
+The following options can be passed to template (as user-supplied parameters)
+
+| Parameter | Default Value | Description |
+|---------------------------|-------------------|-----------------------------------------------------------|
+| CSV_SOURCE_FORMAT | DEFAULT | Source CSV format |
+| CSV_SOURCE_DELIMITER | COMMA | Symbolic name of delimiter, e.g. "COLON" or "SEMICOLON" |
+| CSV_SOURCE_WITH_HEADER | true | Whether the first rows are headers |
+| CSV_TAGRGET_FORMAT | DEFAULT | Target CSV format |
+| CSV_TARGET_DELIMITER | COMMA | Symbolic name of delimiter, e.g. "COLON" or "SEMICOLON" |
+| CSV_TARGET_WITH_HEADER | true | Whether the first rows are headers |
+
+The delimiters are passed as symbolic names since passing seperators breaks command line handling
+
+* COMMA
+* HASH
+* PIPE
+* RS (ASCII record separator)
+* SEMICOLON
+* SPACE
+* TAB
diff --git a/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-directories.md b/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-directories.md
index 9f6a36f..ad694fb 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-directories.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/usage/transforming-directories.md
@@ -1,11 +1,11 @@
## Transforming Directories
-FreeMarker CLI supports the transformation of directories
+`Apache FreeMarker CLI` supports the transformation of directories
* Transform an input directory recursively into an output directory
* If a template has a ".ftl" extension this extension will be removed after processing
-* Only a single directory is support
-* Currently no inclusion / exclusion patterns for templates are supported
+* Only a single directory is supported (for now)
+* Currently no inclusion / exclusion patterns for templates are supported (for now)
The following sample files are used
@@ -13,7 +13,7 @@ The following sample files are used
* template/nginx/nginx.conf.ftl
```
-appassembler> tree examples/data/template/
+tree examples/data/template/
examples/data/template/
|-- application.properties
`-- nginx
@@ -39,7 +39,7 @@ server {
If no output directory is provided all output is written to `stdout`
```
-bin/freemarker-cli -t examples/data/template/
+freemarker-cli -t examples/data/template/
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -60,7 +60,7 @@ The transformed templates are written to an `out` directory
* `nginx.conf.ftl` was changed to `nginx.conf" during the transformation
```
-bin/freemarker-cli -t examples/data/template/ -o out; tree out
+freemarker-cli -t examples/data/template/ -o out; tree out
out
|-- application.properties
`-- nginx
@@ -74,7 +74,7 @@ out
A user-supplied parameter `NGINX_HOSTNAME` is used to render the templates
```
-bin/freemarker-cli -t examples/data/template/ -P NGINX_HOSTNAME=localhost
+freemarker-cli -t examples/data/template/ -P NGINX_HOSTNAME=localhost
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -97,7 +97,7 @@ All environment variables can be copied to the top-level data model by providing
```
export NGINX_PORT=8080
-bin/freemarker-cli -t examples/data/template/ -m env:///
+freemarker-cli -t examples/data/template/ -m env:///
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -117,7 +117,7 @@ Instead of environment variables an environment file (aka properties file) can b
```
echo "NGINX_PORT=8080" > nginx.env
-bin/freemarker-cli -t examples/data/template/ -m nginx.env
+freemarker-cli -t examples/data/template/ -m nginx.env
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -137,7 +137,7 @@ Another option is passing the information as JSON file
```
echo '{"NGINX_PORT":"8443","NGINX_HOSTNAME":"localhost"}' > nginx.json
-bin/freemarker-cli -t examples/data/template/ -m nginx.json
+freemarker-cli -t examples/data/template/ -m nginx.json
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -157,7 +157,7 @@ Yet another option is using a YAML file
```
echo -e "- NGINX_PORT": "\"8443\"\n- NGINX_HOSTNAME": "localhost" > nginx.yaml
-bin/freemarker-cli -t examples/data/template/ -m nginx.yaml
+freemarker-cli -t examples/data/template/ -m nginx.yaml
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -180,7 +180,7 @@ In the cloud it is common to pass JSON configuration as environment variable
```
export NGINX_CONF='{"NGINX_PORT":"8443","NGINX_HOSTNAME":"localhost"}'
-bin/freemarker-cli -t examples/data/template/ -m env:///NGINX_CONF#mimetype=application/json
+freemarker-cli -t examples/data/template/ -m env:///NGINX_CONF#mimetype=application/json
# == application.properties ==================================================
server.name=localhost
server.logs=/var/log/nginx
@@ -200,7 +200,7 @@ For testing purpose it is useful to override certain settings
```
export NGINX_CONF='{"NGINX_PORT":"8443","NGINX_HOSTNAME":"localhost"}'
-bin/freemarker-cli -t examples/data/template/ -PNGINX_HOSTNAME=www.mydomain.com -m env:///NGINX_CONF#mimetype=application/json
+freemarker-cli -t examples/data/template/ -PNGINX_HOSTNAME=www.mydomain.com -m env:///NGINX_CONF#mimetype=application/json
# == application.properties ==================================================
server.name=www.mydomain.com
server.logs=/var/log/nginx
diff --git a/freemarker-generator-cli/src/site/markdown/cli/tools/dataframe.md b/freemarker-generator-cli/src/site/markdown/cli/usage/using-dataframes.md
similarity index 92%
rename from freemarker-generator-cli/src/site/markdown/cli/tools/dataframe.md
rename to freemarker-generator-cli/src/site/markdown/cli/usage/using-dataframes.md
index 83809ed..25c0c0e 100644
--- a/freemarker-generator-cli/src/site/markdown/cli/tools/dataframe.md
+++ b/freemarker-generator-cli/src/site/markdown/cli/usage/using-dataframes.md
@@ -1,8 +1,8 @@
-# DataFrameTool
+## DataFrames
-The `DataFrameTool` uses [nRo/DataFrame](https://github.com/nRo/DataFrame) to convert tabular data into a `DataFrame`.
+A `DataFrame` allows declarative filtering and transformation of tabular data, i.e. less code to write.
-A `DataFrame` allows declartive filtering and transformation of tabular data, i.e. little code to write.
+The `DataFrameTool` uses [nRo/DataFrame](https://github.com/nRo/DataFrame) to convert tabular data into a `DataFrame`.
Currently the following sources are supported
@@ -10,7 +10,7 @@ Currently the following sources are supported
* JSON arrays represented as collection of maps
* Excel sheets represented as rows
-## CSV Examples
+### Working With CSV
[nRo/DataFrame]("https://raw.githubusercontent.com/nRo/DataFrame/master/src/test/resources/users.csv") provides the following CSV file
@@ -27,15 +27,27 @@ Schmitt;30;Germany
Meier;30;Germany
```
-and create a `DateFrame` using the following code
+and create a `DateFrame` using the following code snippet
+
+```
+<#assign dataSource = DataSources.get(0)>
+<#assign csvParser = CSVTool.parse(dataSource, CSVTool.formats["DATAFRAME"])>
+<#assign users = DataFrameTool.fromCSVParser(csvParser)>
+```
+
+The example can be executed by running
+
+```
+bin/freemarker-cli -PCSV_SOURCE_FORMAT=DATAFRAME -t examples/templates/dataframe/example.ftl examples/data/csv/dataframe.csv
+```
+
+or
```
-<#assign cvsFormat = CSVTool.formats["DEFAULT"].withHeader().withDelimiter(';')>
-<#assign csvParser = CSVTool.parse(DataSources.get(0), cvsFormat)>
-<#assign users = DataFrameTool.toDataFrame(csvParser)>
+bin/freemarker-cli -PCSV_SOURCE_FORMAT=DATAFRAME -t examples/templates/dataframe/example.ftl https://raw.githubusercontent.com/nRo/DataFrame/master/src/test/resources/users.csv
```
-### Select By Age
+#### Select By Age
```
${DataFrameTool.print(users.select("(age > 40)"))}
@@ -53,7 +65,7 @@ which shows
└────────────┴────────────┴────────────┘
```
-### Complex Select & Sort
+#### Complex Select & Sort
Now we want to create a new `DataFrame` by selecting `name` and `country`
@@ -82,7 +94,7 @@ which shows
└────────────┴────────────┴────────────┘
```
-### Count Column Values
+#### Count Column Values
Let's assume we want to count the records for each `country`
@@ -106,7 +118,7 @@ returns the following `DataFrame`
└────────────┴────────────┘
```
-### Group By Age And Country
+#### Group By Age And Country
Let's assume that we want to group the `DataFrame` by `age` and `country`
@@ -136,13 +148,14 @@ which results in
└────────────┴────────────┘
```
-## JSON Examples
+### Working With JSON
-Here we load a `examples/data/json/github-users.json` which represents a tabular data be
-being parsed as a list of maps and print the JSOB as dataframe
+Here we load a `examples/data/json/github-users.json` which represents a tabular
+data being parsed as a list of maps and print the JSON as dataframe. Technically
+it is a list of maps hence we invoke `DataFrameTool.fromMaps()
```
-./bin/freemarker-cli \
+freemarker-cli \
-i '${DataFrameTool.print(DataFrameTool.fromMaps(GsonTool.parse(DataSources.get(0))))}' \
examples/data/json/github-users.json
@@ -165,7 +178,7 @@ being parsed as a list of maps and print the JSOB as dataframe
└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
```
-## Excel Examples
+### Working With Excel
Let's transform an Excel Sheet to a `DataFrame` being printed using the following template
@@ -182,7 +195,7 @@ Let's transform an Excel Sheet to a `DataFrame` being printed using the followin
which is rendered by the following command line invocation
```
-./bin/freemarker-cli -t examples/templates/excel/dataframe/transform.ftl examples/data/excel/test.xls
+freemarker-cli -t examples/templates/excel/dataframe/transform.ftl examples/data/excel/test.xls
┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
│#Text │#Date │#Number │#Currency │#Time │#Percentag │#Forumula │
diff --git a/freemarker-generator-cli/src/site/markdown/index.md b/freemarker-generator-cli/src/site/markdown/index.md
index 2eb0dfd..a3ecefa 100644
--- a/freemarker-generator-cli/src/site/markdown/index.md
+++ b/freemarker-generator-cli/src/site/markdown/index.md
@@ -1,17 +1,36 @@
## FreeMarker Generator CLI
+`Apache FreeMarker CLI` is a command-line interface for [Apache FreeMarker](https://freemarker.apache.org/)
+
+* Requires JDK 1.8+ on Linux, Mac OSX and Windows
+* Use stdin, files and URLs to load data and templates
+* Reads a variety of structured data formats such as access logs, CSV, Excel, JSON, HTML, YAML, XML
+* Provides JSONPath, XPath and DataFrames for advanced data manipulation
+* Transform a directory with a single command-line invocation
+* Made for the heavy lifting of data by using lazy-loading and streaming
+
+### Getting Started
+
+* [Installation](cli/introduction/getting-started.html)
+* [Running Examples](cli/usage/running-examples.html)
+
### Concepts
+* [Design Goals](cli/concepts/design-goals.html)
* [Named URIs](cli/concepts/named-uris.html)
* [Data Sources](cli/concepts/data-sources.html)
* [Data Models](cli/concepts/data-models.html)
-* [User-Supplied Parameters](cli/concepts/user-parameters.html)
+* [Passing Configuration Data](cli/concepts/passing-data.html)
+* [Template Loading](cli/concepts/template-loading.html)
* [Transformation](cli/concepts/transformation.html)
-
-### Tools
-
-* [DataFrameTool](cli/tools/dataframe.html)
+* [Tools](cli/concepts/tools.html)
### Usage
* [Transforming Directories](cli/usage/transforming-directories.html)
+* [Using DataFrames](cli/usage/using-dataframes.html)
+* [Transforming CSV](cli/usage/transforming-csv.html)
+
+### Advanced Topics
+
+* [FreeMarker CLI Configuration](cli/advanced/cli-configuration.html)
diff --git a/freemarker-generator-cli/site/image/contract.png b/freemarker-generator-cli/src/site/resources/images/examples/contract.png
similarity index 100%
rename from freemarker-generator-cli/site/image/contract.png
rename to freemarker-generator-cli/src/site/resources/images/examples/contract.png
diff --git a/freemarker-generator-cli/site/image/excel-to-html.png b/freemarker-generator-cli/src/site/resources/images/examples/excel-to-html.png
similarity index 100%
rename from freemarker-generator-cli/site/image/excel-to-html.png
rename to freemarker-generator-cli/src/site/resources/images/examples/excel-to-html.png
diff --git a/freemarker-generator-cli/site/image/github.png b/freemarker-generator-cli/src/site/resources/images/examples/github.png
similarity index 100%
rename from freemarker-generator-cli/site/image/github.png
rename to freemarker-generator-cli/src/site/resources/images/examples/github.png
diff --git a/freemarker-generator-cli/site/image/locker-test-users-pdf.png b/freemarker-generator-cli/src/site/resources/images/examples/locker-test-users-pdf.png
similarity index 100%
rename from freemarker-generator-cli/site/image/locker-test-users-pdf.png
rename to freemarker-generator-cli/src/site/resources/images/examples/locker-test-users-pdf.png
diff --git a/freemarker-generator-cli/site/image/transactions.png b/freemarker-generator-cli/src/site/resources/images/examples/transactions.png
similarity index 100%
rename from freemarker-generator-cli/site/image/transactions.png
rename to freemarker-generator-cli/src/site/resources/images/examples/transactions.png
diff --git a/freemarker-generator-cli/src/test/java/org/apache/freemarker/generator/cli/ManualTest.java b/freemarker-generator-cli/src/test/java/org/apache/freemarker/generator/cli/ManualTest.java
index faa6080..2b7f456 100644
--- a/freemarker-generator-cli/src/test/java/org/apache/freemarker/generator/cli/ManualTest.java
+++ b/freemarker-generator-cli/src/test/java/org/apache/freemarker/generator/cli/ManualTest.java
@@ -24,8 +24,11 @@ import java.util.Arrays;
public class ManualTest {
private static final String SPACE = " ";
+ private static final String CMD = "-V";
// private static final String CMD = "-PCSV_SOURCE_FORMAT=DATAFRAME -t examples/templates/dataframe/example.ftl https://raw.githubusercontent.com/nRo/DataFrame/master/src/test/resources/users.csv";
- private static final String CMD = "-PCSV_SOURCE_WITH_HEADER=true -PCSV_SOURCE_FORMAT=DEFAULT -PCSV_TARGET_FORMAT=EXCEL -t templates/csv/md/transform.ftl examples/data/csv/contract.csv";
+ // private static final String CMD = "-PCSV_SOURCE_WITH_HEADER=false -PCSV_SOURCE_FORMAT=DEFAULT -PCSV_TARGET_FORMAT=EXCEL -PCSV_TARGET_WITH_HEADER=true -t templates/csv/csv/transform.ftl examples/data/csv/contract.csv";
+ // private static final String CMD = "-t examples/templates/json/dataframe/github-users.ftl examples/data/json/github-users.json";
+
public static void main(String[] args) {
Main.execute(toArgs(CMD));
diff --git a/freemarker-generator-cli/examples/templates/cat.ftl b/freemarker-generator-cli/templates/cat.ftl
similarity index 94%
rename from freemarker-generator-cli/examples/templates/cat.ftl
rename to freemarker-generator-cli/templates/cat.ftl
index a878261..c912c12 100644
--- a/freemarker-generator-cli/examples/templates/cat.ftl
+++ b/freemarker-generator-cli/templates/cat.ftl
@@ -1,4 +1,3 @@
-<#ftl output_format="plainText" strip_whitespace=true>
<#--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/freemarker-generator-cli/templates/csv/csv/transform.ftl b/freemarker-generator-cli/templates/csv/csv/transform.ftl
index 656131d..ef8cf26 100644
--- a/freemarker-generator-cli/templates/csv/csv/transform.ftl
+++ b/freemarker-generator-cli/templates/csv/csv/transform.ftl
@@ -18,7 +18,12 @@
<#import "/templates/lib/commons-csv.ftl" as csv />
<#assign dataSource = DataSources.get(0)>
<#assign csvParser = CSVTool.parse(dataSource, csv.sourceFormat())>
-<#assign csvPrinter = CSVTool.printer(csv.targetFormat())>
+<#assign csvTargetFormat = csv.targetFormat()>
+<#assign csvPrinter = CSVTool.printer(csvTargetFormat)>
+<#assign csvHeaders = (csvParser.getHeaderMap()!{})?keys>
+<#if csvHeaders?has_content && csvTargetFormat.getSkipHeaderRecord()>
+ ${csvPrinter.printRecord(csvHeaders)}<#t>
+</#if>
<#list csvParser.iterator() as record>
${csvPrinter.printRecord(record)}<#t>
</#list>
\ No newline at end of file
diff --git a/freemarker-generator-cli/templates/info.ftl b/freemarker-generator-cli/templates/info.ftl
index 838ac29..11275b6 100644
--- a/freemarker-generator-cli/templates/info.ftl
+++ b/freemarker-generator-cli/templates/info.ftl
@@ -25,7 +25,7 @@ Timestamp : ${.now}
Output encoding : ${.output_encoding}
Output format : ${.output_format}
-FreeMarker CLI Template Directories
+FreeMarker CLI Template Loader Directories
------------------------------------------------------------------------------
<#list SystemTool.getTemplateDirectories() as directory>
[#${directory?counter}] ${directory}
diff --git a/freemarker-generator-cli/templates/lib/commons-csv.ftl b/freemarker-generator-cli/templates/lib/commons-csv.ftl
index 4082fa8..4e3c771 100644
--- a/freemarker-generator-cli/templates/lib/commons-csv.ftl
+++ b/freemarker-generator-cli/templates/lib/commons-csv.ftl
@@ -27,10 +27,10 @@
<#function sourceFormat>
<#assign format = CSVTool.formats[CSV_SOURCE_FORMAT!"DEFAULT"]>
<#assign delimiter = CSVTool.toDelimiter(CSV_SOURCE_DELIMITER!format.getDelimiter())>
- <#assign withHeader = CSV_SOURCE_WITH_HEADER!"false">
+ <#assign withHeader = CSV_SOURCE_WITH_HEADER!"true">
<#assign format = format.withDelimiter(delimiter)>
<#if withHeader?boolean>
- <#assign format = format.withHeader()>
+ <#assign format = format.withFirstRecordAsHeader()>
</#if>
<#return format>
</#function>
@@ -46,10 +46,10 @@
<#function targetFormat>
<#assign format = CSVTool.formats[CSV_TARGET_FORMAT!"DEFAULT"]>
<#assign delimiter = CSVTool.toDelimiter(CSV_TARGET_DELIMITER!format.getDelimiter())>
- <#assign withHeader = CSV_TARGET_WITH_HEADER!"false">
+ <#assign withHeader = CSV_TARGET_WITH_HEADER!"true">
<#assign format = format.withDelimiter(delimiter)>
<#if withHeader?boolean>
- <#assign format = format.withHeader>
+ <#assign format = format.withFirstRecordAsHeader()>
</#if>
<#return format>
</#function>
diff --git a/freemarker-generator-maven-plugin/CHANGELOG.md b/freemarker-generator-maven-plugin/CHANGELOG.md
new file mode 100644
index 0000000..015a87f
--- /dev/null
+++ b/freemarker-generator-maven-plugin/CHANGELOG.md
@@ -0,0 +1,12 @@
+# Change Log
+
+All notable changes to this project will be documented in this file. We try to adhere to https://github.com/olivierlacan/keep-a-changelog.
+
+## 0.1.0-SNAPSHOT
+
+### Changed
+* [FREEMARKER-129] Change artifact `freemarker-maven-plugin` to `freemarker-generator-maven-plugin`
+* [FREEMARKER-128] Update `freemarker-maven-plugin` to Apache FreeMarker 2.3.29
+
+[FREEMARKER-128]: https://issues.apache.org/jira/browse/FREEMARKER-128
+[FREEMARKER-129]: https://issues.apache.org/jira/browse/FREEMARKER-129
diff --git a/freemarker-generator-maven-plugin/pom.xml b/freemarker-generator-maven-plugin/pom.xml
index e028aa8..be360ec 100644
--- a/freemarker-generator-maven-plugin/pom.xml
+++ b/freemarker-generator-maven-plugin/pom.xml
@@ -40,8 +40,6 @@
<jmockit.version>1.32</jmockit.version>
<org.testng.version>6.8</org.testng.version>
<assertj-core.version>3.8.0</assertj-core.version>
- <clover-target-percentage>100</clover-target-percentage>
- <clover-phase>pre-site</clover-phase>
</properties>
<dependencies>
@@ -153,27 +151,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.openclover</groupId>
- <artifactId>clover-maven-plugin</artifactId>
- <executions>
- <execution>
- <id>clover</id>
- <phase>${clover-phase}</phase>
- <goals>
- <goal>instrument-test</goal>
- <goal>clover</goal>
- <goal>check</goal>
- </goals>
- <configuration>
- <targetPercentage>${clover-target-percentage}</targetPercentage>
- <generateHtml>true</generateHtml>
- <generateXml>true</generateXml>
- <jdk>${maven.compiler.source}</jdk>
- </configuration>
- </execution>
- </executions>
- </plugin>
</plugins>
</build>
diff --git a/freemarker-generator-tools/README.md b/freemarker-generator-tools/README.md
index e624eb2..0c49388 100644
--- a/freemarker-generator-tools/README.md
+++ b/freemarker-generator-tools/README.md
@@ -1,35 +1,21 @@
Apache FreeMarker Generator Tools
=============================================================================
-A "tool" is just a POJO (plain old java object) that is "useful" in a template and is not meant to be rendered in the output.
+The following tools are currently implemented
-Design Considerations
-------------------------------------------------------------------------------
-
-* A tool shall expose a default constructor and/or a constructor taking a `Map<String, Object> settings`
-* A tool processing data sources shall provide a `parse(Datasource)` method
-* A tool shall be stateless, multi-thread safe and potentially long-lived to enable usage in a different context
-* A tool shall support arbitrary large source data sources and process them efficiently
-* A tool shall expose a `toString()` method for documentation purposes
-
-Available Tools
-------------------------------------------------------------------------------
-
-The following tools are currently provided
-
-| Helper | Description |
+| Tool | Description |
|-----------------------|-----------------------------------------------------------------------------------------------------------|
| CSVTool | Process CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/) |
+| DataFrameTool | Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame) |
| ExecTool | Execute command line tools using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/) |
| ExcelTool | Process Excels files (XLS, XLSX) using [Apache POI](https://poi.apache.org) |
-| FreeMarkerTool | Expose useful Apache FreeMarker classes |
-| GrokTool | Process text files using [Grok](https://github.com/thekrakken/java-grok) expressions |
-| JsonPathTool | Process JSON files using [Java JSON Path](https://github.com/json-path/JsonPath) |
-| JsoupTool | Process HTML files using [Jsoup](https://jsoup.org) |
+| FreeMarkerTool | Expose useful FreeMarker classes |
+| GrokTool | Process text files using [Grok](https://github.com/thekrakken/java-grok) instead of regular expressions |
+| GsonTool | Process JSON files using [GSON](https://github.com/google/gson) |
+| JsonPathTool | Process JSON file using [Java JSON Path](https://github.com/json-path/JsonPath) |
+| JsoupTool | Processing HTML files using [Jsoup](https://jsoup.org) |
| PropertiesTool | Process JDK properties files |
-| SystemTool | Expose System-related utility methods |
-| XmlTool | Process XML files using [Apache FreeMarker](https://freemarker.apache.org/docs/xgui.html) |
-| YamlTool | Process YAML files using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home) |
+| SystemTool | System-related utility methods |
| UUIDTool | Create UUIDs |
-
-
+| XmlTool | Process XML files using [Apache FreeMarker](https://freemarker.apache.org/docs/xgui.html) |
+| YamlTool | Process YAML files using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home) |
\ No newline at end of file
diff --git a/freemarker-generator-tools/pom.xml b/freemarker-generator-tools/pom.xml
index 57c6b59..256987a 100644
--- a/freemarker-generator-tools/pom.xml
+++ b/freemarker-generator-tools/pom.xml
@@ -122,10 +122,12 @@
<reporting>
<plugins>
<plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
<plugin>
+ <artifactId>maven-jxr-plugin</artifactId>
+ </plugin>
+ <plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<configuration>
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVTool.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVTool.java
index 7d2eab6..1dcda46 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVTool.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVTool.java
@@ -22,6 +22,7 @@ import org.apache.commons.csv.CSVRecord;
import org.apache.commons.io.input.BOMInputStream;
import org.apache.freemarker.generator.base.datasource.DataSource;
import org.apache.freemarker.generator.base.util.StringUtils;
+import org.apache.freemarker.generator.tools.commonscsv.impl.CommonsCSVPrinterFacade;
import java.io.IOException;
import java.io.InputStream;
@@ -72,7 +73,7 @@ public class CommonsCSVTool {
}
try {
- // We don't need to close the underyling ByteArrayInputStream
+ // We don't need to close the underlying ByteArrayInputStream
return parse(toInputStream(csv, UTF_8), UTF_8, format);
} catch (IOException e) {
throw new RuntimeException("Failed to parse CSV", e);
@@ -115,7 +116,7 @@ public class CommonsCSVTool {
/**
* Map the given value of the CVS record into (key to record). If duplicates
- * are encountered return the first occurence of the CVS record. The map
+ * are encountered return the first occurrence of the CVS record. The map
* retains the insertion order of they keys.
*
* @param records records to process
@@ -128,7 +129,7 @@ public class CommonsCSVTool {
/**
* Map the given value of the CVS record into (key to record). If duplicates
- * are encountered return the first occurence of the CVS record. The map
+ * are encountered return the first occurrence of the CVS record. The map
* retains the insertion order of they keys.
*
* @param records records to process
@@ -162,8 +163,8 @@ public class CommonsCSVTool {
}
/**
- * Maps the sybmolic name of a delimiter to a single character since it
- * is not possible to define commony used delimiters on the command line.
+ * Maps the symbolic name of a delimiter to a single character since it
+ * is not possible to define commonly used delimiters on the command line.
*
* @param name symbolic name of delimiter
* @return CSV delimiter
@@ -238,7 +239,7 @@ public class CommonsCSVTool {
private static Map<String, CSVFormat> createCSVFormats() {
final Map<String, CSVFormat> result = new HashMap<>();
result.put("DEFAULT", CSVFormat.DEFAULT);
- result.put("DATAFRAME", CSVFormat.DEFAULT.withDelimiter(';').withFirstRecordAsHeader());
+ result.put("DATAFRAME", CSVFormat.RFC4180.withDelimiter(';').withFirstRecordAsHeader());
result.put("EXCEL", CSVFormat.EXCEL);
result.put("INFORMIX_UNLOAD", CSVFormat.INFORMIX_UNLOAD);
result.put("INFORMIX_UNLOAD_CSV", CSVFormat.INFORMIX_UNLOAD_CSV);
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVPrinterFacade.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/impl/CommonsCSVPrinterFacade.java
similarity index 97%
rename from freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVPrinterFacade.java
rename to freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/impl/CommonsCSVPrinterFacade.java
index b8c0574..c8e8ac5 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVPrinterFacade.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/commonscsv/impl/CommonsCSVPrinterFacade.java
@@ -1,4 +1,4 @@
-package org.apache.freemarker.generator.tools.commonscsv;
+package org.apache.freemarker.generator.tools.commonscsv.impl;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/DataFrameTool.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/DataFrameTool.java
index bef1b7a..640d798 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/DataFrameTool.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/DataFrameTool.java
@@ -23,9 +23,9 @@ import de.unknownreality.dataframe.sort.SortColumn.Direction;
import de.unknownreality.dataframe.transform.ColumnDataFrameTransform;
import de.unknownreality.dataframe.transform.CountTransformer;
import org.apache.commons.csv.CSVParser;
-import org.apache.freemarker.generator.tools.dataframe.converter.CSVConverter;
-import org.apache.freemarker.generator.tools.dataframe.converter.ListConverter;
-import org.apache.freemarker.generator.tools.dataframe.converter.MapConverter;
+import org.apache.freemarker.generator.tools.dataframe.impl.CSVConverter;
+import org.apache.freemarker.generator.tools.dataframe.impl.ListConverter;
+import org.apache.freemarker.generator.tools.dataframe.impl.MapConverter;
import java.io.StringWriter;
import java.util.Collection;
@@ -43,7 +43,9 @@ import static de.unknownreality.dataframe.DataFrameWriter.DEFAULT_PRINT_FORMAT;
public class DataFrameTool {
/**
- * Create a default data frame.
+ * Create an empty data frame.
+ *
+ * @return data frame
*/
public DataFrame create() {
return new DefaultDataFrame();
@@ -81,7 +83,7 @@ public class DataFrameTool {
}
/**
- * Provide a convinience map with predefined sort orders to be used by templates.
+ * Provide a convenience map with predefined sort orders to be used by templates.
*
* @return available sort orders
*/
@@ -93,7 +95,7 @@ public class DataFrameTool {
}
/**
- * Provide a convinience map with predefined transformers.
+ * Provide a convenience map with predefined transformers.
*
* @return available transformers
*/
@@ -107,6 +109,7 @@ public class DataFrameTool {
* Print the <code>DataFrame</code> to the FreeMarker writer.
*
* @param dataFrame data frame
+ * @return text representation of data frame
*/
public String print(DataFrame dataFrame) {
final StringWriter writer = new StringWriter();
@@ -119,6 +122,7 @@ public class DataFrameTool {
return "Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame)";
}
+ @SuppressWarnings("rawtypes")
private static CountTransformer countTransformer(boolean ignoreNA) {
return new CountTransformer(ignoreNA);
}
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/CSVConverter.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/CSVConverter.java
similarity index 97%
rename from freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/CSVConverter.java
rename to freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/CSVConverter.java
index 055b508..5745c35 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/CSVConverter.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/CSVConverter.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.freemarker.generator.tools.dataframe.converter;
+package org.apache.freemarker.generator.tools.dataframe.impl;
import de.unknownreality.dataframe.DataFrame;
import de.unknownreality.dataframe.DataFrameBuilder;
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/ConverterUtils.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/ConverterUtils.java
similarity index 93%
rename from freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/ConverterUtils.java
rename to freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/ConverterUtils.java
index 5237f52..09beacd 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/ConverterUtils.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/ConverterUtils.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.freemarker.generator.tools.dataframe.converter;
+package org.apache.freemarker.generator.tools.dataframe.impl;
import de.unknownreality.dataframe.DataFrame;
import de.unknownreality.dataframe.DataFrameBuilder;
@@ -30,15 +30,15 @@ public class ConverterUtils {
}
static String getAlphaColumnName(int num) {
- String result = "";
+ final StringBuilder result = new StringBuilder();
while (num > 0) {
num--; // 1 => a, not 0 => a
final int remainder = num % 26;
final char digit = (char) (remainder + 65);
- result = digit + result;
+ result.insert(0, digit);
num = (num - remainder) / 26;
}
- return result;
+ return result.toString();
}
private static DataFrameBuilder addColumn(DataFrameBuilder builder, String columnName, Class<?> columnType) {
@@ -66,7 +66,7 @@ public class ConverterUtils {
case "java.util.Date":
return builder.addStringColumn(columnName);
default:
- throw new RuntimeException("Unable to add colum for the following type: " + columnType.getName());
+ throw new RuntimeException("Unable to add column for the following type: " + columnType.getName());
}
}
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/ListConverter.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/ListConverter.java
similarity index 91%
rename from freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/ListConverter.java
rename to freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/ListConverter.java
index ecf891a..4f83d04 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/ListConverter.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/ListConverter.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.freemarker.generator.tools.dataframe.converter;
+package org.apache.freemarker.generator.tools.dataframe.impl;
import de.unknownreality.dataframe.DataFrame;
import org.apache.freemarker.generator.base.table.Table;
@@ -28,6 +28,7 @@ public class ListConverter {
* that the rows represent tabular data.
*
* @param rows rows to build the data frame
+ * @param withFirstRowAsColumnNames column names as first row?
* @return <code>DataFrame</code>
*/
public static DataFrame toDataFrame(List<List<Object>> rows, boolean withFirstRowAsColumnNames) {
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/MapConverter.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/MapConverter.java
similarity index 92%
rename from freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/MapConverter.java
rename to freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/MapConverter.java
index d8906c1..bc1a287 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/converter/MapConverter.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/dataframe/impl/MapConverter.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.freemarker.generator.tools.dataframe.converter;
+package org.apache.freemarker.generator.tools.dataframe.impl;
import de.unknownreality.dataframe.DataFrame;
import org.apache.freemarker.generator.base.table.Table;
@@ -22,8 +22,6 @@ import org.apache.freemarker.generator.base.table.Table;
import java.util.Collection;
import java.util.Map;
-import static java.util.Collections.singletonList;
-
public class MapConverter {
/**
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/excel/ExcelTool.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/excel/ExcelTool.java
index b5ead1d..d9bb473 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/excel/ExcelTool.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/excel/ExcelTool.java
@@ -53,7 +53,7 @@ public class ExcelTool {
// make sure that the workbook is closed together with the data source
return dataSource.addClosable(workbook);
} catch (IOException e) {
- throw new RuntimeException("Failed to parse Ecxel data source: " + dataSource, e);
+ throw new RuntimeException("Failed to parse Excel data source: " + dataSource, e);
}
}
@@ -94,7 +94,7 @@ public class ExcelTool {
/**
* EXPERIMENTAL FEATURE
* <p>
- * Transform the sheet to table contaning raw Java objects, e.g. Date, Double, ...
+ * Transform the sheet to table containing raw Java objects, e.g. Date, Double, ...
*
* @param sheet Excel sheet
* @return Table containing cells as raw Java objects
@@ -121,8 +121,8 @@ public class ExcelTool {
final List<Object> columnValues = new ArrayList<>();
for (int columnIndex = 0; columnIndex < row.getLastCellNum(); columnIndex++) {
final Cell cell = row.getCell(columnIndex, CREATE_NULL_AS_BLANK);
- final String formatedCellValue = dataFormatter.formatCellValue(cell).trim();
- columnValues.add(formatedCellValue);
+ final String formattedCellValue = dataFormatter.formatCellValue(cell).trim();
+ columnValues.add(formattedCellValue);
}
return columnValues;
}
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/freemarker/FreeMarkerTool.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/freemarker/FreeMarkerTool.java
index 45b43e6..217aa8f 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/freemarker/FreeMarkerTool.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/freemarker/FreeMarkerTool.java
@@ -22,11 +22,19 @@ import freemarker.template.Configuration;
import freemarker.template.TemplateHashModel;
import freemarker.template.utility.ObjectConstructor;
+/**
+ * Exposes useful Apache FreeMarker classes for advanced FTL usage.
+ */
public class FreeMarkerTool {
- private BeansWrapper beansWrapper;
- private ObjectConstructor objectConstructor;
+ private volatile BeansWrapper beansWrapper;
+ private volatile ObjectConstructor objectConstructor;
+ /**
+ * See <a href="https://freemarker.apache.org/docs/api/freemarker/template/utility/ObjectConstructor.html">FreeMarker ObjectConstructor</a>.
+ *
+ * @return ObjectConstructor
+ */
public synchronized ObjectConstructor getObjectConstructor() {
if (objectConstructor == null) {
objectConstructor = new ObjectConstructor();
@@ -34,6 +42,11 @@ public class FreeMarkerTool {
return objectConstructor;
}
+ /**
+ * See <a href="https://freemarker.apache.org/docs/api/freemarker/ext/beans/BeansWrapper.html">FreeMarker BeansWrapper</a>.
+ *
+ * @return BeansWrapper
+ */
public synchronized BeansWrapper getBeansWrapper() {
if (beansWrapper == null) {
beansWrapper = new BeansWrapperBuilder(Configuration.VERSION_2_3_30).build();
@@ -51,7 +64,7 @@ public class FreeMarkerTool {
@Override
public String toString() {
- return "Expose useful Apache FreeMarker classes";
+ return "Expose advanced Apache FreeMarker classes";
}
}
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokTool.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokTool.java
index 984d22c..0beab72 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokTool.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokTool.java
@@ -18,6 +18,7 @@ package org.apache.freemarker.generator.tools.grok;
import io.krakens.grok.api.Grok;
import io.krakens.grok.api.GrokCompiler;
+import org.apache.freemarker.generator.tools.grok.impl.GrokWrapper;
public class GrokTool {
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokWrapper.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/impl/GrokWrapper.java
similarity index 95%
rename from freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokWrapper.java
rename to freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/impl/GrokWrapper.java
index 7a6931e..626a94f 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/GrokWrapper.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/grok/impl/GrokWrapper.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.freemarker.generator.tools.grok;
+package org.apache.freemarker.generator.tools.grok.impl;
import io.krakens.grok.api.Grok;
import org.apache.freemarker.generator.base.util.StringUtils;
diff --git a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/system/SystemTool.java b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/system/SystemTool.java
index 3f038f7..36a8139 100644
--- a/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/system/SystemTool.java
+++ b/freemarker-generator-tools/src/main/java/org/apache/freemarker/generator/tools/system/SystemTool.java
@@ -125,7 +125,7 @@ public class SystemTool {
}
/**
- * Convinience lookup of a configuration value based on
+ * Convenience lookup of a configuration value based on
* user-supplied parameters, system properties and
* environment variables.
*
@@ -140,7 +140,7 @@ public class SystemTool {
}
/**
- * Convinience lookup of a configuration value based on
+ * Convenience lookup of a configuration value based on
* user-supplied parameters, system properties and
* environment variables.
*
diff --git a/freemarker-generator-tools/src/site/markdown/index.md b/freemarker-generator-tools/src/site/markdown/index.md
index 2fd9f95..19d0bb2 100644
--- a/freemarker-generator-tools/src/site/markdown/index.md
+++ b/freemarker-generator-tools/src/site/markdown/index.md
@@ -1 +1,66 @@
-TBD
\ No newline at end of file
+# Apache FreeMarker CLI Tools
+
+## Overview
+
+The implementation of the `Apache FreeMarker CLI Tools` was inspired by [Apache Velocity Tools](https://velocity.apache.org/tools/devel/) - a `tool` is just a POJO (plain old Java object) that is "useful" in a template and is not meant to be rendered in the output.
+
+Let's have a look at the anatomy and life-cycle of a `Apache FreeMarker CLI Tool`
+
+* The meta-data, e.g. class name, is read from `freemarker-cli.properties`
+* It provides a default constructor
+* Its `toString` methods prints a short description
+* It exposes public methods being used directly by the template
+* It is instantiated once and is multi-thread-safe
+
+## Available Tools
+
+The following `tools` are currently implemented
+
+| Tool | Description |
+|-----------------------|-----------------------------------------------------------------------------------------------------------|
+| CSVTool | Process CSV files using [Apache Commons CSV](https://commons.apache.org/proper/commons-csv/) |
+| DataFrameTool | Bridge to [nRo/DataFrame](https://github.com/nRo/DataFrame) |
+| ExecTool | Execute command line tools using [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/) |
+| ExcelTool | Process Excels files (XLS, XLSX) using [Apache POI](https://poi.apache.org) |
+| FreeMarkerTool | Expose useful FreeMarker classes |
+| GrokTool | Process text files using [Grok](https://github.com/thekrakken/java-grok) instead of regular expressions |
+| GsonTool | Process JSON files using [GSON](https://github.com/google/gson) |
+| JsonPathTool | Process JSON file using [Java JSON Path](https://github.com/json-path/JsonPath) |
+| JsoupTool | Processing HTML files using [Jsoup](https://jsoup.org) |
+| PropertiesTool | Process JDK properties files |
+| SystemTool | System-related utility methods |
+| UUIDTool | Create UUIDs |
+| XmlTool | Process XML files using [Apache FreeMarker](https://freemarker.apache.org/docs/xgui.html) |
+| YamlTool | Process YAML files using [SnakeYAML](https://bitbucket.org/asomov/snakeyaml/wiki/Home) |
+
+## Advanced Topics
+
+### Auto-closing Resources
+
+The user can create objects which need to be closed later on to avoid excessive resource usage. This is less of a concern for a short-lived CLI application but if many data sources are processed or the code is used in a different context the problem becomes more severe.
+
+The `Excel Tool` provides the following code to keep track of `Closables`
+
+```java
+package org.apache.freemarker.generator.tools.excel;
+
+public class ExcelTool {
+
+ public Workbook parse(DataSource dataSource) {
+ try (InputStream is = dataSource.getUnsafeInputStream()) {
+ final Workbook workbook = WorkbookFactory.create(is);
+ // make sure that the workbook is closed together with the data source
+ return dataSource.addClosable(workbook);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to parse Excel data source: " + dataSource, e);
+ }
+ }
+}
+```
+
+So what is happening here
+
+* The `Workbook` is tracked by the originating `DataSource`
+* The `DataSource` implements the `Closable` interface
+* All `DataSources` are closed automatically when rendering is done.
+
diff --git a/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVToolTest.java b/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVToolTest.java
index 29f8e0e..472e378 100644
--- a/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVToolTest.java
+++ b/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/commonscsv/CommonsCSVToolTest.java
@@ -21,6 +21,7 @@ import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import org.apache.freemarker.generator.base.datasource.DataSource;
import org.apache.freemarker.generator.base.datasource.DataSourceFactory;
+import org.apache.freemarker.generator.tools.commonscsv.impl.CommonsCSVPrinterFacade;
import org.junit.Test;
import java.io.File;
diff --git a/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/grok/GrokToolTest.java b/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/grok/GrokToolTest.java
index 2bf21dc..30c0ed7 100644
--- a/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/grok/GrokToolTest.java
+++ b/freemarker-generator-tools/src/test/java/org/apache/freemarker/generator/tools/grok/GrokToolTest.java
@@ -16,6 +16,7 @@
*/
package org.apache.freemarker.generator.tools.grok;
+import org.apache.freemarker.generator.tools.grok.impl.GrokWrapper;
import org.junit.Test;
import java.util.Map;
diff --git a/pom.xml b/pom.xml
index 58b46f7..45cb794 100644
--- a/pom.xml
+++ b/pom.xml
@@ -123,6 +123,11 @@
<version>1.7.28</version>
</dependency>
<dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-nop</artifactId>
+ <version>1.7.28</version>
+ </dependency>
+ <dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
@@ -213,7 +218,7 @@
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
- <version>2.2.6</version>
+ <version>4.0.0</version>
</plugin>
<plugin>
<groupId>org.apache.rat</groupId>
diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md
index 455ebb0..bf0072c 100644
--- a/src/site/markdown/index.md
+++ b/src/site/markdown/index.md
@@ -2,7 +2,5 @@ The Apache FreeMarker Generator projects provides additional tools to generate t
| Name | Description |
| ------------------------------------------------------------- | ----------------------------------------------------------------- |
-| [Base](freemarker-generator-base/index.html) | Common functionality independent from Apache FreeMarker |
-| [Tools](freemarker-generator-tools/index.html) | Data source processing tools for Apache FreeMarker Generator |
| [CLI](freemarker-generator-cli/index.html) | Command-line client for Apache FreeMarker |
| [Maven Plugin](freemarker-generator-maven-plugin/index.html) | Maven plugin for Apache FreeMarker |
\ No newline at end of file
diff --git a/travis.sh b/travis.sh
index f76ae9d..1c55d1e 100755
--- a/travis.sh
+++ b/travis.sh
@@ -1,6 +1,24 @@
#!/bin/sh
-mvn clean install
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Simulate the Travis build locally
+mvn clean install site site:stage
cd ./freemarker-generator-cli
sh ./run-examples.sh
cd ../freemarker-generator-maven-plugin-sample
-mvn clean package
+mvn clean package
\ No newline at end of file