You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metamodel.apache.org by ka...@apache.org on 2016/05/16 03:54:10 UTC
[35/42] metamodel git commit: METAMODEL-247: Fixed
METAMODEL-247: Fixed
Closes #97
Project: http://git-wip-us.apache.org/repos/asf/metamodel/repo
Commit: http://git-wip-us.apache.org/repos/asf/metamodel/commit/47c6d56c
Tree: http://git-wip-us.apache.org/repos/asf/metamodel/tree/47c6d56c
Diff: http://git-wip-us.apache.org/repos/asf/metamodel/diff/47c6d56c
Branch: refs/heads/5.x
Commit: 47c6d56cc7dbc3e6772b94d3816b94252622cbef
Parents: 025d07f
Author: kaspersorensen <i....@gmail.com>
Authored: Thu Apr 28 16:39:41 2016 -0700
Committer: kaspersorensen <i....@gmail.com>
Committed: Thu Apr 28 16:39:41 2016 -0700
----------------------------------------------------------------------
CHANGES.md | 335 ++++++++---------
.../schema/naming/ColumnNamingStrategies.java | 10 +
.../naming/CustomColumnNamingStrategy.java | 62 ++++
fixedwidth/pom.xml | 98 ++---
.../fixedwidth/FixedWidthColumnSpec.java | 45 +++
.../fixedwidth/FixedWidthConfiguration.java | 358 ++++++++++---------
.../FixedWidthConfigurationReader.java | 180 ++++++++++
.../FixedWidthConfigurationReaderTest.java | 89 +++++
.../src/test/resources/metadata_spec1/data.txt | 5 +
.../metadata_spec1/sas-formatfile-metadata.txt | 4 +
.../metadata_spec1/sas-input-metadata.txt | 19 +
11 files changed, 823 insertions(+), 382 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/CHANGES.md
----------------------------------------------------------------------
diff --git a/CHANGES.md b/CHANGES.md
index 4356071..dd03195 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,167 +1,168 @@
-### Apache MetaModel 4.5.3 (work in progress)
-
- * [METAMODEL-235] - Fixed a bug related to handling of null or missing values in ElasticSearch using REST client.
- * [METAMODEL-225] - Fixed support for nested objects and arrays in ElasticSearch using REST client.
- * [METAMODEL-244] - Added ColumnNamingStrategies concept which allows custom column naming and column name overriding.
- * [METAMODEL-242] - Fixed issue when de-serializing old enum-instances of FunctionType.
-
-### Apache MetaModel 4.5.2
-
- * [METAMODEL-236] - Made OperatorType and FunctionType Serializable to ensure that serialization of Query is possible.
-
-### Apache MetaModel 4.5.1
-
- * [METAMODEL-227] - Fix for respecting CSV escape character also when no quote character is set.
- * [METAMODEL-183] - MongoDB module split into three: common, Mongo2 and Mongo3 to allow use of either old or new MongoDB API.
- * [METAMODEL-231] - Fixed a bug causing the Neo4j to represent the same table multiple times within a schema.
- * [METAMODEL-228] - Fixed a bug causing Number.class to not be converted to ColumnType.NUMBER.
-
-### Apache MetaModel 4.5.0
-
- * [METAMODEL-212] - New module for ElasticSearch via REST client.
- * [METAMODEL-99] - New module for Neo4j connectivity with MetaModel.
- * [METAMODEL-207] - Ensured the serializability of the SingleLineCsvRow class.
- * [METAMODEL-211] - Fixed a bug related to lookup by primary key (_id) on MongoDB.
- * [METAMODEL-216] - Added new aggregate functions: FIRST, LAST and RANDOM.
- * [METAMODEL-195] - Added new function MAP_VALUE which allows extracting a nested value from within a key/value map field.
- * [METAMODEL-15] - Query parser support for table names with space. Delimitters can be double quote or square brackets.
- * [METAMODEL-215] - Improved the capability of NumberComparator to support Integer, Long, Double, BigInteger and other built-in Number classes.
- * [METAMODEL-218] - Fixed conversion of STRING and NUMBER types to database-specific types in JDBC module.
- * [METAMODEL-205] - Added validation of Excel sheet name before attempting to create table (sheet).
- * [METAMODEL-219] - Made HdfsResource capable of incorporating Hadoop configuration files core-site.xml and hdfs-site.xml
- * [METAMODEL-220] - Made HdfsResource capable of working with other URI schemes than 'hdfs'.
-
-### Apache MetaModel 4.4.1
-
- * [METAMODEL-198] - Fixed support for JDBC TIMESTAMP precision to match the underlying database's precision.
- * [METAMODEL-200] - Added optional "APPROXIMATE" keyword to query syntax for aggregate functions.
- * [METAMODEL-144] - Automated binary packaging of the MetaModel project.
- * [METAMODEL-197] - ElasticSearch schema update/change after CREATE TABLE statements.
- * [METAMODEL-199] - Fixed a bug in query parser when parsing two consecutive WHERE items with parentheses around them.
- * [METAMODEL-203] - Upgraded MongoDB dependency version and API to the 3.x line.
-
-### Apache MetaModel 4.4.0
-
- * [METAMODEL-192] - Added support for Scalar functions. We have a basic set of datatype conversion functions as well as support for UDF via implementing the ScalarFunction interface.
- * [METAMODEL-194] - Added support for setting the "Max rows" flag of a query to 0. This will always return an empty dataset.
- * [METAMODEL-173] - Improved CSV writing to non-file destinations. Added .write() and .append() methods to Resource interface.
- * [METAMODEL-170] - Dropped support for Java 6.
- * [METAMODEL-176] - Trimmed the transient dependencies of the JDBC module.
- * [METAMODEL-178] - Added AggregateFunction and ScalarFunction interfaces. Changed FunctionType enum to be super-interface of those. Compatibility is retained but a recompile of code using FunctionType is needed.
- * [METAMODEL-188] - Changed OperatorType enum to be an interface. Compatibility is retained but a recompile of code is needed.
- * [METAMODEL-179] - Ensured that HdfsResource is not closing a shared HDFS file system reference.
- * [METAMODEL-171] - Made integration tests for Cassandra module function properly in all environments.
- * [METAMODEL-177] - Fixed a bug pertaining to the serializability of HdfsResource.
- * [METAMODEL-172] - ElasticSearch Date types should be converted properly.
- * [METAMODEL-184] - ElasticSearch querying with "IS NULL" and "IS NOT NULL" now uses MissingFilter and ExistsFilter.
- * [METAMODEL-190] - Improved decimal number support in Excel module.
- * [METAMODEL-187] - Improved memory consumption of Excel module by passing random-access-file handles to POI when possible.
- * [METAMODEL-191] - Resolved a number of dependency conflicts/overlaps when combining multiple MetaModel modules.
- * [METAMODEL-157] - Fixed an issue in DELETE FROM statements with WHERE clauses requiring client-side data type conversion on JDBC databases.
- * [METAMODEL-182] - Improved HdfsResource and FileResource directory-based implementations by adding also getSize() and getLastModified() directory-based implementations.
-
-### Apache MetaModel 4.3.6
-
- * [METAMODEL-161] - Upgraded HBase client API to version 1.1.1
- * [METAMODEL-160] - Added support for Apache Hive via the JDBC module of MetaModel.
- * [METAMODEL-162] - Made HdfsResource Serializable and added property getters
- * [METAMODEL-163] - Made FileResource and HdfsResource work with folders containing file chunks instead of only single files
- * [METAMODEL-104] - Added 'hadoop' and 'hbase' modules to dependencies of 'MetaModel-full'.
- * [METAMODEL-164] - Fixed a bug in data type parsing of ElasticSearch mapping document.
-
-### Apache MetaModel 4.3.5
-
- * [METAMODEL-148] - Added a 'hadoop' module with a HdfsResource class to allow CSV, Excel and Fixed-width file access on HDFS.
- * [METAMODEL-152] - Fixed an issue of not clearing schema cache when refreshSchemas() is invoked.
- * [METAMODEL-149] - Added support for COUNTER data type in Cassandra.
- * [METAMODEL-151] - Added support for DOUBLE data type mapping in PostgreSQL
- * [METAMODEL-154] - Use embedded Cassandra server for integration tests.
-
-### Apache MetaModel 4.3.4
-
- * [METAMODEL-136] - Added LIKE operator native support (using conversion to regex) for MongoDB.
- * [METAMODEL-138] - Allow empty characters before AS keyword in query parsing.
- * [METAMODEL-141] - Improved mapping of ColumnType to SQL data types for Oracle, SQL Server, MySQL, DB2 and PostgreSQL
- * [METAMODEL-142] - Ensured that JDBC schema refreshes in an UpdateScript is using same Connection/Transaction as rest of operations
- * [METAMODEL-133] - Improved query parser support for multiple JOINs in same query.
- * [METAMODEL-140] - Fixed support for ElasticSearch mappings with additional property attributes.
-
-### Apache MetaModel 4.3.3
-
- * [METAMODEL-123] - Added compatibility with ElasticSearch version 1.4.x
- * [METAMODEL-93] - Added compatibility with Apache HBase version 1.0.0
- * [METAMODEL-124] - Invoked ElasticSearch cross-version incompatible methods via reflection
- * [METAMODEL-125] - Added support for comma-separated select items in Query.select(String) method argument.
- * [METAMODEL-128] - Fixed bug in DataSet ordering when aggregation functions are applied to non-JDBC modules.
- * [METAMODEL-131] - Added support for composite primary keys in JDBC CREATE TABLE statements.
-
-### Apache MetaModel 4.3.2
-
- * [METAMODEL-78] - Fixed a bug that caused SELECT DISTINCT to sometimes return duplicates records on certain DataContext implementations.
- * [METAMODEL-106] - Improved handling of invalid or non-existing index names passed to ElasticSearchDataContext
- * [METAMODEL-79] - Added update execution support on ElasticSearch module. Increased capability of pushing down WHERE items to ElasticSearch searches.
- * [METAMODEL-115] - Improved query parsing to allow lower-case function names, operators etc.
-
-### Apache MetaModel 4.3.1
-
- * [METAMODEL-100] - Fixed bug when having multiple columns of same name. Added column no. comparison when calling Column.equals(...).
-
-### Apache MetaModel 4.3.0-incubating
-
- * [METAMODEL-77] - New module 'elasticsearch' for connecting and modeling ElasticSearch indexes through MetaModel.
- * [METAMODEL-18] - New module 'cassandra' for connecting and modelling Apache Cassandra databases through MetaModel.
- * [METAMODEL-83] - Added new operator types: GREATER_THAN_OR_EQUAL ('>=') and LESS_THAN_OR_EQUAL ('<=').
- * [METAMODEL-92] - For JSON, MongoDB and CouchDB: Made it possible to specify column names referring nested fields such as "name.first" or "addresses[0].city".
- * [METAMODEL-76] - Query parser improved to handle filters without spaces inbetween operator and operands.
- * [METAMODEL-95] - Fixed a critical bug in the Salesforce.com module which caused all number values to be interpreted as '1'.
- * [METAMODEL-74] - Fixed a bug related to skipping blank values when applying an aggregate function (SUM, AVG etc.)
- * [METAMODEL-85] - Fixed a bug that caused NULL values to be evaluated with equal-sign in JDBC update and delete statements instead of 'IS NULL'.
-
-### Apache MetaModel 4.2.0-incubating
-
- * [METAMODEL-38] - New module 'json' for handling json files (containing JSON arrays of documents or line-delimited JSON documents)
- * [METAMODEL-54] - ColumnType converted from enum to interface to allow for further specialization in modules.
- * [METAMODEL-57] - Changed the column type VARCHAR into STRING for the modules: CSV, Fixedwidth, Excel and XML.
- * [METAMODEL-56] - Made separate column types for converted JDBC LOBs - "CLOB as String" and "BLOB as bytes".
- * [METAMODEL-46] - Improved row-lookup by primary key (ID) in CouchDB
- * [METAMODEL-58] - Fixed a bug related to using CreateTable class and primary keys not getting created.
- * [METAMODEL-3] - Improved writing of Byte-Order-Mark (BOM) for various encoding spelling variants.
- * [METAMODEL-70] - Made the build compatible with both JDK versions 6 and 7.
- * [METAMODEL-59] - Fixed a bug related to handling of date/time literals in MS SQL Server queries.
- * [METAMODEL-60] - Fixed a bug related to DISTINCT and TOP keywords in MS SQL Server queries.
- * [METAMODEL-45] - Improved and standardized way of handling integration test connection information towards external databases.
- * [METAMODEL-62] - Fixed a bug related to fault-tolerant handling of malformed CSV lines when reading CSVs in single-line mode
- * [METAMODEL-68] - Made it possible to create a CSV table without a header line in the file, if the user configures it.
- * [METAMODEL-67] - Upgraded Jackson (JSON library) dependency from org.codehaus namespace to the newer com.fasterxml namespace.
- * [METAMODEL-69] - Fixed issue with deserialization of ColumnType into the new interface instead of the old enum.
-
-### Apache MetaModel 4.1.0-incubating
-
- * [METAMODEL-13] - Added support for Apache HBase via the new module "MetaModel-hbase"
- * [METAMODEL-41] - Added a parser for SimpleTableDef objects (SimpleTableDefParser). It parses statements similar to CREATE TABLE statements, although without the "CREATE TABLE" prefix. For example: foo (bar INTEGER, baz VARCHAR)
- * [METAMODEL-11] - New module "MetaModel-spring" which adds a convenient FactoryBean to produce various types of DataContext objects based on externalizable parameters, for Spring framework users.
- * [METAMODEL-32] - Fixed thread-safety issue in Excel module when tables (sheets) metadata is updated.
- * [METAMODEL-47] - Fixed issue in Excel of loading schema if query is fired based on metadata from a previous DataContext instance.
- * [METAMODEL-35] - Improved query rewriting for DB2 when paged queries contain ORDER BY clause.
- * [METAMODEL-44] - Added an optional method for QueryPostprocessDataContext implementations to do a row-lookup by primary key value.
- * [METAMODEL-43] - Made CSV datastores skip empty lines in file instead of treating them of rows with null values.
- * [METAMODEL-39] - Added pooling of active/used Connections and PreparedStatements in JDBC compiled queries.
- * [METAMODEL-34] - Updated LICENSE file to not include bundled dependencies' licenses.
- * [METAMODEL-33] - Ensured that Apache Rat plugin for Maven is properly activated.
- * [METAMODEL-37] - Removed old site sources from project.
-
-### Apache MetaModel 4.0.0-incubating
-
- * [METAMODEL-9] - SalesforceDataSet is throwing exception for insert sql of record having date/time.
- * [METAMODEL-4] - Use folder name as schema name for file based DataContexts
- * [METAMODEL-5] - Faster CsvDataContext implementation for single-line values
- * [METAMODEL-26] - Provide endpoint URL with SalesforceDataContext
- * Upgraded Apache POI dependency to v. 3.9
- * Improved fluent Query builder API by adding string parameter based methods for joining tables
- * Added a utility ObjectInputStream for deserializing legacy MetaModel objects
- * Performance improvement to CSV reading when values are only single line based
- * Setting up the project on apache infrastructure
- * [METAMODEL-10] - Exclude Jackcess dependency (Access module) from MetaModel
- * Renaming the package hierarchy from org.eobjects.metamodel to org.apache.metamodel
- * [METAMODEL-29] - Fixed issue in CreateTable builder class, causing it to only support a single column definition
- * [METAMODEL-30] - Fixed issue with count(*) queries on CSV resources that does not provide a byte stream length
+### Apache MetaModel 4.5.3 (work in progress)
+
+ * [METAMODEL-235] - Fixed a bug related to handling of null or missing values in ElasticSearch using REST client.
+ * [METAMODEL-225] - Fixed support for nested objects and arrays in ElasticSearch using REST client.
+ * [METAMODEL-244] - Added ColumnNamingStrategies concept which allows custom column naming and column name overriding.
+ * [METAMODEL-242] - Fixed issue when de-serializing old enum-instances of FunctionType.
+ * [METAMODEL-247] - Added FixedWidthConfigurationReader for reading fixed width file metadata from external files.
+
+### Apache MetaModel 4.5.2
+
+ * [METAMODEL-236] - Made OperatorType and FunctionType Serializable to ensure that serialization of Query is possible.
+
+### Apache MetaModel 4.5.1
+
+ * [METAMODEL-227] - Fix for respecting CSV escape character also when no quote character is set.
+ * [METAMODEL-183] - MongoDB module split into three: common, Mongo2 and Mongo3 to allow use of either old or new MongoDB API.
+ * [METAMODEL-231] - Fixed a bug causing the Neo4j to represent the same table multiple times within a schema.
+ * [METAMODEL-228] - Fixed a bug causing Number.class to not be converted to ColumnType.NUMBER.
+
+### Apache MetaModel 4.5.0
+
+ * [METAMODEL-212] - New module for ElasticSearch via REST client.
+ * [METAMODEL-99] - New module for Neo4j connectivity with MetaModel.
+ * [METAMODEL-207] - Ensured the serializability of the SingleLineCsvRow class.
+ * [METAMODEL-211] - Fixed a bug related to lookup by primary key (_id) on MongoDB.
+ * [METAMODEL-216] - Added new aggregate functions: FIRST, LAST and RANDOM.
+ * [METAMODEL-195] - Added new function MAP_VALUE which allows extracting a nested value from within a key/value map field.
+ * [METAMODEL-15] - Query parser support for table names with space. Delimitters can be double quote or square brackets.
+ * [METAMODEL-215] - Improved the capability of NumberComparator to support Integer, Long, Double, BigInteger and other built-in Number classes.
+ * [METAMODEL-218] - Fixed conversion of STRING and NUMBER types to database-specific types in JDBC module.
+ * [METAMODEL-205] - Added validation of Excel sheet name before attempting to create table (sheet).
+ * [METAMODEL-219] - Made HdfsResource capable of incorporating Hadoop configuration files core-site.xml and hdfs-site.xml
+ * [METAMODEL-220] - Made HdfsResource capable of working with other URI schemes than 'hdfs'.
+
+### Apache MetaModel 4.4.1
+
+ * [METAMODEL-198] - Fixed support for JDBC TIMESTAMP precision to match the underlying database's precision.
+ * [METAMODEL-200] - Added optional "APPROXIMATE" keyword to query syntax for aggregate functions.
+ * [METAMODEL-144] - Automated binary packaging of the MetaModel project.
+ * [METAMODEL-197] - ElasticSearch schema update/change after CREATE TABLE statements.
+ * [METAMODEL-199] - Fixed a bug in query parser when parsing two consecutive WHERE items with parentheses around them.
+ * [METAMODEL-203] - Upgraded MongoDB dependency version and API to the 3.x line.
+
+### Apache MetaModel 4.4.0
+
+ * [METAMODEL-192] - Added support for Scalar functions. We have a basic set of datatype conversion functions as well as support for UDF via implementing the ScalarFunction interface.
+ * [METAMODEL-194] - Added support for setting the "Max rows" flag of a query to 0. This will always return an empty dataset.
+ * [METAMODEL-173] - Improved CSV writing to non-file destinations. Added .write() and .append() methods to Resource interface.
+ * [METAMODEL-170] - Dropped support for Java 6.
+ * [METAMODEL-176] - Trimmed the transient dependencies of the JDBC module.
+ * [METAMODEL-178] - Added AggregateFunction and ScalarFunction interfaces. Changed FunctionType enum to be super-interface of those. Compatibility is retained but a recompile of code using FunctionType is needed.
+ * [METAMODEL-188] - Changed OperatorType enum to be an interface. Compatibility is retained but a recompile of code is needed.
+ * [METAMODEL-179] - Ensured that HdfsResource is not closing a shared HDFS file system reference.
+ * [METAMODEL-171] - Made integration tests for Cassandra module function properly in all environments.
+ * [METAMODEL-177] - Fixed a bug pertaining to the serializability of HdfsResource.
+ * [METAMODEL-172] - ElasticSearch Date types should be converted properly.
+ * [METAMODEL-184] - ElasticSearch querying with "IS NULL" and "IS NOT NULL" now uses MissingFilter and ExistsFilter.
+ * [METAMODEL-190] - Improved decimal number support in Excel module.
+ * [METAMODEL-187] - Improved memory consumption of Excel module by passing random-access-file handles to POI when possible.
+ * [METAMODEL-191] - Resolved a number of dependency conflicts/overlaps when combining multiple MetaModel modules.
+ * [METAMODEL-157] - Fixed an issue in DELETE FROM statements with WHERE clauses requiring client-side data type conversion on JDBC databases.
+ * [METAMODEL-182] - Improved HdfsResource and FileResource directory-based implementations by adding also getSize() and getLastModified() directory-based implementations.
+
+### Apache MetaModel 4.3.6
+
+ * [METAMODEL-161] - Upgraded HBase client API to version 1.1.1
+ * [METAMODEL-160] - Added support for Apache Hive via the JDBC module of MetaModel.
+ * [METAMODEL-162] - Made HdfsResource Serializable and added property getters
+ * [METAMODEL-163] - Made FileResource and HdfsResource work with folders containing file chunks instead of only single files
+ * [METAMODEL-104] - Added 'hadoop' and 'hbase' modules to dependencies of 'MetaModel-full'.
+ * [METAMODEL-164] - Fixed a bug in data type parsing of ElasticSearch mapping document.
+
+### Apache MetaModel 4.3.5
+
+ * [METAMODEL-148] - Added a 'hadoop' module with a HdfsResource class to allow CSV, Excel and Fixed-width file access on HDFS.
+ * [METAMODEL-152] - Fixed an issue of not clearing schema cache when refreshSchemas() is invoked.
+ * [METAMODEL-149] - Added support for COUNTER data type in Cassandra.
+ * [METAMODEL-151] - Added support for DOUBLE data type mapping in PostgreSQL
+ * [METAMODEL-154] - Use embedded Cassandra server for integration tests.
+
+### Apache MetaModel 4.3.4
+
+ * [METAMODEL-136] - Added LIKE operator native support (using conversion to regex) for MongoDB.
+ * [METAMODEL-138] - Allow empty characters before AS keyword in query parsing.
+ * [METAMODEL-141] - Improved mapping of ColumnType to SQL data types for Oracle, SQL Server, MySQL, DB2 and PostgreSQL
+ * [METAMODEL-142] - Ensured that JDBC schema refreshes in an UpdateScript is using same Connection/Transaction as rest of operations
+ * [METAMODEL-133] - Improved query parser support for multiple JOINs in same query.
+ * [METAMODEL-140] - Fixed support for ElasticSearch mappings with additional property attributes.
+
+### Apache MetaModel 4.3.3
+
+ * [METAMODEL-123] - Added compatibility with ElasticSearch version 1.4.x
+ * [METAMODEL-93] - Added compatibility with Apache HBase version 1.0.0
+ * [METAMODEL-124] - Invoked ElasticSearch cross-version incompatible methods via reflection
+ * [METAMODEL-125] - Added support for comma-separated select items in Query.select(String) method argument.
+ * [METAMODEL-128] - Fixed bug in DataSet ordering when aggregation functions are applied to non-JDBC modules.
+ * [METAMODEL-131] - Added support for composite primary keys in JDBC CREATE TABLE statements.
+
+### Apache MetaModel 4.3.2
+
+ * [METAMODEL-78] - Fixed a bug that caused SELECT DISTINCT to sometimes return duplicates records on certain DataContext implementations.
+ * [METAMODEL-106] - Improved handling of invalid or non-existing index names passed to ElasticSearchDataContext
+ * [METAMODEL-79] - Added update execution support on ElasticSearch module. Increased capability of pushing down WHERE items to ElasticSearch searches.
+ * [METAMODEL-115] - Improved query parsing to allow lower-case function names, operators etc.
+
+### Apache MetaModel 4.3.1
+
+ * [METAMODEL-100] - Fixed bug when having multiple columns of same name. Added column no. comparison when calling Column.equals(...).
+
+### Apache MetaModel 4.3.0-incubating
+
+ * [METAMODEL-77] - New module 'elasticsearch' for connecting and modeling ElasticSearch indexes through MetaModel.
+ * [METAMODEL-18] - New module 'cassandra' for connecting and modelling Apache Cassandra databases through MetaModel.
+ * [METAMODEL-83] - Added new operator types: GREATER_THAN_OR_EQUAL ('>=') and LESS_THAN_OR_EQUAL ('<=').
+ * [METAMODEL-92] - For JSON, MongoDB and CouchDB: Made it possible to specify column names referring nested fields such as "name.first" or "addresses[0].city".
+ * [METAMODEL-76] - Query parser improved to handle filters without spaces inbetween operator and operands.
+ * [METAMODEL-95] - Fixed a critical bug in the Salesforce.com module which caused all number values to be interpreted as '1'.
+ * [METAMODEL-74] - Fixed a bug related to skipping blank values when applying an aggregate function (SUM, AVG etc.)
+ * [METAMODEL-85] - Fixed a bug that caused NULL values to be evaluated with equal-sign in JDBC update and delete statements instead of 'IS NULL'.
+
+### Apache MetaModel 4.2.0-incubating
+
+ * [METAMODEL-38] - New module 'json' for handling json files (containing JSON arrays of documents or line-delimited JSON documents)
+ * [METAMODEL-54] - ColumnType converted from enum to interface to allow for further specialization in modules.
+ * [METAMODEL-57] - Changed the column type VARCHAR into STRING for the modules: CSV, Fixedwidth, Excel and XML.
+ * [METAMODEL-56] - Made separate column types for converted JDBC LOBs - "CLOB as String" and "BLOB as bytes".
+ * [METAMODEL-46] - Improved row-lookup by primary key (ID) in CouchDB
+ * [METAMODEL-58] - Fixed a bug related to using CreateTable class and primary keys not getting created.
+ * [METAMODEL-3] - Improved writing of Byte-Order-Mark (BOM) for various encoding spelling variants.
+ * [METAMODEL-70] - Made the build compatible with both JDK versions 6 and 7.
+ * [METAMODEL-59] - Fixed a bug related to handling of date/time literals in MS SQL Server queries.
+ * [METAMODEL-60] - Fixed a bug related to DISTINCT and TOP keywords in MS SQL Server queries.
+ * [METAMODEL-45] - Improved and standardized way of handling integration test connection information towards external databases.
+ * [METAMODEL-62] - Fixed a bug related to fault-tolerant handling of malformed CSV lines when reading CSVs in single-line mode
+ * [METAMODEL-68] - Made it possible to create a CSV table without a header line in the file, if the user configures it.
+ * [METAMODEL-67] - Upgraded Jackson (JSON library) dependency from org.codehaus namespace to the newer com.fasterxml namespace.
+ * [METAMODEL-69] - Fixed issue with deserialization of ColumnType into the new interface instead of the old enum.
+
+### Apache MetaModel 4.1.0-incubating
+
+ * [METAMODEL-13] - Added support for Apache HBase via the new module "MetaModel-hbase"
+ * [METAMODEL-41] - Added a parser for SimpleTableDef objects (SimpleTableDefParser). It parses statements similar to CREATE TABLE statements, although without the "CREATE TABLE" prefix. For example: foo (bar INTEGER, baz VARCHAR)
+ * [METAMODEL-11] - New module "MetaModel-spring" which adds a convenient FactoryBean to produce various types of DataContext objects based on externalizable parameters, for Spring framework users.
+ * [METAMODEL-32] - Fixed thread-safety issue in Excel module when tables (sheets) metadata is updated.
+ * [METAMODEL-47] - Fixed issue in Excel of loading schema if query is fired based on metadata from a previous DataContext instance.
+ * [METAMODEL-35] - Improved query rewriting for DB2 when paged queries contain ORDER BY clause.
+ * [METAMODEL-44] - Added an optional method for QueryPostprocessDataContext implementations to do a row-lookup by primary key value.
+ * [METAMODEL-43] - Made CSV datastores skip empty lines in file instead of treating them of rows with null values.
+ * [METAMODEL-39] - Added pooling of active/used Connections and PreparedStatements in JDBC compiled queries.
+ * [METAMODEL-34] - Updated LICENSE file to not include bundled dependencies' licenses.
+ * [METAMODEL-33] - Ensured that Apache Rat plugin for Maven is properly activated.
+ * [METAMODEL-37] - Removed old site sources from project.
+
+### Apache MetaModel 4.0.0-incubating
+
+ * [METAMODEL-9] - SalesforceDataSet is throwing exception for insert sql of record having date/time.
+ * [METAMODEL-4] - Use folder name as schema name for file based DataContexts
+ * [METAMODEL-5] - Faster CsvDataContext implementation for single-line values
+ * [METAMODEL-26] - Provide endpoint URL with SalesforceDataContext
+ * Upgraded Apache POI dependency to v. 3.9
+ * Improved fluent Query builder API by adding string parameter based methods for joining tables
+ * Added a utility ObjectInputStream for deserializing legacy MetaModel objects
+ * Performance improvement to CSV reading when values are only single line based
+ * Setting up the project on apache infrastructure
+ * [METAMODEL-10] - Exclude Jackcess dependency (Access module) from MetaModel
+ * Renaming the package hierarchy from org.eobjects.metamodel to org.apache.metamodel
+ * [METAMODEL-29] - Fixed issue in CreateTable builder class, causing it to only support a single column definition
+ * [METAMODEL-30] - Fixed issue with count(*) queries on CSV resources that does not provide a byte stream length
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java b/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java
index de78338..0696376 100644
--- a/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java
+++ b/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java
@@ -18,6 +18,8 @@
*/
package org.apache.metamodel.schema.naming;
+import java.util.List;
+
/**
* Constructors and common utilities for {@link ColumnNamingStrategy} objects.
*/
@@ -32,4 +34,12 @@ public class ColumnNamingStrategies {
public static ColumnNamingStrategy defaultStrategy() {
return DEFAULT_STRATEGY;
}
+
+ public static ColumnNamingStrategy customNames(List<String> columnNames) {
+ return new CustomColumnNamingStrategy(columnNames);
+ }
+
+ public static ColumnNamingStrategy customNames(String ... columnNames) {
+ return new CustomColumnNamingStrategy(columnNames);
+ }
}
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java b/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java
new file mode 100644
index 0000000..e6cc706
--- /dev/null
+++ b/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.schema.naming;
+
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A {@link ColumnNamingStrategy} that allows the user to supply his own list of
+ * custom column names.
+ */
+public class CustomColumnNamingStrategy implements ColumnNamingStrategy {
+
+ private static final long serialVersionUID = 1L;
+
+ private final List<String> columnNames;
+
+ public CustomColumnNamingStrategy(List<String> columnNames) {
+ this.columnNames = columnNames;
+ }
+
+ public CustomColumnNamingStrategy(String... columnNames) {
+ this(Arrays.asList(columnNames));
+ }
+
+ @Override
+ public ColumnNamingSession startColumnNamingSession() {
+ final Iterator<String> iterator = columnNames.iterator();
+ return new ColumnNamingSession() {
+
+ @Override
+ public String getNextColumnName(ColumnNamingContext ctx) {
+ if (iterator.hasNext()) {
+ return iterator.next();
+ }
+ return null;
+ }
+
+ @Override
+ public void close() {
+ }
+ };
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/pom.xml
----------------------------------------------------------------------
diff --git a/fixedwidth/pom.xml b/fixedwidth/pom.xml
index e667a10..43e4021 100644
--- a/fixedwidth/pom.xml
+++ b/fixedwidth/pom.xml
@@ -1,46 +1,52 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
- <parent>
- <artifactId>MetaModel</artifactId>
- <groupId>org.apache.metamodel</groupId>
- <version>4.5.3-SNAPSHOT</version>
- </parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>MetaModel-fixedwidth</artifactId>
- <name>MetaModel module for fixed width value files</name>
- <dependencies>
- <dependency>
- <groupId>org.apache.metamodel</groupId>
- <artifactId>MetaModel-core</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-nop</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <scope>test</scope>
- </dependency>
- </dependencies>
-</project>
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <parent>
+ <artifactId>MetaModel</artifactId>
+ <groupId>org.apache.metamodel</groupId>
+ <version>4.5.3-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>MetaModel-fixedwidth</artifactId>
+ <name>MetaModel module for fixed width value files</name>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.metamodel</groupId>
+ <artifactId>MetaModel-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.metamodel</groupId>
+ <artifactId>MetaModel-csv</artifactId>
+ <version>${project.version}</version>
+ <optional>true</optional>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-nop</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+</project>
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java
----------------------------------------------------------------------
diff --git a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java
new file mode 100644
index 0000000..65ec219
--- /dev/null
+++ b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.fixedwidth;
+
+import org.apache.metamodel.util.HasName;
+
+/**
+ * Represents the specification of a single column for a
+ * {@link FixedWidthDataContext}.
+ */
+public final class FixedWidthColumnSpec implements HasName {
+
+ private final String name;
+ private final int width;
+
+ public FixedWidthColumnSpec(String name, int width) {
+ this.name = name;
+ this.width = width;
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ public int getWidth() {
+ return width;
+ }
+}
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java
----------------------------------------------------------------------
diff --git a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java
index 6e9f0f1..2b2cae5 100644
--- a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java
+++ b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java
@@ -1,169 +1,189 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.metamodel.fixedwidth;
-
-import java.io.Serializable;
-import java.util.Arrays;
-import java.util.List;
-
-import org.apache.metamodel.data.DataSet;
-import org.apache.metamodel.schema.naming.ColumnNamingStrategies;
-import org.apache.metamodel.schema.naming.ColumnNamingStrategy;
-import org.apache.metamodel.util.BaseObject;
-import org.apache.metamodel.util.FileHelper;
-
-/**
- * Configuration of metadata about a fixed width values datacontext.
- */
-public final class FixedWidthConfiguration extends BaseObject implements
- Serializable {
-
- private static final long serialVersionUID = 1L;
-
- public static final int NO_COLUMN_NAME_LINE = 0;
- public static final int DEFAULT_COLUMN_NAME_LINE = 1;
-
- private final String encoding;
- private final int fixedValueWidth;
- private final int[] valueWidths;
- private final int columnNameLineNumber;
- private final boolean failOnInconsistentLineWidth;
- private final ColumnNamingStrategy columnNamingStrategy;
-
- public FixedWidthConfiguration(int fixedValueWidth) {
- this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING,
- fixedValueWidth);
- }
-
- public FixedWidthConfiguration(int[] valueWidth) {
- this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING, valueWidth,
- false);
- }
-
- public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth) {
- this(columnNameLineNumber, encoding, fixedValueWidth, false);
- }
-
- public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth,
- boolean failOnInconsistentLineWidth) {
- this.encoding = encoding;
- this.fixedValueWidth = fixedValueWidth;
- this.columnNameLineNumber = columnNameLineNumber;
- this.failOnInconsistentLineWidth = failOnInconsistentLineWidth;
- this.columnNamingStrategy = null;
- this.valueWidths = new int[0];
- }
-
- public FixedWidthConfiguration(int columnNameLineNumber, String encoding,
- int[] valueWidths, boolean failOnInconsistentLineWidth) {
- this(columnNameLineNumber, null, encoding, valueWidths, failOnInconsistentLineWidth);
- }
-
- public FixedWidthConfiguration(int columnNameLineNumber, ColumnNamingStrategy columnNamingStrategy, String encoding,
- int[] valueWidths, boolean failOnInconsistentLineWidth) {
- this.encoding = encoding;
- this.fixedValueWidth = -1;
- this.columnNameLineNumber = columnNameLineNumber;
- this.failOnInconsistentLineWidth = failOnInconsistentLineWidth;
- this.columnNamingStrategy = columnNamingStrategy;
- this.valueWidths = valueWidths;
- }
-
- /**
- * The line number (1 based) from which to get the names of the columns.
- *
- * @return an int representing the line number of the column headers/names.
- */
- public int getColumnNameLineNumber() {
- return columnNameLineNumber;
- }
-
- /**
- * Gets a {@link ColumnNamingStrategy} to use if needed.
- * @return
- */
- public ColumnNamingStrategy getColumnNamingStrategy() {
- if (columnNamingStrategy == null) {
- return ColumnNamingStrategies.defaultStrategy();
- }
- return columnNamingStrategy;
- }
-
- /**
- * Gets the file encoding to use for reading the file.
- *
- * @return the text encoding to use for reading the file.
- */
- public String getEncoding() {
- return encoding;
- }
-
- /**
- * Gets the width of each value within the fixed width value file.
- *
- * @return the fixed width to use when parsing the file.
- */
- public int getFixedValueWidth() {
- return fixedValueWidth;
- }
-
- public int[] getValueWidths() {
- return valueWidths;
- }
-
- /**
- * Determines if the {@link DataSet#next()} should throw an exception in
- * case of inconsistent line width in the fixed width value file.
- *
- * @return a boolean indicating whether or not to fail on inconsistent line
- * widths.
- */
- public boolean isFailOnInconsistentLineWidth() {
- return failOnInconsistentLineWidth;
- }
-
- @Override
- protected void decorateIdentity(List<Object> identifiers) {
- identifiers.add(columnNameLineNumber);
- identifiers.add(encoding);
- identifiers.add(fixedValueWidth);
- identifiers.add(valueWidths);
- identifiers.add(failOnInconsistentLineWidth);
- }
-
- @Override
- public String toString() {
- return "FixedWidthConfiguration[encoding=" + encoding
- + ", fixedValueWidth=" + fixedValueWidth + ", valueWidths="
- + Arrays.toString(valueWidths) + ", columnNameLineNumber="
- + columnNameLineNumber + ", failOnInconsistentLineWidth="
- + failOnInconsistentLineWidth + "]";
- }
-
- public boolean isConstantValueWidth() {
- return fixedValueWidth != -1;
- }
-
- public int getValueWidth(int columnIndex) {
- if (isConstantValueWidth()) {
- return fixedValueWidth;
- }
- return valueWidths[columnIndex];
- }
-}
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.fixedwidth;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.schema.naming.ColumnNamingStrategies;
+import org.apache.metamodel.schema.naming.ColumnNamingStrategy;
+import org.apache.metamodel.util.BaseObject;
+import org.apache.metamodel.util.CollectionUtils;
+import org.apache.metamodel.util.FileHelper;
+import org.apache.metamodel.util.HasNameMapper;
+
+/**
+ * Configuration of metadata about a fixed width values datacontext.
+ */
+public final class FixedWidthConfiguration extends BaseObject implements
+ Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ public static final int NO_COLUMN_NAME_LINE = 0;
+ public static final int DEFAULT_COLUMN_NAME_LINE = 1;
+
+ private final String encoding;
+ private final int fixedValueWidth;
+ private final int[] valueWidths;
+ private final int columnNameLineNumber;
+ private final boolean failOnInconsistentLineWidth;
+ private final ColumnNamingStrategy columnNamingStrategy;
+
+ public FixedWidthConfiguration(int fixedValueWidth) {
+ this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING,
+ fixedValueWidth);
+ }
+
+ public FixedWidthConfiguration(int[] valueWidth) {
+ this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING, valueWidth,
+ false);
+ }
+
+ public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth) {
+ this(columnNameLineNumber, encoding, fixedValueWidth, false);
+ }
+
+ public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth,
+ boolean failOnInconsistentLineWidth) {
+ this.encoding = encoding;
+ this.fixedValueWidth = fixedValueWidth;
+ this.columnNameLineNumber = columnNameLineNumber;
+ this.failOnInconsistentLineWidth = failOnInconsistentLineWidth;
+ this.columnNamingStrategy = null;
+ this.valueWidths = new int[0];
+ }
+
+ public FixedWidthConfiguration(int columnNameLineNumber, String encoding,
+ int[] valueWidths, boolean failOnInconsistentLineWidth) {
+ this(columnNameLineNumber, null, encoding, valueWidths, failOnInconsistentLineWidth);
+ }
+
+ public FixedWidthConfiguration(int columnNameLineNumber, ColumnNamingStrategy columnNamingStrategy, String encoding,
+ int[] valueWidths, boolean failOnInconsistentLineWidth) {
+ this.encoding = encoding;
+ this.fixedValueWidth = -1;
+ this.columnNameLineNumber = columnNameLineNumber;
+ this.failOnInconsistentLineWidth = failOnInconsistentLineWidth;
+ this.columnNamingStrategy = columnNamingStrategy;
+ this.valueWidths = valueWidths;
+ }
+
+ public FixedWidthConfiguration(String encoding, List<FixedWidthColumnSpec> columnSpecs) {
+ this(encoding, columnSpecs, false);
+ }
+
+ public FixedWidthConfiguration(String encoding, List<FixedWidthColumnSpec> columnSpecs,
+ boolean failOnInconsistentLineWidth) {
+ this.encoding = encoding;
+ this.fixedValueWidth = -1;
+ this.columnNameLineNumber = NO_COLUMN_NAME_LINE;
+ this.columnNamingStrategy = ColumnNamingStrategies.customNames(CollectionUtils.map(columnSpecs,
+ new HasNameMapper()));
+ this.valueWidths = new int[columnSpecs.size()];
+ for (int i = 0; i < valueWidths.length; i++) {
+ valueWidths[i] = columnSpecs.get(i).getWidth();
+ }
+ this.failOnInconsistentLineWidth = failOnInconsistentLineWidth;
+ }
+
+ /**
+ * The line number (1 based) from which to get the names of the columns.
+ *
+ * @return an int representing the line number of the column headers/names.
+ */
+ public int getColumnNameLineNumber() {
+ return columnNameLineNumber;
+ }
+
+ /**
+ * Gets a {@link ColumnNamingStrategy} to use if needed.
+ * @return
+ */
+ public ColumnNamingStrategy getColumnNamingStrategy() {
+ if (columnNamingStrategy == null) {
+ return ColumnNamingStrategies.defaultStrategy();
+ }
+ return columnNamingStrategy;
+ }
+
+ /**
+ * Gets the file encoding to use for reading the file.
+ *
+ * @return the text encoding to use for reading the file.
+ */
+ public String getEncoding() {
+ return encoding;
+ }
+
+ /**
+ * Gets the width of each value within the fixed width value file.
+ *
+ * @return the fixed width to use when parsing the file.
+ */
+ public int getFixedValueWidth() {
+ return fixedValueWidth;
+ }
+
+ public int[] getValueWidths() {
+ return valueWidths;
+ }
+
+ /**
+ * Determines if the {@link DataSet#next()} should throw an exception in
+ * case of inconsistent line width in the fixed width value file.
+ *
+ * @return a boolean indicating whether or not to fail on inconsistent line
+ * widths.
+ */
+ public boolean isFailOnInconsistentLineWidth() {
+ return failOnInconsistentLineWidth;
+ }
+
+ @Override
+ protected void decorateIdentity(List<Object> identifiers) {
+ identifiers.add(columnNameLineNumber);
+ identifiers.add(encoding);
+ identifiers.add(fixedValueWidth);
+ identifiers.add(valueWidths);
+ identifiers.add(failOnInconsistentLineWidth);
+ }
+
+ @Override
+ public String toString() {
+ return "FixedWidthConfiguration[encoding=" + encoding
+ + ", fixedValueWidth=" + fixedValueWidth + ", valueWidths="
+ + Arrays.toString(valueWidths) + ", columnNameLineNumber="
+ + columnNameLineNumber + ", failOnInconsistentLineWidth="
+ + failOnInconsistentLineWidth + "]";
+ }
+
+ public boolean isConstantValueWidth() {
+ return fixedValueWidth != -1;
+ }
+
+ public int getValueWidth(int columnIndex) {
+ if (isConstantValueWidth()) {
+ return fixedValueWidth;
+ }
+ return valueWidths[columnIndex];
+ }
+}
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java
----------------------------------------------------------------------
diff --git a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java
new file mode 100644
index 0000000..9154e5e
--- /dev/null
+++ b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java
@@ -0,0 +1,180 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.fixedwidth;
+
+import java.io.BufferedReader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metamodel.csv.CsvConfiguration;
+import org.apache.metamodel.csv.CsvDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.Action;
+import org.apache.metamodel.util.Resource;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Object capable of reading fixed width metadata from external sources and
+ * thereby producing an appropriate {@link FixedWidthConfiguration} to use with
+ * a {@link FixedWidthDataContext}.
+ */
+public class FixedWidthConfigurationReader {
+
+ private static final Logger logger = LoggerFactory.getLogger(FixedWidthConfigurationReader.class);
+
+ // example: @1 COL1 $char1.
+ private final Pattern PATTERN_SAS_INPUT_LINE = Pattern.compile("\\@(\\d+) (.+) .*?(\\d+)\\.");
+
+ // example: COL1 "Record type"
+ private final Pattern PATTERN_SAS_LABEL_LINE = Pattern.compile("(.+) \\\"(.+)\\\"");
+
+ /**
+ * Reads a {@link FixedWidthConfiguration} based on a SAS 'format file',
+ * <a href=
+ * "http://support.sas.com/documentation/cdl/en/etlug/67323/HTML/default/viewer.htm#p0h03yig7fp1qan1arghp3lwjqi6.htm">
+ * described here</a>.
+ *
+ * @param encoding
+ * @param resource
+ * the format file resource
+ * @param failOnInconsistentLineWidth
+ * @return a {@link FixedWidthConfiguration} object to use
+ */
+ public FixedWidthConfiguration readFromSasFormatFile(String encoding, Resource resource,
+ boolean failOnInconsistentLineWidth) {
+ final List<FixedWidthColumnSpec> columnSpecs = new ArrayList<>();
+
+ final CsvDataContext dataContext = new CsvDataContext(resource, new CsvConfiguration());
+ final Table table = dataContext.getDefaultSchema().getTable(0);
+ try (final DataSet dataSet = dataContext.query().from(table).select("Name", "BeginPosition", "EndPosition")
+ .execute()) {
+ while (dataSet.next()) {
+ final String name = (String) dataSet.getRow().getValue(0);
+ final int beginPosition = Integer.parseInt((String) dataSet.getRow().getValue(1));
+ final int endPosition = Integer.parseInt((String) dataSet.getRow().getValue(2));
+ final int width = 1 + endPosition - beginPosition;
+ columnSpecs.add(new FixedWidthColumnSpec(name, width));
+ }
+ }
+
+ return new FixedWidthConfiguration(encoding, columnSpecs, failOnInconsistentLineWidth);
+ }
+
+ /**
+ * Reads a {@link FixedWidthConfiguration} based on a SAS INPUT declaration.
+ * The reader method also optionally will look for a LABEL defintion for
+ * column naming.
+ *
+ * @param encoding
+ * @param resource
+ * the format file resource
+ * @param failOnInconsistentLineWidth
+ * @return a {@link FixedWidthConfiguration} object to use
+ */
+ public FixedWidthConfiguration readFromSasInputDefinition(String encoding, Resource resource,
+ boolean failOnInconsistentLineWidth) {
+
+ final Map<String, Integer> inputWidthDeclarations = new LinkedHashMap<>();
+ final Map<String, String> labelDeclarations = new HashMap<>();
+
+ resource.read(new Action<InputStream>() {
+
+ private boolean inInputSection = false;
+ private boolean inLabelSection = false;
+
+ @Override
+ public void run(InputStream in) throws Exception {
+ try (final BufferedReader reader = new BufferedReader(new InputStreamReader(in))) {
+ for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+ processLine(line);
+ }
+ }
+ }
+
+ private void processLine(String line) {
+ line = line.trim();
+ if (line.isEmpty()) {
+ return;
+ }
+ if (";".equals(line)) {
+ inInputSection = false;
+ inLabelSection = false;
+ return;
+ } else if ("INPUT".equals(line)) {
+ inInputSection = true;
+ return;
+ } else if ("LABEL".equals(line)) {
+ inLabelSection = true;
+ return;
+ }
+
+ if (inInputSection) {
+ final Matcher matcher = PATTERN_SAS_INPUT_LINE.matcher(line);
+ if (matcher.matches()) {
+ final String positionSpec = matcher.group(1);
+ final String nameSpec = matcher.group(2);
+ final int width = Integer.parseInt(matcher.group(3));
+ logger.debug("Parsed INPUT line \"{}\": position={}, name={}, width={}", line, positionSpec,
+ nameSpec, width);
+ inputWidthDeclarations.put(nameSpec, width);
+ } else {
+ logger.debug("Failed to parse/recognize INPUT line \"{}\"", line);
+ }
+ } else if (inLabelSection) {
+ final Matcher matcher = PATTERN_SAS_LABEL_LINE.matcher(line);
+ if (matcher.matches()) {
+ final String nameSpec = matcher.group(1);
+ final String labelSpec = matcher.group(2);
+ logger.debug("Parsed LABEL line \"{}\": name={}, label={}", line, nameSpec, labelSpec);
+ labelDeclarations.put(nameSpec, labelSpec);
+ } else {
+ logger.debug("Failed to parse/recognize LABEL line \"{}\"", line);
+ }
+ }
+
+ if (line.endsWith(";")) {
+ inInputSection = false;
+ inLabelSection = false;
+ }
+ }
+ });
+
+ final List<FixedWidthColumnSpec> columnSpecs = new ArrayList<>();
+ for (Entry<String, Integer> entry : inputWidthDeclarations.entrySet()) {
+ final String columnKey = entry.getKey();
+ final Integer columnWidth = entry.getValue();
+ final String columnLabel = labelDeclarations.get(columnKey);
+ final String columnName = columnLabel == null ? columnKey : columnLabel;
+ columnSpecs.add(new FixedWidthColumnSpec(columnName, columnWidth));
+ }
+
+ return new FixedWidthConfiguration(encoding, columnSpecs, failOnInconsistentLineWidth);
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java
----------------------------------------------------------------------
diff --git a/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java b/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java
new file mode 100644
index 0000000..eb57233
--- /dev/null
+++ b/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.fixedwidth;
+
+import static org.junit.Assert.*;
+
+import java.util.Arrays;
+
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.FileResource;
+import org.apache.metamodel.util.Resource;
+import org.junit.Test;
+
+public class FixedWidthConfigurationReaderTest {
+
+ private final FileResource dataResource = new FileResource("src/test/resources/metadata_spec1/data.txt");
+
+ @Test
+ public void testReadConfigurationFromSasFormatFile() throws Exception {
+ final FixedWidthConfigurationReader reader = new FixedWidthConfigurationReader();
+ final Resource resource = new FileResource("src/test/resources/metadata_spec1/sas-formatfile-metadata.txt");
+ assertTrue(resource.isExists());
+
+ final FixedWidthConfiguration configuration = reader.readFromSasFormatFile("UTF8", resource, false);
+ assertEquals("[1, 20, 2]", Arrays.toString(configuration.getValueWidths()));
+
+ final FixedWidthDataContext dataContext = new FixedWidthDataContext(dataResource, configuration);
+
+ performAssertionsOnSpec1(dataContext);
+ }
+
+ @Test
+ public void testReadConfigurationFromSasInputMetadata() throws Exception {
+ final FixedWidthConfigurationReader reader = new FixedWidthConfigurationReader();
+ final Resource resource = new FileResource("src/test/resources/metadata_spec1/sas-input-metadata.txt");
+ assertTrue(resource.isExists());
+
+ final FixedWidthConfiguration configuration = reader.readFromSasInputDefinition("UTF8", resource, false);
+ assertEquals("[1, 20, 2]", Arrays.toString(configuration.getValueWidths()));
+
+ final FixedWidthDataContext dataContext = new FixedWidthDataContext(dataResource, configuration);
+
+ performAssertionsOnSpec1(dataContext);
+ }
+
+ /**
+ * Shared assertions section once the 'metadata_spec1' {@link DataContext}
+ * has been loaded.
+ *
+ * @param dataContext
+ */
+ private void performAssertionsOnSpec1(FixedWidthDataContext dataContext) {
+ final Table table = dataContext.getDefaultSchema().getTable(0);
+ final String[] columnNames = table.getColumnNames();
+ assertEquals("[Record type, Description, Initials]", Arrays.toString(columnNames));
+
+ try (final DataSet dataSet = dataContext.query().from(table).selectAll().execute()) {
+ assertTrue(dataSet.next());
+ assertEquals("Row[values=[P, Kasper Sorensen, KS]]", dataSet.getRow().toString());
+ assertTrue(dataSet.next());
+ assertEquals("Row[values=[C, Human Inference, HI]]", dataSet.getRow().toString());
+ assertTrue(dataSet.next());
+ assertEquals("Row[values=[P, Ankit Kumar, AK]]", dataSet.getRow().toString());
+ assertTrue(dataSet.next());
+ assertEquals("Row[values=[C, Stratio, S]]", dataSet.getRow().toString());
+ assertTrue(dataSet.next());
+ assertEquals("Row[values=[U, Unknown, ]]", dataSet.getRow().toString());
+ assertFalse(dataSet.next());
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/resources/metadata_spec1/data.txt
----------------------------------------------------------------------
diff --git a/fixedwidth/src/test/resources/metadata_spec1/data.txt b/fixedwidth/src/test/resources/metadata_spec1/data.txt
new file mode 100644
index 0000000..785a539
--- /dev/null
+++ b/fixedwidth/src/test/resources/metadata_spec1/data.txt
@@ -0,0 +1,5 @@
+PKasper Sorensen KS
+CHuman Inference HI
+PAnkit Kumar AK
+CStratio S
+UUnknown
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt
----------------------------------------------------------------------
diff --git a/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt b/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt
new file mode 100644
index 0000000..9bbe411
--- /dev/null
+++ b/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt
@@ -0,0 +1,4 @@
+Name,SASColumnType,BeginPosition,EndPosition,ReadFlag,Desc,SASFormat,SASInformat
+Record type,C,1,1,y,Record Type,$char.,$char.
+Description,C,2,21,y,Description of record,$char.,$char.
+Initials,C,22,23,y,Initials of record,,
http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt
----------------------------------------------------------------------
diff --git a/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt b/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt
new file mode 100644
index 0000000..f12e418
--- /dev/null
+++ b/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt
@@ -0,0 +1,19 @@
+INPUT
+
+ @1 COL1 $char1.
+
+ @2 COL2 $char20.
+
+ @22 COL3 $char2.
+
+;
+
+LABEL
+
+ COL1 "Record type"
+
+ COL2 "Description"
+
+ COL3 "Initials"
+
+;