You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metamodel.apache.org by rafaelgarrote <gi...@git.apache.org> on 2015/04/20 17:36:05 UTC

[GitHub] metamodel pull request: Csv

GitHub user rafaelgarrote opened a pull request:

    https://github.com/apache/metamodel/pull/17

    Csv

    Hi!
    In csv the metadata for all csv fields are STRING and the values are always String type.
    I added some changes to get real metadata csv fields and the cast value for each field.
    I made some new test and changes in others that sould fail without changes and works fine with changes.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rafaelgarrote/metamodel CSV

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metamodel/pull/17.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #17
    
----
commit 78408c077d2f1a41e8a923c2264c63965deedf83
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-04T14:00:58Z

    updated findFilterItem method in order to find either upper and lowercase filter operator types

commit ec6b30bbb6fa6576d9ccf73383b3b537bda9b8a4
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-04T14:23:55Z

    updated findSelectItem method in order to find either upper and lowercase function types

commit fd851bfaa7c988ff7e10810a7c0084ef5e094f63
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-09T17:28:21Z

    updated parse method in order to use distinct in either upper and lower case

commit 52acb8cae92e82a51045021ade0bbc1bdbc04993
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-09T17:29:35Z

    Some test added to test IN, LIKE, DISTINCT, MIN, AVG... in lower case

commit 99bbcc380ff5711b13953553aedecc807027c1fe
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-11T16:24:04Z

    changes in DelimOcurrence to use delim either lower and upper case

commit ad3271ac36155b938aabbbdc3bbdc2fd2da0d0e2
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-11T17:11:54Z

    Merge branch 'master' of https://github.com/apache/metamodel

commit df5d3d89f89947146921f9db0a3f5378ece48903
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-17T09:38:46Z

    Real metadata

commit 0377d866baade0b54ff704a103b39f6014b936dc
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-17T09:55:03Z

    New test added to test Delim in where clause into lowercase

commit aff29ba2b9bc08df5fdd65aa4ec629c4e383b843
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-17T17:38:56Z

    Get real medatadata type and cast values into its real objects

commit 2addd8dcb1d37593518213b66e4db67d42a74c42
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-20T11:25:40Z

    test fixed with Type cast changes and some new test

commit f9f2bc07f30662617f13f587ee703a2133434508
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-04-07T14:36:56Z

    Merge branch 'master' of https://github.com/apache/metamodel

commit 6d36808c8d11040cb7f0b0136ea87c0667c09b24
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-04-20T14:32:08Z

    Merge branch 'master' of https://github.com/apache/metamodel

commit d8c99bf536c04f8c0f46de1f0379037191e8fe45
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-17T09:38:46Z

    Real metadata

commit 51350fa808b0f7843a11171ecf7b3140bee8a380
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-17T17:38:56Z

    Get real medatadata type and cast values into its real objects

commit a22ee080716558abae3d17b5991a9e4f9aeded09
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-03-20T11:25:40Z

    test fixed with Type cast changes and some new test

commit 693e2ec8c93074c706729aacb1fe640d40c83e00
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-04-20T15:28:28Z

    query part parser

commit f037d592df7c07666a6de9e54d56753f0913d9af
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-04-20T15:29:08Z

    Merge branch 'CSV' of https://github.com/rafaelgarrote/metamodel into CSV

commit 37190bab3da3d198e2c9d23df58bb2cf67c48404
Author: Rafael Garrote <rg...@stratio.com>
Date:   2015-04-20T15:31:36Z

    query part parser

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-122785337
  
    Could we either close this PR or continue discussing it? It seems it has gone a bit stale :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by albertostratio <gi...@git.apache.org>.
Github user albertostratio commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-104547058
  
    I've been having a look at these changes and IMHO they address quite well the problem that Kasper identified. They are also pretty consinstent with the approach taken in the MongoDB connector.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-104927693
  
    My worry is still there I am afraid. With the change in CsvDataSet there is now a call to the cast(...) method which casts/narrows the object type on a value-by-value basis. That means that I can very easily be getting values of different types within the same column.
    
    In my opinion it would be much better if we during the analysis of the data set a ColumnType on the Column definition. And then we _only_ do the casting as per the column type. That way we ensure that it is "always or never" that we change the objects we read.
    
    Another aspect that I'd like to request is that this feature can be turned on or off in CsvConfiguration. For the application that I am working on CSV is one of the most important data source types and we have some cases of processing billions of CSV records. If there is just a single "foo bar" value in the middle of such a file with otherwise entirely numbers or so, then we need a way to turn off the casting so that the consumer of the data at least does not get inconsistent value types (or loss of data in case the cast would omit the "foo bar" value).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by rafaelgarrote <gi...@git.apache.org>.
Github user rafaelgarrote commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-104356861
  
    Hi Kasper, I have made some changes in code, and now I evaluate the first 1000 CSV rows in order to get the column types. Maybe this solution is less fragile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-94541818
  
    Hi Rafael,
    
    Thank you for the contribution, really appreciate the work. I am aware of the issue but I think this solution is a bit too fragile. It only reads the first line to determine the data type and it also does type casting/narrowing on each single value instead of trying to establish a data type and only narrow according to that type. That essentially means that as a user I have no guarantee that row 1 is going to have the same data types as row 2. That feels very unsafe to me at least. I could effectively have a table with e.g. column type NUMBER which would make sense on the first row but on the second row I would get a String or something completely different.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by drexler42 <gi...@git.apache.org>.
Github user drexler42 commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-122790553
  
    I'll just add my 2 cents... The CSV data type is not "typed", like a data base. It would be nice if the user can impose a data model on top of the CSV file, to express his intention. For example, the CSV file could be a database dump file. The user would then like to declare String, Numerical and Boolean columns.
    
    The first thing the user wants is then to declare the column types. Detecting column types is much less needed IMHO. The "foo bar" would result in a type exception for the row.
    
    Next, we could think about handling exceptional cases like:
    - Handling missing values with some sort of coalesce function to handling missing values;
    - Converting invalid values into some special value;
    
    This does not conflict with Kaspers notion of changing the column types. But it is a bit less ambitious. I would be very content if it were possible to declare column types statically.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:

    https://github.com/apache/metamodel/pull/17#discussion_r28717818
  
    --- Diff: csv/src/main/java/org/apache/metamodel/csv/CsvDataContext.java ---
    @@ -196,7 +196,7 @@ public CsvDataContext(InputStream inputStream, char separatorChar, char quoteCha
         }
     
         /**
    -     * Gets the CSV configuration used
    +     * tts the CSV configuration used
    --- End diff --
    
    Eh? ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:

    https://github.com/apache/metamodel/pull/17#discussion_r28717852
  
    --- Diff: csv/src/main/java/org/apache/metamodel/csv/CsvDataSet.java ---
    @@ -20,6 +20,7 @@
     
     import java.io.IOException;
     
    +import com.sun.org.apache.xpath.internal.operations.Bool;
    --- End diff --
    
    Eh? ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/metamodel/pull/17


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metamodel pull request: Csv

Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the pull request:

    https://github.com/apache/metamodel/pull/17#issuecomment-104928231
  
    Thinking about this some more, I guess the main principle that I am trying to enforce is:
    
    _metadata about a column must be consistent with the values you get when you query that column_
    
    In other words, it is currently consistent that ColumnType is always STRING because we always return a String value.
    
    I would be very open to a way of working with CSV files where you could specify/override the column type of certain columns. That would actually be a pretty cool feature - very related, but a different way to see it.
    
    If that would be the case then you could first use the untouched CsvDataContext to analyze the content of various columns. If you feel confident enough you might then switch the column type from STRING to something else. And when it is then set to eg. INTEGER then our cast method should automatically apply a Integer cast/conversion operation. Same with other types then.
    
    Would that maybe be a model that fits us all?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---