You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/20 03:27:17 UTC

[GitHub] [beam] haoyuche opened a new issue, #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

haoyuche opened a new issue, #23747:
URL: https://github.com/apache/beam/issues/23747

   ### What happened?
   
           PCollection<Row> rows = pipeline.apply(JdbcIO.<Row>readWithPartitions()
                   .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(
                           input.driverClassName, input.url)
                           .withUsername(input.username)
                           .withPassword(input.password))
                   .withTable(input.tableName)
                   .withPartitionColumn(input.primeKeyForPartitionRead)
                   .withRowOutput());
   
   Now there is a column named "name" which is VARCHAR, which has been read as beam type LOGICAL_TYPE NOT NULL.
   
   Then I do 
           Schema schemaAfterSql = Schema.builder().addNullableField("id", Schema.FieldType.INT64).addNullableField("name", Schema.FieldType.INT64).build();
           String selectQuery = "SELECT id, name FROM PCOLLECTION" ;
           PCollection<Row> processedRows = rows.apply("EnforcePolicy", SqlTransform.query(selectQuery)).setRowSchema(schemaAfterSql);
   It gets "An exception occured while executing the Java class. Unable to parse query SELECT id, policy_name FROM PCOLLECTION: java.lang.IllegalArgumentException: Cannot find a matching Calcite SqlTypeName for Beam type: LOGICAL_TYPE NOT NULL"
   
   ### Issue Priority
   
   Priority: 0
   
   ### Issue Component
   
   Component: io-java-jdbc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1289148391

   Thanks for reporting this. It looks like the cause is CalciteUtil here: https://github.com/apache/beam/blob/4ffeae4d2b800f2df36d2ea2eab549f2204d5691/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L224
   does not consider the logical type FieldType.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] haoyuche commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
haoyuche commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1289717400

   Yes, I think somehow we should not feed LOGICAL_TYPE to CalciteUtils but instead we should feed the  [BaseT](https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L547) so that it can handle the normal sql types.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1289149247

   .remove-label P0, "awaiting triage"
   
   .add_label P2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1289346904

   .remove-labels 'awaiting triage'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1289345418

   .remove-labels P0, "awaiting triage"
   
   .set_labels P2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] haoyuche commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
haoyuche commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1314643692

   Thanks Andrew for fixing this issue!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] apilloud closed issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
apilloud closed issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName
URL: https://github.com/apache/beam/issues/23747


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1289418341

   Label  cannot be managed because it does not exist in the repo. Please check your spelling.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #23747: [Bug]: After JDBCIO read withRowOutput(), the VARCHAR/TEXT -> LOGICAL_TYPE and not compatible with SqlTypeName

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23747:
URL: https://github.com/apache/beam/issues/23747#issuecomment-1295658717

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org