You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/07/30 12:07:43 UTC
[GitHub] [flink] shuiqiangchen opened a new pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
shuiqiangchen opened a new pull request #13029:
URL: https://github.com/apache/flink/pull/13029
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Supports basic TypeInformation including BasicTypeInfo, LocalTimeTypeInfo, PrimitiveArrayTypeInfo, RowTypeInfo.
Types.ROW()/Types.ROW_NAMED()/Types.PRIMITIVE_ARRAY() should also be supported.
## Brief change log
- Add python wrapper class for BasicTypeInformation, PrimitiveArrayTypeInfos and RowTypeInfo, etc.
- Introduce a new type info named PickledArrayTypeInfo to indicate the type of bytes array data derived from pickled python objects.
-Add utils to convert python typeInfo and java typeInfo.
## Verifying this change
This change is already covered by existing tests, such as:
- TypeInfoTests in test_typeinfo.py
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): ( no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? ( not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052)
* fd04ce4ce52a282193a91cc109598ff7df1a0117 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-668490867
@hequn8128 Thank you for your comments, I have updated the code, please take a look at it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39f5be60dc81c30f0cbbfb4e219722cbdd52ea21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070)
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 2d8d302ef214afc070f00d730d6c25c9509e3529 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r464164609
##########
File path: flink-python/pyflink/common/tests/test_typeinfo.py
##########
@@ -0,0 +1,118 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+import unittest
+
+from pyflink.common.typeinfo import Types, RowTypeInfo, TupleTypeInfo, _from_java_type
+
+
+class TypeInfoTests(unittest.TestCase):
+
+ def test_types(self):
+ self.assertEqual(True, Types.BOOLEAN().is_basic_type())
+ self.assertEqual(True, Types.BYTE().is_basic_type())
+ self.assertEqual(True, Types.SHORT().is_basic_type())
+ self.assertEqual(True, Types.INT().is_basic_type())
+ self.assertEqual(True, Types.LONG().is_basic_type())
+ self.assertEqual(True, Types.FLOAT().is_basic_type())
+ self.assertEqual(True, Types.DOUBLE().is_basic_type())
+ self.assertEqual(True, Types.CHAR().is_basic_type())
+ self.assertEqual(True, Types.BIG_INT().is_basic_type())
+ self.assertEqual(True, Types.BIG_DEC().is_basic_type())
+ self.assertEqual(False, Types.PICKLED_BYTE_ARRAY().is_basic_type())
+
+ self.assertEqual(False, Types.STRING().is_tuple_type())
+ self.assertEqual(1, Types.STRING().get_total_fields())
+ self.assertEqual(1, Types.STRING().get_arity())
+
+ self.assertEqual(False, Types.SQL_DATE().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIME().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIMESTAMP().is_basic_type())
+
+ def test_row_type(self):
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()])
+ .get_field_names(), ['f0', 'f1'])
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']).get_field_names(), ['a', 'b'])
+
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.STRING()], ['a', 'b']), True)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.INT()],
+ ['a', 'b']), False)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']).__str__(), "Row(a: String, b: String)")
+
+ self.assertEqual(Types.ROW([Types.STRING(), Types.STRING()]),
+ RowTypeInfo([Types.STRING(), Types.STRING()]), True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_names(), ['a', 'b'], True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_types(), [Types.STRING(), Types.STRING()], True)
+
+ def test_tuple_type(self):
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.INT()]), True)
+
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]).__str__(),
+ "Java Tuple2<String, Integer>")
+
+ self.assertNotEqual(TupleTypeInfo([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.BOOLEAN()]))
+
+ self.assertEqual(Types.TUPLE([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.INT()]))
+
+ self.assertEqual(Types.TUPLE([Types.STRING(), Types.INT()]).get_field_types(),
+ [Types.STRING(), Types.INT()])
+
+ def test_primitive_array_types(self):
+ primitive_int_array_type_info = Types.PRIMITIVE_ARRAY(Types.INT())
+ self.assertEqual(primitive_int_array_type_info.is_basic_type(), False)
+
+ def test_from_java_type(self):
+ basic_int_type_info = Types.INT()
+ self.assertEqual(basic_int_type_info,
+ _from_java_type(basic_int_type_info.get_java_type_info()))
+
+ row_type_info = Types.ROW([Types.INT(), Types.STRING()])
+ self.assertEqual(row_type_info, _from_java_type(row_type_info.get_java_type_info()))
+
+ tuple_type_info = Types.TUPLE([Types.CHAR(), Types.INT()])
+ self.assertEqual(tuple_type_info, _from_java_type(tuple_type_info.get_java_type_info()))
+
+ primitive_int_array_type_info = Types.PRIMITIVE_ARRAY(Types.INT())
+ self.assertEqual(primitive_int_array_type_info,
+ _from_java_type(primitive_int_array_type_info.get_java_type_info()))
+
+ pickled_byte_array_type_info = Types.PICKLED_BYTE_ARRAY()
+ self.assertEqual(pickled_byte_array_type_info,
+ _from_java_type(pickled_byte_array_type_info.get_java_type_info()))
+
+ sql_date_type_info = Types.SQL_DATE()
+ self.assertEqual(sql_date_type_info,
+ _from_java_type(sql_date_type_info.get_java_type_info()))
+
+
+
Review comment:
Thanks, I'll clean it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39f5be60dc81c30f0cbbfb4e219722cbdd52ea21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070)
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066)
* 39f5be60dc81c30f0cbbfb4e219722cbdd52ea21 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39f5be60dc81c30f0cbbfb4e219722cbdd52ea21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070)
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
}, {
"hash" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162",
"triggerID" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 943ff7e98177dec676272e44eae6e8fb1a1fcd2a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463382296
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
Review comment:
We can remove this line. Tuple type is supported in Python originally. We should not simply copy Java comments.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
Review comment:
We don't have case classes in PyFlink.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,561 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
+ """
+ Type Information for BigDecimal. This type includes `precision` and `scale`.
+ """
+
+ def __init__(self, precision, scale):
+ j_type_info = get_gateway().jvm \
+ .org.apache.flink.table.runtime.typeutils.BigDecimalTypeInfo(precision, scale)
+ super(BigDecimalTypeInfo, self).__init__(j_typeinfo=j_type_info)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types, field_names=None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self):
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name):
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ LOCAL_DATE = LocalTimeTypeInfo.LOCAL_DATE
+ LOCAL_TIME = LocalTimeTypeInfo.LOCAL_TIME
+ LOCAL_DATE_TIME = LocalTimeTypeInfo.LOCAL_DATE_TIME
+
+ INSTANT = BasicTypeInfo.INSTANT_TYPE_INFO
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
+ """
+ Returns type information for Row with fields of the given types and with given names. A row
+ must not be null.
+
+ :param names: array of field names.
+ :param types: array of field types.
+ """
+ return RowTypeInfo(types, names)
+
+ @staticmethod
+ def TUPLE(types):
Review comment:
Remove this method. Same as TupleTypeInfo.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
Review comment:
existt => exist
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
+ """
+ Type Information for BigDecimal. This type includes `precision` and `scale`.
+ """
+
+ def __init__(self, precision, scale):
+ j_type_info = get_gateway().jvm \
+ .org.apache.flink.table.runtime.typeutils.BigDecimalTypeInfo(precision, scale)
+ super(BigDecimalTypeInfo, self).__init__(j_typeinfo=j_type_info)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types, field_names=None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self):
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name):
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
Review comment:
Remove this type. We treat tuple as Row in PyFlink. For Python, tuple is a special kind of Row since it can also specify filed names and field values.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
Review comment:
Add type hint: `def is_basic_type(self) -> bool:`. Same for other methods.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
Review comment:
Decimal should exist in `BasicTypeInfo` together with `BIG_INT_TYPE_INFO`. The decimal type info in DataStream is different from table. The precision and scale are not fixed, i.e., not be specified during initialization.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
Review comment:
composite types
##########
File path: flink-python/src/main/java/org/apache/flink/datastream/typeinfo/python/PickledByteArrayTypeInfo.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.typeinfo.python;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+
+/**
+ * A PickledByteArrayTypeInfo indicates that the data of this type is a generated primitive byte
+ * array by pickle.
+ */
+public class PickledByteArrayTypeInfo extends TypeInformation<byte[]> {
Review comment:
Make it a Singleton, similar to BasicTypeInfo in which provides a static final instance.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,561 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
+ """
+ Type Information for BigDecimal. This type includes `precision` and `scale`.
+ """
+
+ def __init__(self, precision, scale):
+ j_type_info = get_gateway().jvm \
+ .org.apache.flink.table.runtime.typeutils.BigDecimalTypeInfo(precision, scale)
+ super(BigDecimalTypeInfo, self).__init__(j_typeinfo=j_type_info)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types, field_names=None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self):
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name):
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ LOCAL_DATE = LocalTimeTypeInfo.LOCAL_DATE
+ LOCAL_TIME = LocalTimeTypeInfo.LOCAL_TIME
+ LOCAL_DATE_TIME = LocalTimeTypeInfo.LOCAL_DATE_TIME
+
+ INSTANT = BasicTypeInfo.INSTANT_TYPE_INFO
Review comment:
Remove these types for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463451469
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
Review comment:
Ok, it's good to add type hints.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r464911311
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,523 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
Review comment:
ditto
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,523 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
Review comment:
We can remove these methods since we don't need them now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-668553682
@shuiqiangchen Thanks a lot for the update. Please check the test azure test failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463451278
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
Review comment:
Yes, I will remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463453354
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
Review comment:
Thank you! I'll revise it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463445937
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,561 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
+ """
+ Type Information for BigDecimal. This type includes `precision` and `scale`.
+ """
+
+ def __init__(self, precision, scale):
+ j_type_info = get_gateway().jvm \
+ .org.apache.flink.table.runtime.typeutils.BigDecimalTypeInfo(precision, scale)
+ super(BigDecimalTypeInfo, self).__init__(j_typeinfo=j_type_info)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types, field_names=None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self):
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name):
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ LOCAL_DATE = LocalTimeTypeInfo.LOCAL_DATE
+ LOCAL_TIME = LocalTimeTypeInfo.LOCAL_TIME
+ LOCAL_DATE_TIME = LocalTimeTypeInfo.LOCAL_DATE_TIME
+
+ INSTANT = BasicTypeInfo.INSTANT_TYPE_INFO
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
+ """
+ Returns type information for Row with fields of the given types and with given names. A row
+ must not be null.
+
+ :param names: array of field names.
+ :param types: array of field types.
+ """
+ return RowTypeInfo(types, names)
+
+ @staticmethod
+ def TUPLE(types):
Review comment:
Remove this method. Same as TupleTypeInfo.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
+ """
+ Type Information for BigDecimal. This type includes `precision` and `scale`.
+ """
+
+ def __init__(self, precision, scale):
+ j_type_info = get_gateway().jvm \
+ .org.apache.flink.table.runtime.typeutils.BigDecimalTypeInfo(precision, scale)
+ super(BigDecimalTypeInfo, self).__init__(j_typeinfo=j_type_info)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types, field_names=None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self):
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name):
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self):
+ return self.types
+
+ def __eq__(self, other):
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self):
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self):
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
Review comment:
Remove this type. We treat tuple as Row in PyFlink. For Python, tuple is a special kind of Row since it can also specify filed names and field values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666990874
Hi @hequn8128 , thank you for your comments, I have updated the pr according to your review suggestions, please have a look at it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666326352
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 (Thu Jul 30 12:09:44 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18763).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079)
* 2d8d302ef214afc070f00d730d6c25c9509e3529 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r464164717
##########
File path: flink-python/pyflink/common/tests/test_typeinfo.py
##########
@@ -0,0 +1,118 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+import unittest
+
+from pyflink.common.typeinfo import Types, RowTypeInfo, TupleTypeInfo, _from_java_type
+
+
+class TypeInfoTests(unittest.TestCase):
+
+ def test_types(self):
+ self.assertEqual(True, Types.BOOLEAN().is_basic_type())
+ self.assertEqual(True, Types.BYTE().is_basic_type())
+ self.assertEqual(True, Types.SHORT().is_basic_type())
+ self.assertEqual(True, Types.INT().is_basic_type())
+ self.assertEqual(True, Types.LONG().is_basic_type())
+ self.assertEqual(True, Types.FLOAT().is_basic_type())
+ self.assertEqual(True, Types.DOUBLE().is_basic_type())
+ self.assertEqual(True, Types.CHAR().is_basic_type())
+ self.assertEqual(True, Types.BIG_INT().is_basic_type())
+ self.assertEqual(True, Types.BIG_DEC().is_basic_type())
+ self.assertEqual(False, Types.PICKLED_BYTE_ARRAY().is_basic_type())
+
+ self.assertEqual(False, Types.STRING().is_tuple_type())
+ self.assertEqual(1, Types.STRING().get_total_fields())
+ self.assertEqual(1, Types.STRING().get_arity())
+
+ self.assertEqual(False, Types.SQL_DATE().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIME().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIMESTAMP().is_basic_type())
+
+ def test_row_type(self):
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()])
+ .get_field_names(), ['f0', 'f1'])
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']).get_field_names(), ['a', 'b'])
+
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.STRING()], ['a', 'b']), True)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.INT()],
+ ['a', 'b']), False)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']).__str__(), "Row(a: String, b: String)")
+
+ self.assertEqual(Types.ROW([Types.STRING(), Types.STRING()]),
+ RowTypeInfo([Types.STRING(), Types.STRING()]), True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_names(), ['a', 'b'], True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_types(), [Types.STRING(), Types.STRING()], True)
+
+ def test_tuple_type(self):
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.INT()]), True)
+
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]).__str__(),
+ "Java Tuple2<String, Integer>")
Review comment:
Yes, maybe we should not directly call to toString() of the java type info object.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r464157768
##########
File path: flink-python/pyflink/common/tests/test_typeinfo.py
##########
@@ -0,0 +1,118 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+import unittest
+
+from pyflink.common.typeinfo import Types, RowTypeInfo, TupleTypeInfo, _from_java_type
+
+
+class TypeInfoTests(unittest.TestCase):
+
+ def test_types(self):
+ self.assertEqual(True, Types.BOOLEAN().is_basic_type())
+ self.assertEqual(True, Types.BYTE().is_basic_type())
+ self.assertEqual(True, Types.SHORT().is_basic_type())
+ self.assertEqual(True, Types.INT().is_basic_type())
+ self.assertEqual(True, Types.LONG().is_basic_type())
+ self.assertEqual(True, Types.FLOAT().is_basic_type())
+ self.assertEqual(True, Types.DOUBLE().is_basic_type())
+ self.assertEqual(True, Types.CHAR().is_basic_type())
+ self.assertEqual(True, Types.BIG_INT().is_basic_type())
+ self.assertEqual(True, Types.BIG_DEC().is_basic_type())
+ self.assertEqual(False, Types.PICKLED_BYTE_ARRAY().is_basic_type())
+
+ self.assertEqual(False, Types.STRING().is_tuple_type())
+ self.assertEqual(1, Types.STRING().get_total_fields())
+ self.assertEqual(1, Types.STRING().get_arity())
+
+ self.assertEqual(False, Types.SQL_DATE().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIME().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIMESTAMP().is_basic_type())
+
+ def test_row_type(self):
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()])
+ .get_field_names(), ['f0', 'f1'])
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']).get_field_names(), ['a', 'b'])
+
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.STRING()], ['a', 'b']), True)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.INT()],
+ ['a', 'b']), False)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']).__str__(), "Row(a: String, b: String)")
+
+ self.assertEqual(Types.ROW([Types.STRING(), Types.STRING()]),
+ RowTypeInfo([Types.STRING(), Types.STRING()]), True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_names(), ['a', 'b'], True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_types(), [Types.STRING(), Types.STRING()], True)
+
+ def test_tuple_type(self):
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.INT()]), True)
+
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]).__str__(),
+ "Java Tuple2<String, Integer>")
Review comment:
It is strange to return a string with `Java` for Python typeinfo. We need to regenerate the string for TupleTypeinfo.
##########
File path: flink-python/pyflink/common/tests/test_typeinfo.py
##########
@@ -0,0 +1,118 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+import unittest
+
+from pyflink.common.typeinfo import Types, RowTypeInfo, TupleTypeInfo, _from_java_type
+
+
+class TypeInfoTests(unittest.TestCase):
+
+ def test_types(self):
+ self.assertEqual(True, Types.BOOLEAN().is_basic_type())
+ self.assertEqual(True, Types.BYTE().is_basic_type())
+ self.assertEqual(True, Types.SHORT().is_basic_type())
+ self.assertEqual(True, Types.INT().is_basic_type())
+ self.assertEqual(True, Types.LONG().is_basic_type())
+ self.assertEqual(True, Types.FLOAT().is_basic_type())
+ self.assertEqual(True, Types.DOUBLE().is_basic_type())
+ self.assertEqual(True, Types.CHAR().is_basic_type())
+ self.assertEqual(True, Types.BIG_INT().is_basic_type())
+ self.assertEqual(True, Types.BIG_DEC().is_basic_type())
+ self.assertEqual(False, Types.PICKLED_BYTE_ARRAY().is_basic_type())
+
+ self.assertEqual(False, Types.STRING().is_tuple_type())
+ self.assertEqual(1, Types.STRING().get_total_fields())
+ self.assertEqual(1, Types.STRING().get_arity())
+
+ self.assertEqual(False, Types.SQL_DATE().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIME().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIMESTAMP().is_basic_type())
+
+ def test_row_type(self):
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()])
+ .get_field_names(), ['f0', 'f1'])
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']).get_field_names(), ['a', 'b'])
+
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.STRING()], ['a', 'b']), True)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.INT()],
+ ['a', 'b']), False)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']).__str__(), "Row(a: String, b: String)")
+
+ self.assertEqual(Types.ROW([Types.STRING(), Types.STRING()]),
+ RowTypeInfo([Types.STRING(), Types.STRING()]), True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_names(), ['a', 'b'], True)
+
+ self.assertEqual(Types.ROW_NAMED(['a', 'b'], [Types.STRING(), Types.STRING()])
+ .get_field_types(), [Types.STRING(), Types.STRING()], True)
+
+ def test_tuple_type(self):
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.INT()]), True)
+
+ self.assertEqual(TupleTypeInfo([Types.STRING(), Types.INT()]).__str__(),
+ "Java Tuple2<String, Integer>")
+
+ self.assertNotEqual(TupleTypeInfo([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.BOOLEAN()]))
+
+ self.assertEqual(Types.TUPLE([Types.STRING(), Types.INT()]),
+ TupleTypeInfo([Types.STRING(), Types.INT()]))
+
+ self.assertEqual(Types.TUPLE([Types.STRING(), Types.INT()]).get_field_types(),
+ [Types.STRING(), Types.INT()])
+
+ def test_primitive_array_types(self):
+ primitive_int_array_type_info = Types.PRIMITIVE_ARRAY(Types.INT())
+ self.assertEqual(primitive_int_array_type_info.is_basic_type(), False)
+
+ def test_from_java_type(self):
+ basic_int_type_info = Types.INT()
+ self.assertEqual(basic_int_type_info,
+ _from_java_type(basic_int_type_info.get_java_type_info()))
+
+ row_type_info = Types.ROW([Types.INT(), Types.STRING()])
+ self.assertEqual(row_type_info, _from_java_type(row_type_info.get_java_type_info()))
+
+ tuple_type_info = Types.TUPLE([Types.CHAR(), Types.INT()])
+ self.assertEqual(tuple_type_info, _from_java_type(tuple_type_info.get_java_type_info()))
+
+ primitive_int_array_type_info = Types.PRIMITIVE_ARRAY(Types.INT())
+ self.assertEqual(primitive_int_array_type_info,
+ _from_java_type(primitive_int_array_type_info.get_java_type_info()))
+
+ pickled_byte_array_type_info = Types.PICKLED_BYTE_ARRAY()
+ self.assertEqual(pickled_byte_array_type_info,
+ _from_java_type(pickled_byte_array_type_info.get_java_type_info()))
+
+ sql_date_type_info = Types.SQL_DATE()
+ self.assertEqual(sql_date_type_info,
+ _from_java_type(sql_date_type_info.get_java_type_info()))
+
+
+
Review comment:
Remove the useless empty lines.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052)
* fd04ce4ce52a282193a91cc109598ff7df1a0117 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062)
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* dc01ebcdb7d9e911b0c3f145867ad1bc90288b30 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105)
* be7e94a339fb20d1cee374495d8d8c1ac0af3c7b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* be7e94a339fb20d1cee374495d8d8c1ac0af3c7b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r464914190
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,523 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
Review comment:
Agree, I will remove them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463458488
##########
File path: flink-python/src/main/java/org/apache/flink/datastream/typeinfo/python/PickledByteArrayTypeInfo.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.typeinfo.python;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+
+/**
+ * A PickledByteArrayTypeInfo indicates that the data of this type is a generated primitive byte
+ * array by pickle.
+ */
+public class PickledByteArrayTypeInfo extends TypeInformation<byte[]> {
Review comment:
Ok, making it singleton is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079)
* 2d8d302ef214afc070f00d730d6c25c9509e3529 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 merged pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 merged pull request #13029:
URL: https://github.com/apache/flink/pull/13029
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
}, {
"hash" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162",
"triggerID" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* be7e94a339fb20d1cee374495d8d8c1ac0af3c7b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106)
* 943ff7e98177dec676272e44eae6e8fb1a1fcd2a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463453594
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self):
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self):
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composit types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self):
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self):
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self):
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self):
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self):
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self):
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def INSTANT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INSTANT_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class LocalTimeTypeInfo(TypeInformation, ABC):
+ """
+ Type information for Java LocalDate/LocalTime/LocalDateTime.
+ """
+
+ @staticmethod
+ def LOCAL_DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_DATE)
+
+ @staticmethod
+ def LOCAL_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo.LOCAL_TIME)
+
+ @staticmethod
+ def LOCAL_DATE_TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.LocalTimeTypeInfo
+ .LOCAL_DATE_TIME)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo())
+
+
+class BigDecimalTypeInfo(WrapperTypeInfo):
Review comment:
Yes, I'll remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066)
* 39f5be60dc81c30f0cbbfb4e219722cbdd52ea21 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070)
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066)
* 39f5be60dc81c30f0cbbfb4e219722cbdd52ea21 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r464176789
##########
File path: flink-python/pyflink/common/tests/test_typeinfo.py
##########
@@ -0,0 +1,115 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+import unittest
+
+from pyflink.common.typeinfo import Types, RowTypeInfo, TupleTypeInfo, _from_java_type
+
+
+class TypeInfoTests(unittest.TestCase):
+
+ def test_types(self):
+ self.assertEqual(True, Types.BOOLEAN().is_basic_type())
+ self.assertEqual(True, Types.BYTE().is_basic_type())
+ self.assertEqual(True, Types.SHORT().is_basic_type())
+ self.assertEqual(True, Types.INT().is_basic_type())
+ self.assertEqual(True, Types.LONG().is_basic_type())
+ self.assertEqual(True, Types.FLOAT().is_basic_type())
+ self.assertEqual(True, Types.DOUBLE().is_basic_type())
+ self.assertEqual(True, Types.CHAR().is_basic_type())
+ self.assertEqual(True, Types.BIG_INT().is_basic_type())
+ self.assertEqual(True, Types.BIG_DEC().is_basic_type())
+ self.assertEqual(False, Types.PICKLED_BYTE_ARRAY().is_basic_type())
+
+ self.assertEqual(False, Types.STRING().is_tuple_type())
+ self.assertEqual(1, Types.STRING().get_total_fields())
+ self.assertEqual(1, Types.STRING().get_arity())
+
+ self.assertEqual(False, Types.SQL_DATE().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIME().is_basic_type())
+ self.assertEqual(False, Types.SQL_TIMESTAMP().is_basic_type())
+
+ def test_row_type(self):
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()])
+ .get_field_names(), ['f0', 'f1'])
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']).get_field_names(), ['a', 'b'])
+
+ self.assertEqual(RowTypeInfo([Types.STRING(), Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.STRING()], ['a', 'b']), True)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']) == RowTypeInfo([Types.STRING(),
+ Types.INT()],
+ ['a', 'b']), False)
+ self.assertEqual(RowTypeInfo([Types.STRING(),
+ Types.STRING()],
+ ['a', 'b']).__str__(), "RowTypeInfo[String, String]")
Review comment:
We should also print field name for row type info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd04ce4ce52a282193a91cc109598ff7df1a0117 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062)
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* dc01ebcdb7d9e911b0c3f145867ad1bc90288b30 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105)
* be7e94a339fb20d1cee374495d8d8c1ac0af3c7b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
}, {
"hash" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* be7e94a339fb20d1cee374495d8d8c1ac0af3c7b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106)
* 943ff7e98177dec676272e44eae6e8fb1a1fcd2a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052)
* fd04ce4ce52a282193a91cc109598ff7df1a0117 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463473374
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types: List[TypeInformation]):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
+ """
+ Returns type information for Row with fields of the given types and with given names. A row
+ must not be null.
+
+ :param names: array of field names.
+ :param types: array of field types.
+ """
+ return RowTypeInfo(types, names)
+
+ @staticmethod
+ def TUPLE(types):
Review comment:
Add type hint
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
Review comment:
Add type hint.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types: List[TypeInformation]):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
Review comment:
Add type hint
##########
File path: flink-python/pyflink/common/tests/test_typeinfo.py
##########
@@ -0,0 +1,63 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+import unittest
+
+from pyflink.common.typeinfo import Types, RowTypeInfo
+
+
+class TypeInfoTests(unittest.TestCase):
Review comment:
Add tests to cover `TupleTypeInfo` and methods in Types(`Types.ROW(), Types.ROW_NAMED, etc`).
##########
File path: flink-python/src/main/java/org/apache/flink/datastream/typeinfo/python/PickledByteArrayTypeInfo.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.typeinfo.python;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+
+/**
+ * A PickledByteArrayTypeInfo indicates that the data of this type is a generated primitive byte
+ * array by pickle.
+ */
+public class PickledByteArrayTypeInfo extends TypeInformation<byte[]> {
+
+ public static final PickledByteArrayTypeInfo PICKLED_BYTE_ARRAY_TYPE_INFO = new PickledByteArrayTypeInfo();
+
+ @Override
+ public boolean isBasicType() {
+ return false;
+ }
+
+ @Override
+ public boolean isTupleType() {
+ return false;
+ }
+
+ @Override
+ public int getArity() {
+ return 1;
+ }
+
+ @Override
+ public int getTotalFields() {
+ return 1;
+ }
+
+ @Override
+ public Class<byte[]> getTypeClass() {
+ return byte[].class;
+ }
+
+ @Override
+ public boolean isKeyType() {
+ return false;
+ }
+
+ @Override
+ public TypeSerializer<byte[]> createSerializer(ExecutionConfig config) {
+
Review comment:
Remove the blank line.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types: List[TypeInformation]):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
+ """
+ Returns type information for Row with fields of the given types and with given names. A row
+ must not be null.
+
+ :param names: array of field names.
+ :param types: array of field types.
+ """
+ return RowTypeInfo(types, names)
+
+ @staticmethod
+ def TUPLE(types):
+ """
+ Returns type information for Tuple with fields of the given types. A Tuple itself must not
+ be null.
+
+ :param types: array of field types.
+ """
+ return TupleTypeInfo(types)
+
+ @staticmethod
+ def PRIMITIVE_ARRAY(element_type):
Review comment:
Add type hint
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types: List[TypeInformation]):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
Review comment:
Add type hint
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types: List[TypeInformation]):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
+ """
+ Returns type information for Row with fields of the given types and with given names. A row
+ must not be null.
+
+ :param names: array of field names.
+ :param types: array of field types.
+ """
+ return RowTypeInfo(types, names)
+
+ @staticmethod
+ def TUPLE(types):
+ """
+ Returns type information for Tuple with fields of the given types. A Tuple itself must not
+ be null.
+
+ :param types: array of field types.
+ """
+ return TupleTypeInfo(types)
+
+ @staticmethod
+ def PRIMITIVE_ARRAY(element_type):
+ """
+ Returns type information for arrays of primitive type (such as byte[]). The array must not
+ be null.
+
+ :param element_type element type of the array (e.g. Types.BOOLEAN(), Types.INT(),
+ Types.DOUBLE())
+ """
+ if element_type == Types.BOOLEAN():
+ return PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.BYTE():
+ return PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.SHORT():
+ return PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.INT():
+ return PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.LONG():
+ return PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.FLOAT():
+ return PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.DOUBLE():
+ return PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.CHAR():
+ return PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO()
+ else:
+ raise TypeError("Invalid element type for a primitive array.")
+
+
+def from_java_type(j_type_info: JavaObject) -> TypeInformation:
Review comment:
Private method. Rename to `_from_java_type`.
##########
File path: flink-python/src/main/java/org/apache/flink/datastream/typeinfo/python/PickledByteArrayTypeInfo.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.typeinfo.python;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+
+/**
+ * A PickledByteArrayTypeInfo indicates that the data of this type is a generated primitive byte
+ * array by pickle.
+ */
+public class PickledByteArrayTypeInfo extends TypeInformation<byte[]> {
Review comment:
Make the constructor protected.
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,508 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+from typing import List, Union
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually exist.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples represents as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self) -> bool:
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self) -> bool:
+ """
+ Checks if this type information represents a Tuple type.
+
+ :return: True, if this type information describes a tuple type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def get_arity(self) -> int:
+ """
+ Gets the arity of this type - the number of fields without nesting.
+
+ :return: the number of fields in this type without nesting.
+ """
+ pass
+
+ @abstractmethod
+ def get_total_fields(self) -> int:
+ """
+ Gets the number of logical fields in this type. This includes its nested and transitively
+ nested fields, in the case of composite types.
+ The total number of fields must be at lest 1.
+
+ :return: The number of fields in this type, including its sub-fields (for composite types).
+ """
+ pass
+
+
+class WrapperTypeInfo(TypeInformation):
+ """
+ A wrapper class for java TypeInformation Objects.
+ """
+
+ def __init__(self, j_typeinfo):
+ self._j_typeinfo = j_typeinfo
+
+ def is_basic_type(self) -> bool:
+ return self._j_typeinfo.isBasicType()
+
+ def is_tuple_type(self) -> bool:
+ return self._j_typeinfo.isTupleType()
+
+ def get_arity(self) -> int:
+ return self._j_typeinfo.getArity()
+
+ def get_total_fields(self) -> int:
+ return self._j_typeinfo.getTotalFields()
+
+ def get_java_type_info(self) -> JavaObject:
+ return self._j_typeinfo
+
+ def __eq__(self, o) -> bool:
+ if type(o) is type(self):
+ return self._j_typeinfo.equals(o._j_typeinfo)
+ else:
+ return False
+
+ def __hash__(self) -> int:
+ return hash(self._j_typeinfo)
+
+
+class BasicTypeInfo(TypeInformation, ABC):
+ """
+ Type information for primitive types (int, long, double, byte, ...), String, BigInteger,
+ and BigDecimal.
+ """
+
+ @staticmethod
+ def STRING_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm
+ .org.apache.flink.api.common.typeinfo.BasicTypeInfo.STRING_TYPE_INFO)
+
+ @staticmethod
+ def BOOLEAN_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BYTE_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.SHORT_TYPE_INFO)
+
+ @staticmethod
+ def INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO)
+
+ @staticmethod
+ def LONG_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.LONG_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.FLOAT_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.DOUBLE_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.CHAR_TYPE_INFO)
+
+ @staticmethod
+ def BIG_INT_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_INT_TYPE_INFO)
+
+ @staticmethod
+ def BIG_DEC_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo.BIG_DEC_TYPE_INFO)
+
+
+class SqlTimeTypeInfo(TypeInformation, ABC):
+ """
+ SqlTimeTypeInfo enables users to get Sql Time TypeInfo.
+ """
+
+ @staticmethod
+ def DATE():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.DATE)
+
+ @staticmethod
+ def TIME():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIME)
+
+ @staticmethod
+ def TIMESTAMP():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo.TIMESTAMP)
+
+
+class PrimitiveArrayTypeInfo(TypeInformation, ABC):
+ """
+ A TypeInformation for arrays of primitive types (int, long, double, ...).
+ Supports the creation of dedicated efficient serializers for these types.
+ """
+
+ @staticmethod
+ def BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def BYTE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def SHORT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def INT_PRIMITIVE_ARRAY_TYPE_INFO():
+ WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def LONG_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def FLOAT_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO)
+
+ @staticmethod
+ def CHAR_PRIMITIVE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo
+ .PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO)
+
+
+class PickledBytesTypeInfo(TypeInformation, ABC):
+ """
+ A PickledBytesTypeInfo indicates the data is a primitive byte array generated by pickle
+ serializer.
+ """
+
+ @staticmethod
+ def PICKLED_BYTE_ARRAY_TYPE_INFO():
+ return WrapperTypeInfo(get_gateway().jvm.org.apache.flink.datastream.typeinfo.python
+ .PickledByteArrayTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO)
+
+
+class RowTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Row.
+ """
+
+ def __init__(self, types: List[TypeInformation], field_names: List[str] = None):
+ self.types = types
+ self.field_names = field_names
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ if field_names is None:
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array)
+ else:
+ j_names_array = get_gateway().new_array(get_gateway().jvm.java.lang.String,
+ len(field_names))
+ for i in range(len(field_names)):
+ j_names_array[i] = field_names[i]
+ self._j_typeinfo = get_gateway().jvm.org.apache.flink.api.java.typeutils.RowTypeInfo(
+ self.j_types_array, j_names_array)
+ super(RowTypeInfo, self).__init__(self._j_typeinfo)
+
+ def get_field_names(self) -> List[str]:
+ j_field_names = self._j_typeinfo.getFieldNames()
+ field_names = [name for name in j_field_names]
+ return field_names
+
+ def get_field_index(self, field_name) -> int:
+ return self._j_typeinfo.getFieldIndex(field_name)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class TupleTypeInfo(WrapperTypeInfo):
+ """
+ TypeInformation for Tuple.
+ """
+
+ def __init__(self, types: List[TypeInformation]):
+ self.types = types
+ self.j_types_array = get_gateway().new_array(
+ get_gateway().jvm.org.apache.flink.api.common.typeinfo.TypeInformation, len(types))
+
+ for i in range(len(types)):
+ self.j_types_array[i] = types[i].get_java_type_info()
+
+ self._j_typeinfo = get_gateway().jvm \
+ .org.apache.flink.api.java.typeutils.TupleTypeInfo(self.j_types_array)
+
+ def get_field_types(self) -> List[TypeInformation]:
+ return self.types
+
+ def __eq__(self, other) -> bool:
+ return self._j_typeinfo.equals(other._j_typeinfo)
+
+ def __hash__(self) -> int:
+ return self._j_typeinfo.hashCode()
+
+ def __str__(self) -> str:
+ return self._j_typeinfo.toString()
+
+
+class Types(object):
+ """
+ This class gives access to the type information of the most common types for which Flink has
+ built-in serializers and comparators.
+ """
+
+ STRING = BasicTypeInfo.STRING_TYPE_INFO
+ BYTE = BasicTypeInfo.BYTE_TYPE_INFO
+ BOOLEAN = BasicTypeInfo.BOOLEAN_TYPE_INFO
+ SHORT = BasicTypeInfo.SHORT_TYPE_INFO
+ INT = BasicTypeInfo.INT_TYPE_INFO
+ LONG = BasicTypeInfo.LONG_TYPE_INFO
+ FLOAT = BasicTypeInfo.FLOAT_TYPE_INFO
+ DOUBLE = BasicTypeInfo.DOUBLE_TYPE_INFO
+ CHAR = BasicTypeInfo.CHAR_TYPE_INFO
+ BIG_INT = BasicTypeInfo.BIG_INT_TYPE_INFO
+ BIG_DEC = BasicTypeInfo.BIG_DEC_TYPE_INFO
+
+ SQL_DATE = SqlTimeTypeInfo.DATE
+ SQL_TIME = SqlTimeTypeInfo.TIME
+ SQL_TIMESTAMP = SqlTimeTypeInfo.TIMESTAMP
+
+ PICKLED_BYTE_ARRAY = PickledBytesTypeInfo.PICKLED_BYTE_ARRAY_TYPE_INFO
+
+ @staticmethod
+ def ROW(types):
+ """
+ Returns type information for Row with fields of the given types. A row itself must not be
+ null.
+
+ :param types: the types of the row fields, e.g., Types.String(), Types.INT()
+ """
+ return RowTypeInfo(types)
+
+ @staticmethod
+ def ROW_NAMED(names, types):
+ """
+ Returns type information for Row with fields of the given types and with given names. A row
+ must not be null.
+
+ :param names: array of field names.
+ :param types: array of field types.
+ """
+ return RowTypeInfo(types, names)
+
+ @staticmethod
+ def TUPLE(types):
+ """
+ Returns type information for Tuple with fields of the given types. A Tuple itself must not
+ be null.
+
+ :param types: array of field types.
+ """
+ return TupleTypeInfo(types)
+
+ @staticmethod
+ def PRIMITIVE_ARRAY(element_type):
+ """
+ Returns type information for arrays of primitive type (such as byte[]). The array must not
+ be null.
+
+ :param element_type element type of the array (e.g. Types.BOOLEAN(), Types.INT(),
+ Types.DOUBLE())
+ """
+ if element_type == Types.BOOLEAN():
+ return PrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.BYTE():
+ return PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.SHORT():
+ return PrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.INT():
+ return PrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.LONG():
+ return PrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.FLOAT():
+ return PrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.DOUBLE():
+ return PrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO()
+ elif element_type == Types.CHAR():
+ return PrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO()
+ else:
+ raise TypeError("Invalid element type for a primitive array.")
+
+
+def from_java_type(j_type_info: JavaObject) -> TypeInformation:
+ gateway = get_gateway()
+ JBasicTypeInfo = gateway.jvm.org.apache.flink.api.common.typeinfo.BasicTypeInfo
+
+ if is_instance_of(j_type_info, JBasicTypeInfo.STRING_TYPE_INFO):
+ return Types.STRING()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.BOOLEAN_TYPE_INFO):
+ return Types.BOOLEAN()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.BYTE_TYPE_INFO):
+ return Types.BYTE()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.SHORT_TYPE_INFO):
+ return Types.SHORT()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.INT_TYPE_INFO):
+ return Types.INT()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.LONG_TYPE_INFO):
+ return Types.LONG()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.FLOAT_TYPE_INFO):
+ return Types.FLOAT()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.DOUBLE_TYPE_INFO):
+ return Types.DOUBLE()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.CHAR_TYPE_INFO):
+ return Types.CHAR()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.BIG_INT_TYPE_INFO):
+ return Types.BIG_INT()
+ elif is_instance_of(j_type_info, JBasicTypeInfo.BIG_DEC_TYPE_INFO):
+ return Types.BIG_DEC()
+
+ JSqlTimeTypeInfo = gateway.jvm.org.apache.flink.api.common.typeinfo.SqlTimeTypeInfo
+ if is_instance_of(j_type_info, JSqlTimeTypeInfo.DATE):
+ return Types.SQL_DATE()
+ elif is_instance_of(j_type_info, JSqlTimeTypeInfo.TIME):
+ return Types.SQL_TIME()
+ elif is_instance_of(j_type_info, JSqlTimeTypeInfo.TIMESTAMP):
+ return Types.SQL_TIMESTAMP()
+
+ JPrimitiveArrayTypeInfo = gateway.jvm.org.apache.flink.api.common.typeinfo \
+ .PrimitiveArrayTypeInfo
+
+ if is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.BOOLEAN_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.BOOLEAN())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.BYTE())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.SHORT_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.SHORT())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.INT_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.INT())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.LONG_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.LONG())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.FLOAT_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.FLOAT())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.DOUBLE_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.DOUBLE())
+ elif is_instance_of(j_type_info, JPrimitiveArrayTypeInfo.CHAR_PRIMITIVE_ARRAY_TYPE_INFO):
+ return Types.PRIMITIVE_ARRAY(Types.CHAR())
+
+ JPickledBytesTypeInfo = gateway.jvm \
+ .org.apache.flink.datastream.typeinfo.python.PickledByteArrayTypeInfo\
+ .PICKLED_BYTE_ARRAY_TYPE_INFO
+ if is_instance_of(j_type_info, JPickledBytesTypeInfo):
+ return Types.PICKLED_BYTE_ARRAY()
+
+ JRowTypeInfo = gateway.jvm.org.apache.flink.api.java.typeutils.RowTypeInfo
+ if is_instance_of(j_type_info, JRowTypeInfo):
+ j_row_field_names = j_type_info.getFieldNames()
+ j_row_field_types = j_type_info.getFieldTypes()
+ row_field_types = [from_java_type(j_row_field_type) for j_row_field_type in
+ j_row_field_types]
+ return Types.ROW_NAMED(j_row_field_names, row_field_types)
+
+ raise TypeError("The java type info: %s is not supported in PyFlink currently." % j_type_info)
+
+
+def is_instance_of(java_object: JavaObject, java_type: Union[JavaObject, JavaClass]) -> bool:
Review comment:
Private method. Rename to `_is_instance_of`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463450905
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
Review comment:
Thank you! I will revise it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd04ce4ce52a282193a91cc109598ff7df1a0117 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062)
* 3d89fc037c6cf7b1dfe19514ff831af4a4d33b50 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
}, {
"hash" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162",
"triggerID" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"triggerType" : "PUSH"
}, {
"hash" : "2c7ff6ceffd4f6be815e98befb669a07d98dc768",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5169",
"triggerID" : "2c7ff6ceffd4f6be815e98befb669a07d98dc768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 2c7ff6ceffd4f6be815e98befb669a07d98dc768 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5169)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
}, {
"hash" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162",
"triggerID" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"triggerType" : "PUSH"
}, {
"hash" : "2c7ff6ceffd4f6be815e98befb669a07d98dc768",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c7ff6ceffd4f6be815e98befb669a07d98dc768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 943ff7e98177dec676272e44eae6e8fb1a1fcd2a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162)
* 2c7ff6ceffd4f6be815e98befb669a07d98dc768 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 2d8d302ef214afc070f00d730d6c25c9509e3529 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098)
* dc01ebcdb7d9e911b0c3f145867ad1bc90288b30 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hequn8128 edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
hequn8128 edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-668553682
@shuiqiangchen Thanks a lot for the update. Please check the azure test failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #13029:
URL: https://github.com/apache/flink/pull/13029#discussion_r463453223
##########
File path: flink-python/pyflink/common/typeinfo.py
##########
@@ -0,0 +1,576 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+from abc import ABC, abstractmethod
+
+from py4j.java_gateway import JavaClass, JavaObject
+
+from pyflink.java_gateway import get_gateway
+
+
+class TypeInformation(ABC):
+ """
+ TypeInformation is the core class of Flink's type system. FLink requires a type information
+ for all types that are used as input or return type of a user function. This type information
+ class acts as the tool to generate serializers and comparators, and to perform semantic checks
+ such as whether the fields that are used as join/grouping keys actually existt.
+
+ The type information also bridges between the programming languages object model and a logical
+ flat schema. It maps fields from the types to columns (fields) in a flat schema. Not all fields
+ from a type are mapped to a separate fields in the flat schema and often, entire types are
+ mapped to one field.. It is important to notice that the schema must hold for all instances of a
+ type. For that reason, elements in lists and arrays are not assigned to individual fields, but
+ the lists and arrays are considered to be one field in total, to account for different lengths
+ in the arrays.
+ a) Basic types are indivisible and are considered as a single field.
+ b) Arrays and collections are one field.
+ c) Tuples and case classes represent as many fields as the class has fields.
+ To represent this properly, each type has an arity (the number of fields it contains directly),
+ and a total number of fields (number of fields in the entire schema of this type, including
+ nested types).
+ """
+
+ @abstractmethod
+ def is_basic_type(self):
+ """
+ Checks if this type information represents a basic type.
+ Basic types are defined in BasicTypeInfo and are primitives, their boxing type, Strings ...
+
+ :return: True, if this type information describes a basic type, false otherwise.
+ """
+ pass
+
+ @abstractmethod
+ def is_tuple_type(self):
+ """
+ Checks if this type information represents a Tuple type.
+ Tuple types are subclasses of the Java API tuples.
Review comment:
Sure, there should be differences between java and python comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5105",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
}, {
"hash" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5106",
"triggerID" : "be7e94a339fb20d1cee374495d8d8c1ac0af3c7b",
"triggerType" : "PUSH"
}, {
"hash" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162",
"triggerID" : "943ff7e98177dec676272e44eae6e8fb1a1fcd2a",
"triggerType" : "PUSH"
}, {
"hash" : "2c7ff6ceffd4f6be815e98befb669a07d98dc768",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5169",
"triggerID" : "2c7ff6ceffd4f6be815e98befb669a07d98dc768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 943ff7e98177dec676272e44eae6e8fb1a1fcd2a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5162)
* 2c7ff6ceffd4f6be815e98befb669a07d98dc768 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5169)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5052",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
}, {
"hash" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5062",
"triggerID" : "fd04ce4ce52a282193a91cc109598ff7df1a0117",
"triggerType" : "PUSH"
}, {
"hash" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5066",
"triggerID" : "3d89fc037c6cf7b1dfe19514ff831af4a4d33b50",
"triggerType" : "PUSH"
}, {
"hash" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5070",
"triggerID" : "39f5be60dc81c30f0cbbfb4e219722cbdd52ea21",
"triggerType" : "PUSH"
}, {
"hash" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a3c5b93a697dea00a2e8237deb03b5848ffb1a9",
"triggerType" : "PUSH"
}, {
"hash" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5079",
"triggerID" : "9e004f9f8ebc001e7a8d63e39eb89f6b2b5cc431",
"triggerType" : "PUSH"
}, {
"hash" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098",
"triggerID" : "2d8d302ef214afc070f00d730d6c25c9509e3529",
"triggerType" : "PUSH"
}, {
"hash" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dc01ebcdb7d9e911b0c3f145867ad1bc90288b30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a3c5b93a697dea00a2e8237deb03b5848ffb1a9 UNKNOWN
* 2d8d302ef214afc070f00d730d6c25c9509e3529 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5098)
* dc01ebcdb7d9e911b0c3f145867ad1bc90288b30 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13029: [FLINK-18763][python] Support basic TypeInformation for Python DataSt…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13029:
URL: https://github.com/apache/flink/pull/13029#issuecomment-666332680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2cb0d813bfc779b913096c3cd6e646ac2d4f64e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2cb0d813bfc779b913096c3cd6e646ac2d4f64e1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org