You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/16 07:02:50 UTC
[GitHub] [flink] HuangXingBo opened a new pull request #11767:
[FLINK-17119][python] Support Composite DataTypes in Cython
HuangXingBo opened a new pull request #11767: [FLINK-17119][python] Support Composite DataTypes in Cython
URL: https://github.com/apache/flink/pull/11767
## What is the purpose of the change
*This pull request will support complex DataTypes in Cython*
## Brief change log
- *Add the cython implementation of complex DataTypes*
## Verifying this change
This change added tests and can be verified as follows:
- *Added ut test in test_fast_coders.py*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160502233",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160535260",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7591",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
}, {
"hash" : "64600d36a0518019bb02bf80fd4164ca845aa99f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64600d36a0518019bb02bf80fd4164ca845aa99f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f59cc087b30926d77f3fc208246bcc288bf4f38c Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160502233) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566)
* 0f591d4b6b4b34fb4875e813aafc789b35b0c1d2 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160535260) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7591)
* 64600d36a0518019bb02bf80fd4164ca845aa99f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160502233",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160535260",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7591",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
}, {
"hash" : "64600d36a0518019bb02bf80fd4164ca845aa99f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64600d36a0518019bb02bf80fd4164ca845aa99f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f591d4b6b4b34fb4875e813aafc789b35b0c1d2 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160535260) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7591)
* 64600d36a0518019bb02bf80fd4164ca845aa99f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160502233",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160535260",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7591",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
}, {
"hash" : "64600d36a0518019bb02bf80fd4164ca845aa99f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64600d36a0518019bb02bf80fd4164ca845aa99f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f591d4b6b4b34fb4875e813aafc789b35b0c1d2 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160535260) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7591)
* 64600d36a0518019bb02bf80fd4164ca845aa99f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160502233",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f591d4b6b4b34fb4875e813aafc789b35b0c1d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f59cc087b30926d77f3fc208246bcc288bf4f38c Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160502233) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566)
* 0f591d4b6b4b34fb4875e813aafc789b35b0c1d2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangXingBo commented on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
HuangXingBo commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-614992754
Not sure why the tests have not been triggered by CiBot. The tests have passed in my personal travis: https://travis-ci.org/github/HuangXingBo/flink/builds/675692170
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on issue #11767: [FLINK-17119][python] Add
Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615077677
@pnowojski Sorry about this. The CiBot has not been triggered for about 10 hours after pushing a new commit. So just merge the PR when the tests passed in the personal travis. I'll pay more attention to this and will keep an eye on the tests both in travis and azure for the next time. Sorry about this again!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11767: [FLINK-17119][python]
Support Composite DataTypes in Cython
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11767: [FLINK-17119][python] Support Composite DataTypes in Cython
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f59cc087b30926d77f3fc208246bcc288bf4f38c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] sunjincheng121 commented on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615076310
Thanks @pnowojski
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on issue #11767: [FLINK-17119][python] Add
Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615093284
@pnowojski Definitely agree with you. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11767:
[FLINK-17119][python] Support Composite DataTypes in Cython
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11767: [FLINK-17119][python] Support Composite DataTypes in Cython
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160502233",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f59cc087b30926d77f3fc208246bcc288bf4f38c Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160502233) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on issue #11767: [FLINK-17119][python]
Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
pnowojski commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615086973
Yes, we have some build queue issues, but that doesn't mean we should be ignoring CI. For couple of PRs that I was merging recently I had to wait before merging them. Also as far as I know there is a way to have a private azure setup similar to travis, maybe that would help in such situations. I'm fine with merging PRs if private azure/travis builds have passed. But not fine if just travis passed - travis is running way less tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#discussion_r409466671
##########
File path: flink-python/pyflink/fn_execution/fast_coder_impl.pyx
##########
@@ -340,6 +444,86 @@ cdef class FlattenRowCoderImpl(StreamCoderImpl):
milliseconds = hour * 3600000 + minute * 60000 + seconds * 1000 + microsecond // 1000
self._encode_int(milliseconds)
+ cdef void _encode_field_complex(self, TypeName field_type, BaseCoder field_coder, item):
+ cdef libc.stdint.int32_t nanoseconds, microseconds_of_second, length, row_field_count
+ cdef libc.stdint.int32_t leading_complete_bytes_num, remaining_bits_num
+ cdef libc.stdint.int64_t timestamp_milliseconds, timestamp_seconds
+ cdef BaseCoder value_coder, key_coder
+ cdef TypeName value_type, key_type
+ cdef CoderType value_coder_type, key_coder_type
+ cdef BaseCoder row_field_coder
+ cdef list row_field_coders, row_value
+
+ if field_type == DECIMAL:
+ # decimal
+ user_context = decimal.getcontext()
+ decimal.setcontext((<DecimalCoderImpl> field_coder).context)
+ bytes_value = str(item.quantize((<DecimalCoderImpl> field_coder).scale_format)).encode(
+ "utf-8")
+ self._encode_bytes(bytes_value)
+ decimal.setcontext(user_context)
+ elif field_type == TIMESTAMP or field_type == LOCAL_ZONED_TIMESTAMP:
+ # Timestamp
+ timestamp_seconds = <libc.stdint.int64_t> (
+ item.replace(tzinfo=datetime.timezone.utc).timestamp())
+ microseconds_of_second = item.microsecond
+ timestamp_milliseconds = timestamp_seconds * 1000 + microseconds_of_second // 1000
+ nanoseconds = microseconds_of_second % 1000 * 1000
+ if field_coder.is_compact:
+ self._encode_bigint(timestamp_milliseconds)
+ else:
+ self._encode_bigint(timestamp_milliseconds)
+ self._encode_int(nanoseconds)
+ elif field_type == ARRAY:
+ # Array
+ length = len(item)
+ value_coder = (<ArrayCoderImpl> field_coder).elem_coder
+ value_type = value_coder.type_name()
+ value_coder_type = value_coder.coder_type()
+ self._encode_int(length)
+ for i in range(length):
+ value = item[i]
+ if value is None:
+ self._encode_byte(False)
+ else:
+ self._encode_byte(True)
+ self._encode_field(value_coder_type, value_type, value_coder, value)
+
Review comment:
remove the empty line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11767:
[FLINK-17119][python] Support Composite DataTypes in Cython
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11767: [FLINK-17119][python] Support Composite DataTypes in Cython
URL: https://github.com/apache/flink/pull/11767#issuecomment-614460127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
}, {
"hash" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160502233",
"triggerID" : "f59cc087b30926d77f3fc208246bcc288bf4f38c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f59cc087b30926d77f3fc208246bcc288bf4f38c Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160502233) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7566)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11767: [FLINK-17119][python]
Support Composite DataTypes in Cython
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11767: [FLINK-17119][python] Support Composite DataTypes in Cython
URL: https://github.com/apache/flink/pull/11767#issuecomment-614455684
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit f59cc087b30926d77f3fc208246bcc288bf4f38c (Thu Apr 16 07:05:33 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-17119).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#discussion_r409471208
##########
File path: flink-python/pyflink/fn_execution/fast_coder_impl.pyx
##########
@@ -143,6 +145,9 @@ cdef class FlattenRowCoderImpl(StreamCoderImpl):
if item is not None:
if self._output_coder_type[i] == SIMPLE:
Review comment:
use _encode_field?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on issue #11767: [FLINK-17119][python] Add
Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-614996254
Thanks. Merging...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangXingBo commented on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
HuangXingBo commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615138432
Currently, there is no gcc in azure which makes it impossible to run the cython tests in azure pipelines. I have created a JIRA https://issues.apache.org/jira/browse/FLINK-17212 to disable cython tests until flink-container install gcc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski edited a comment on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
pnowojski edited a comment on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615070234
Guys, can you please pay attention to test failures? This failed in this PR:
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7591&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=14487301-07d2-5d56-5690-6dfab9ffd4d9
https://issues.apache.org/jira/browse/FLINK-17205
I'm reverting this change CC @sunjincheng121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski edited a comment on issue #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
pnowojski edited a comment on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615086973
Yes, we have some build queue issues, but that doesn't mean we should be ignoring CI. For couple of PRs that I was merging recently I had to wait before merging them. Also as far as I know there is a way to have a private azure setup similar to travis, maybe that would help in such situations. I'm fine with merging PRs if private azure/travis builds have passed. But not fine if just travis passed - travis is running way fewer tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#discussion_r409471531
##########
File path: flink-python/pyflink/fn_execution/fast_coder_impl.pyx
##########
@@ -180,6 +185,15 @@ cdef class FlattenRowCoderImpl(StreamCoderImpl):
else:
if self._input_coder_type[i] == SIMPLE:
self.row[i] = self._decode_field_simple(self._input_field_type[i])
Review comment:
use _decode_field
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11767:
[FLINK-17119][python] Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#discussion_r409470147
##########
File path: flink-python/pyflink/fn_execution/fast_coder_impl.pxd
##########
@@ -190,3 +195,24 @@ cdef class DateCoderImpl(BaseCoder):
cdef class TimeCoderImpl(BaseCoder):
pass
+
+cdef class DecimalCoderImpl(BaseCoder):
+ cdef readonly object context
+ cdef readonly object scale_format
+
+cdef class TimestampCoderImpl(BaseCoder):
+ cdef readonly bint is_compact
+
+cdef class LocalZonedTimestampCoderImpl(BaseCoder):
Review comment:
inherits TimestampCoderImpl?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on issue #11767: [FLINK-17119][python]
Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
pnowojski commented on issue #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767#issuecomment-615070234
Guys, can you please pay attention to travis failures? This failed in this PR:
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7591&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=14487301-07d2-5d56-5690-6dfab9ffd4d9
https://issues.apache.org/jira/browse/FLINK-17205
I'm reverting this change CC @sunjincheng121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu merged pull request #11767: [FLINK-17119][python]
Add Cython support for composite types
Posted by GitBox <gi...@apache.org>.
dianfu merged pull request #11767: [FLINK-17119][python] Add Cython support for composite types
URL: https://github.com/apache/flink/pull/11767
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services