You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/11 13:55:13 UTC
[GitHub] [incubator-seatunnel] FWLamb opened a new pull request, #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
FWLamb opened a new pull request, #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
Clickhouse Support Int128,Int256 Type https://github.com/apache/incubator-seatunnel/issues/3057
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [x] Code changed are covered with tests, or it does not need tests for reason:
* [x] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1278389899
I have a question, why not use decimal to implement int128 and int256?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
FWLamb commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1275484737
@TyrantLucifer https://github.com/apache/incubator-seatunnel/issues/3071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wenwei8268 commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
wenwei8268 commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1275459990
OK, thanks for your advice;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1274801393
Your pull request changed the core module without proposal. Please refer to [Coding Guide](https://seatunnel.apache.org/docs/contribution/coding-guide) change your pr request. I think Int128,Int256 Type can using `Long` data type.
@Hisoka-X PLAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
FWLamb commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1278385276
> We should support it both source side and sink side, so we should test read int128 from clickhouse source and write it into clickhouse table.
**Yes, I have already tested.**
The following is a case of e2d testing.
source_table = """
create table if not exists `default`.source_table(
`id` Int64,
`c_map` Map(String, Int32),
`c_array_string` Array(String),
`c_array_short` Array(Int16),
`c_array_int` Array(Int32),
`c_array_long` Array(Int64),
`c_array_float` Array(Float32),
`c_array_double` Array(Float64),
`c_string` String,
`c_boolean` Boolean,
`c_int8` Int8,
`c_int16` Int16,
`c_int32` Int32,
`c_int64` Int64,
`c_float32` Float32,
`c_float64` Float64,
`c_decimal` Decimal(9,4),
`c_date` Date,
`c_datetime` DateTime64,
`c_nullable` Nullable(Int32),
`c_lowcardinality` LowCardinality(String),
`c_nested` Nested
(
`int` UInt32,
`double` Int64,
`string` String
),
`c_int128` Int128,
`c_int256` Int256,
`c_uint128` UInt128,
`c_uint256` UInt256
)engine=Memory;
"""
sink_table = """
create table if not exists `default`.sink_table(
`id` Int64,
`c_map` Map(String, Int32),
`c_array_string` Array(String),
`c_array_short` Array(Int16),
`c_array_int` Array(Int32),
`c_array_long` Array(Int64),
`c_array_float` Array(Float32),
`c_array_double` Array(Float64),
`c_string` String,
`c_boolean` Boolean,
`c_int8` Int8,
`c_int16` Int16,
`c_int32` Int32,
`c_int64` Int64,
`c_float32` Float32,
`c_float64` Float64,
`c_decimal` Decimal(9,4),
`c_date` Date,
`c_datetime` DateTime64,
`c_nullable` Nullable(Int32),
`c_lowcardinality` LowCardinality(String),
`c_nested` Nested
(
`int` UInt32,
`double` Int64,
`string` String
),
`c_int128` Int128,
`c_int256` Int256,
`c_uint128` UInt128,
`c_uint256` UInt256
)engine=Memory;
"""
env {
execution.parallelism = 1
job.mode = "BATCH"
}
source {
Clickhouse {
host = "clickhouse:8123"
database = "default"
sql = "select * from source_table"
username = "default"
password = ""
result_table_name = "source_table"
}
}
transform {
}
sink {
Clickhouse {
host = "clickhouse:8123"
database = "default"
table = "sink_table"
fields = [
"id",
"c_map",
"c_array_string",
"c_array_short",
"c_array_int",
"c_array_long",
"c_array_float",
"c_array_double",
"c_string",
"c_boolean",
"c_int8",
"c_int16",
"c_int32",
"c_int64",
"c_float32",
"c_float64",
"c_decimal",
"c_date",
"c_datetime",
"c_nullable",
"c_lowcardinality",
"c_nested.int",
"c_nested.double",
"c_nested.string",
"c_int128",
"c_int256",
"c_uint128",
"c_uint256"
]
username = "default"
password = ""
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1278380633
We should support it both source side and sink side, so we should test read int128 from clickhouse source and write it into clickhouse table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
FWLamb commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1278398354
I use String type in spark, Decimal type in other places, it's ok. So I thought, can I only use the String type, I tested it later and found that it is possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
FWLamb commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1278395571
Because decimal only supports up to 38 digits in spark, Int128 and Int256 have exceeded 38 digits, and Spark reported an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
FWLamb commented on PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067#issuecomment-1278364625
I found that it can be achieved with String data type, which has passed the E2E test. @Hisoka-X PLAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 merged pull request #3067: [Improve] [Clickhouse-V2] Clickhouse Support Int128,Int256 Type
Posted by GitBox <gi...@apache.org>.
EricJoy2048 merged PR #3067:
URL: https://github.com/apache/incubator-seatunnel/pull/3067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org