You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by GitBox <gi...@apache.org> on 2022/10/25 06:49:12 UTC
[GitHub] [incubator-linkis] rarexixi opened a new pull request, #3715: spark etl
rarexixi opened a new pull request, #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715
Add a new simple ETL feature，implemented by executing our configured json.
### Related issues/PRs
Related issues: #3435
### Brief change log
- Add a new simple ETL feature，implemented by executing our configured json.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] rarexixi closed pull request #3715: spark etl
Posted by GitBox <gi...@apache.org>.
rarexixi closed pull request #3715: spark etl
URL: https://github.com/apache/incubator-linkis/pull/3715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] rarexixi commented on a diff in pull request #3715: [Feature][linkis-engineplugin-spark] add a new Executor (json ETL)
Posted by GitBox <gi...@apache.org>.
rarexixi commented on code in PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715#discussion_r1005107299
##########
linkis-engineconn-plugins/spark/src/main/java/org/apache/linkis/engineplugin/spark/datacalc/exception/ConfigRuntimeException.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.datacalc.exception;
+
+public class ConfigRuntimeException extends RuntimeException {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on a diff in pull request #3715: [Feature][linkis-engineplugin-spark] add a new Executor (json ETL)
Posted by GitBox <gi...@apache.org>.
peacewong commented on code in PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715#discussion_r1006729719
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/factory/SparkDataCalcExecutorFactory.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.factory
+
+import org.apache.linkis.engineconn.common.creation.EngineCreationContext
+import org.apache.linkis.engineconn.common.engineconn.EngineConn
+import org.apache.linkis.engineconn.computation.executor.creation.ComputationExecutorFactory
+import org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor
+import org.apache.linkis.engineplugin.spark.entity.SparkEngineSession
+import org.apache.linkis.engineplugin.spark.exception.NotSupportSparkDataCalcTypeException
+import org.apache.linkis.engineplugin.spark.executor.SparkDataCalcExecutor
+import org.apache.linkis.manager.label.entity.Label
+import org.apache.linkis.manager.label.entity.engine.RunType
+import org.apache.linkis.manager.label.entity.engine.RunType.RunType
+
+/**
+ * xishihao_modify 添加 data_calc 类型
Review Comment:
need remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] codecov[bot] commented on pull request #3715: spark etl
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715#issuecomment-1290128127
# [Codecov](https://codecov.io/gh/apache/incubator-linkis/pull/3715?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3715](https://codecov.io/gh/apache/incubator-linkis/pull/3715?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (270f48e) into [dev-1.3.2](https://codecov.io/gh/apache/incubator-linkis/commit/885c9d35f690c7a0c4a414c3ae637143e8bcdfb8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (885c9d3) will **decrease** coverage by `0.27%`.
> The diff coverage is `0.39%`.
```diff
@@ Coverage Diff @@
## dev-1.3.2 #3715 +/- ##
===============================================
- Coverage 14.14% 13.86% -0.28%
- Complexity 1512 1513 +1
===============================================
Files 1046 1077 +31
Lines 38647 39406 +759
Branches 5426 5519 +93
===============================================
Hits 5465 5465
- Misses 32422 33180 +758
- Partials 760 761 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-linkis/pull/3715?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...plugin/spark/datacalc/model/DataCalcArrayData.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9EYXRhQ2FsY0FycmF5RGF0YS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...lugin/spark/datacalc/model/DataCalcDataConfig.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9EYXRhQ2FsY0RhdGFDb25maWcuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...lugin/spark/datacalc/model/DataCalcDataSource.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9EYXRhQ2FsY0RhdGFTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...plugin/spark/datacalc/model/DataCalcGroupData.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9EYXRhQ2FsY0dyb3VwRGF0YS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...gin/spark/datacalc/model/DataCalcPluginConfig.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9EYXRhQ2FsY1BsdWdpbkNvbmZpZy5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../engineplugin/spark/datacalc/model/SinkConfig.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9TaW5rQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ngineplugin/spark/datacalc/model/SourceConfig.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9Tb3VyY2VDb25maWcuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...neplugin/spark/datacalc/model/TransformConfig.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9tb2RlbC9UcmFuc2Zvcm1Db25maWcuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...park/datacalc/service/LinkisDataSourceService.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9zZXJ2aWNlL0xpbmtpc0RhdGFTb3VyY2VTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...gineplugin/spark/datacalc/sink/FileSinkConfig.java](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWVuZ2luZWNvbm4tcGx1Z2lucy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGlua2lzL2VuZ2luZXBsdWdpbi9zcGFyay9kYXRhY2FsYy9zaW5rL0ZpbGVTaW5rQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [24 more](https://codecov.io/gh/apache/incubator-linkis/pull/3715/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] rarexixi commented on a diff in pull request #3715: [Feature][linkis-engineplugin-spark] add a new Executor (json ETL)
Posted by GitBox <gi...@apache.org>.
rarexixi commented on code in PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715#discussion_r1005101472
##########
linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/engine/RunType.scala:
##########
@@ -43,4 +43,6 @@ object RunType extends Enumeration {
val TRINO_SQL = Value("tsql")
+ val DATA_CALC = Value("data_calc") // spark datacalc (ETL)
Review Comment:
Data calculation，数据计算
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong merged pull request #3715: [Feature][linkis-engineplugin-spark] add a new Executor (json ETL)
Posted by GitBox <gi...@apache.org>.
peacewong merged PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] rarexixi commented on a diff in pull request #3715: [Feature][linkis-engineplugin-spark] add a new Executor (json ETL)
Posted by GitBox <gi...@apache.org>.
rarexixi commented on code in PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715#discussion_r1005101472
##########
linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/engine/RunType.scala:
##########
@@ -43,4 +43,6 @@ object RunType extends Enumeration {
val TRINO_SQL = Value("tsql")
+ val DATA_CALC = Value("data_calc") // spark datacalc (ETL)
Review Comment:
Data calculation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on a diff in pull request #3715: [Feature][linkis-engineplugin-spark] add a new Executor (json ETL)
Posted by GitBox <gi...@apache.org>.
peacewong commented on code in PR #3715:
URL: https://github.com/apache/incubator-linkis/pull/3715#discussion_r1004435839
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -26,7 +26,7 @@
<artifactId>linkis-engineplugin-spark</artifactId>
<properties>
- <spark.version>2.4.3</spark.version>
+ <spark.version>2.4.7</spark.version>
Review Comment:
There are many version upgrades and modifications, it is recommended to keep 2.4.3 for the time being
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkDataCalcExecutor.scala:
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.executor
+
+import org.apache.linkis.common.exception.FatalException
+import org.apache.linkis.common.log.LogUtils
+import org.apache.linkis.common.utils.{ByteTimeUtils, Logging, Utils}
+import org.apache.linkis.engineconn.computation.executor.execute.{
+ ComputationExecutor,
+ EngineExecutionContext
+}
+import org.apache.linkis.engineconn.computation.executor.utlis.ProgressUtils
+import org.apache.linkis.engineconn.core.executor.ExecutorManager
+import org.apache.linkis.engineplugin.spark.datacalc.{DataCalcExecution, DataCalcTempData}
+import org.apache.linkis.engineplugin.spark.datacalc.model.{DataCalcArrayData, DataCalcGroupData}
+import org.apache.linkis.engineplugin.spark.entity.SparkEngineSession
+import org.apache.linkis.engineplugin.spark.extension.SparkPostExecutionHook
+import org.apache.linkis.engineplugin.spark.utils.{EngineUtils, JobProgressUtil}
+import org.apache.linkis.governance.common.exception.LinkisJobRetryException
+import org.apache.linkis.governance.common.paser.EmptyCodeParser
+import org.apache.linkis.manager.common.entity.enumeration.NodeStatus
+import org.apache.linkis.manager.common.entity.resource._
+import org.apache.linkis.manager.label.entity.Label
+import org.apache.linkis.manager.label.entity.engine.CodeLanguageLabel
+import org.apache.linkis.protocol.engine.JobProgressInfo
+import org.apache.linkis.scheduler.executer.{
+ CompletedExecuteResponse,
+ ErrorExecuteResponse,
+ ExecuteResponse,
+ SuccessExecuteResponse
+}
+
+import org.apache.arrow.memory.OutOfMemoryException
+import org.apache.commons.lang.exception.ExceptionUtils
+import org.apache.orc.storage.common.io.Allocator.AllocatorOutOfMemoryException
+
+import java.lang.reflect.InvocationTargetException
+import java.util
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+class SparkDataCalcExecutor(sparkEngineSession: SparkEngineSession, id: Long)
Review Comment:
Is it possible to inherit: SparkEngineConnExecutor
##########
linkis-engineconn-plugins/spark/src/main/java/org/apache/linkis/engineplugin/spark/datacalc/exception/DatabaseNotConfigException.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.datacalc.exception;
+
+public class DatabaseNotConfigException extends RuntimeException {
Review Comment:
should extend LinkisRuntimeException
##########
linkis-engineconn-plugins/spark/src/main/java/org/apache/linkis/engineplugin/spark/datacalc/exception/HiveSinkException.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.datacalc.exception;
+
+public class HiveSinkException extends RuntimeException {
Review Comment:
should extend LinkisRuntimeException
##########
linkis-engineconn-plugins/spark/src/main/java/org/apache/linkis/engineplugin/spark/datacalc/model/DataCalcArrayData.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.datacalc.model;
+
+import java.io.Serializable;
+
+import com.alibaba.fastjson.JSON;
+
+public class DataCalcArrayData extends DataCalcPluginConfig implements Serializable {
+
+ private DataCalcDataConfig[] plugins;
+
+ public DataCalcDataConfig[] getPlugins() {
+ return plugins;
+ }
+
+ public void setPlugins(DataCalcDataConfig[] plugins) {
+ this.plugins = plugins;
+ }
+
+ public static DataCalcArrayData getData(String data) {
+ return JSON.parseObject(data, DataCalcArrayData.class);
Review Comment:
should use BDPJettyServerHelper.gson
##########
linkis-engineconn-plugins/spark/src/main/java/org/apache/linkis/engineplugin/spark/datacalc/exception/ConfigRuntimeException.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.datacalc.exception;
+
+public class ConfigRuntimeException extends RuntimeException {
Review Comment:
should extend LinkisRuntimeException
##########
linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/engine/RunType.scala:
##########
@@ -43,4 +43,6 @@ object RunType extends Enumeration {
val TRINO_SQL = Value("tsql")
+ val DATA_CALC = Value("data_calc") // spark datacalc (ETL)
Review Comment:
What does datacalc mean?
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/datacalc/sink/HiveSink.scala:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.engineplugin.spark.datacalc.sink
+
+import org.apache.linkis.engineplugin.spark.datacalc.api.DataCalcSink
+import org.apache.linkis.engineplugin.spark.datacalc.exception.{
+ DatabaseNotConfigException,
+ HiveSinkException
+}
+import org.apache.linkis.engineplugin.spark.datacalc.service.LinkisDataSourceService
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.{DataFrame, DataFrameWriter, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRelation}
+import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.sources.DataSourceRegister
+import org.apache.spark.sql.types.StructField
+
+import org.slf4j.{Logger, LoggerFactory}
+
+class HiveSink extends DataCalcSink[HiveSinkConfig] {
+
+ private val log: Logger = LoggerFactory.getLogger(classOf[HiveSink])
+
+ def output(spark: SparkSession, ds: Dataset[Row]): Unit = {
+ val db = LinkisDataSourceService.getDatabase(config.getTargetDatabase)
+ if (db == null) {
+ throw new DatabaseNotConfigException(
+ s"Database ${config.getTargetDatabase} is not configured!"
+ )
+ }
+
+ val targetTable = db.getDatabaseName + "." + config.getTargetTable
+ val targetFields = spark.table(targetTable).schema.fields
+ if (config.getWriteAsFile != null && config.getWriteAsFile) {
+ val partitionsColumns = spark.catalog
+ .listColumns(targetTable)
+ .where(col("isPartition") === true)
+ .select("name")
+ .collect()
+ .map(_.getAs[String]("name"))
+ val location = getLocation(spark, targetTable, partitionsColumns)
+ val fileFormat = getTableFileFormat(spark, targetTable)
+
+ log.info(
+ s"Write $fileFormat into target table: $targetTable, location: $location, file format: $fileFormat"
+ )
+ val writer = getSaveWriter(
+ ds,
+ targetFields.filter(field => !partitionsColumns.contains(field.name)),
+ targetTable
+ )
+ fileFormat match {
+ case FileFormat.PARQUET => writer.parquet(location)
+ case FileFormat.ORC => writer.orc(location)
+ case _ =>
+ }
+
+ val partition = partitionsColumns
+ .map(colName => s"$colName='${config.getVariables.get(colName)}'")
+ .mkString(",")
+ if (StringUtils.isNotBlank(partition)) {
+ log.info(s"Refresh table partition: $partition")
+ refreshPartition(spark, targetTable, partition)
+ }
+ } else {
+ val writer = getSaveWriter(ds, targetFields, targetTable)
+ log.info(s"InsertInto data to hive table: $targetTable")
+ writer.format("hive").insertInto(targetTable)
+ }
+ }
+
+ def getSaveWriter(
+ ds: Dataset[Row],
+ targetFields: Array[StructField],
+ targetTable: String
+ ): DataFrameWriter[Row] = {
+ val dsSource = sequenceFields(ds, ds.schema.fields, targetFields, targetTable)
+ val sourceFields = dsSource.schema.fields
+
+ // 开启强校验时，校验字段类型是否匹配
Review Comment:
use English
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org