You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "xglv1985 (via GitHub)" <gi...@apache.org> on 2025/02/10 03:03:23 UTC
[I] [Bug] Lacking "The judgment to Empty" introduces a "wrong value of a Spark lineage object" issue [kyuubi]
xglv1985 opened a new issue, #6912:
URL: https://github.com/apache/kyuubi/issues/6912
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
## How to reproduce the issue?
The changes in this PR will avoid a wrong result when generating the instance of org.apache.kyuubi.plugin.lineage.Lineage, in the certain case as follows:
step 1: create a temporary view from a file
`CREATE OR REPLACE TEMPORARY VIEW temp_view
(
`a` STRING COMMENT '',
`b` STRING COMMENT ''
)
USING csv OPTIONS(
sep='\t',
path='${sourceFile.path}'
);`
step 2: insert into a table by selecting from the temporary view in step 1
`insert overwrite table test_db.test_table_from_dir
SELECT
`a`,
`b`
FROM temp_view`
step 3: generate the lineage when executing the insert statement in step 2
Then a **None** object of org.apache.kyuubi.plugin.lineage.Lineage will be generated. However, the correct value of it should be:
`inputTables(List())
outputTables(List(spark_catalog.test_db.test_table_from_dir))
columnLineage(List(ColumnLineage(spark_catalog.test_db.test_table_from_dir.a0,Set()), ColumnLineage(spark_catalog.test_db.test_table_from_dir.b0,Set())))`
## How is the issue introduced?
Let's see the current code when getting the Lineage object by resolving a LogicalPlan object:
<img width="694" alt="image" src="https://github.com/user-attachments/assets/65256a0d-320d-4271-968f-59eafb74de9f" />
According to the above logic, a None org.apache.kyuubi.plugin.lineage.Lineage object will be generated due to "try-recover" self-protection, in this certain case.
## The consequence of this bug
### Unit Test Environment
In Unit Test, when the code runs here a "None.get" exception will be raised:
<img width="682" alt="image" src="https://github.com/user-attachments/assets/102dc9bd-294f-4b1e-b1c6-01b6fee50fed" />
Here's the runtime exception stack:
`None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineageWithoutExecuting(SparkSQLLineageParserHelperSuite.scala:1485)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$83(SparkSQLLineageParserHelperSuite.scala:1465)`
### Production Environment
This Lineage object cannot be used in the production environment because it has a None value which lacks necessary lineage information.
### Affects Version(s)
all versions
### Kyuubi Server Log Output
```logtalk
unrelated
```
### Kyuubi Engine Log Output
```logtalk
unrelated
```
### Kyuubi Server Configurations
```yaml
unrelated
```
### Kyuubi Engine Configurations
```yaml
unrelated
```
### Additional context
I have already proposed a PR to this issue, please see:
https://github.com/apache/kyuubi/pull/6911
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Lacking "The judgment to Empty" introduces a "wrong value of a Spark lineage object" issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #6912: [Bug] Lacking "The judgment to Empty" introduces a "wrong value of a Spark lineage object" issue
URL: https://github.com/apache/kyuubi/issues/6912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Lacking "The judgment to Empty" introduces a "wrong value of a Spark lineage object" issue [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6912:
URL: https://github.com/apache/kyuubi/issues/6912#issuecomment-2646816854
Hello @xglv1985,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org