You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "danielemoraschi (via GitHub)" <gi...@apache.org> on 2026/06/20 16:34:56 UTC
[PR] fix(customize): skip orphaned _raw_data_id rows instead of aborting extraction [devlake]
danielemoraschi opened a new pull request, #8946:
URL: https://github.com/apache/devlake/pull/8946
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
### ⚠️ Pre Checklist
> Please complete _ALL_ items in this checklist, and remove before submitting
- [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/).
- [x] I have added relevant tests.
- [x] I have added relevant documentation.
- [x] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc.
<!--
Thanks for submitting a pull request!
We appreciate you spending the time to work on these changes.
Please fill out as many sections below as possible.
-->
### Summary
ExtractCustomizedFields used a `LEFT JOIN` to the raw table, so a domain row whose `_raw_data_id` has no matching raw record yields `NULL` data. That hit the type switch default branch, which did `return nil` and exited the whole subtask, silently leaving every later row in the (unordered) cursor unprocessed while still reporting success.
Change the default branch to `continue` so an orphaned row is skipped and extraction proceeds with the remaining rows.
### Does this close any open issues?
Closes #8945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(customize): skip orphaned _raw_data_id rows instead of aborting extraction [devlake]
Posted by "klesh (via GitHub)" <gi...@apache.org>.
klesh commented on code in PR #8946:
URL: https://github.com/apache/devlake/pull/8946#discussion_r3652518012
##########
backend/plugins/customize/tasks/customized_fields_extractor.go:
##########
@@ -142,7 +142,9 @@ func extractCustomizedFields(ctx context.Context, d dal.Dal, table, rawTable, ra
}
}
default:
- return nil
+ // orphaned/NULL raw data (LEFT JOIN found no matching raw record): skip this
+ // row and keep scanning, instead of aborting the whole subtask.
Review Comment:
Could you please print a warning message here? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(customize): skip orphaned _raw_data_id rows instead of aborting extraction [devlake]
Posted by "danielemoraschi (via GitHub)" <gi...@apache.org>.
danielemoraschi commented on PR #8946:
URL: https://github.com/apache/devlake/pull/8946#issuecomment-5182189092
Closing this as superseded by #9027, which landed on `main` in 5eee167.
That change covers the same root cause. The LEFT JOIN against the raw table
yields a NULL `data` column for a domain row whose `_raw_data_id` no longer has
a matching raw record, and that value fell through to `default: return nil`,
ending the scan. #9027 also goes further than this PR did:
* @klesh's review note is addressed there. `extractCustomizedFields` returns the
number of orphaned rows, and `ExtractCustomizedFields` logs a warning naming
the table, the raw table, and the count.
* `rows.Err()` is now returned at the end of the scan, so an error raised while
iterating the cursor is no longer discarded. That is the same class of silent
partial processing this PR left in place.
* Test coverage is broader: skipping an orphan and continuing, every row
orphaned, no orphans at all, and a cancelled context. This PR had one
regression test.
Nothing in this branch is additive on top of that, so there is no point in
rebasing. Thanks @klesh for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(customize): skip orphaned _raw_data_id rows instead of aborting extraction [devlake]
Posted by "danielemoraschi (via GitHub)" <gi...@apache.org>.
danielemoraschi closed pull request #8946: fix(customize): skip orphaned _raw_data_id rows instead of aborting extraction
URL: https://github.com/apache/devlake/pull/8946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org