You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "okumin (via GitHub)" <gi...@apache.org> on 2025/10/06 03:34:43 UTC
[PR] HIVE-29245: Normalize Java license headers [hive]
okumin opened a new pull request, #6115:
URL: https://github.com/apache/hive/pull/6115
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Normalize license headers of all Java files.
The following ones are exceptional.
- `iceberg/**`: most of them are ported from `apache/iceberg`
- [serde/src/java/org/apache/hadoop/hive/serde2/lazy/fast/StringToDouble.java](https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazy/fast/StringToDouble.java): Looks like it was ported from another library
- [llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/PriorityBlockingDeque.java](https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/PriorityBlockingDeque.java): Ditto
- `hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/*.java`: Generated by Avro
- `kafka-handler/src/test/gen/org/apache/hadoop/hive/kafka/SimpleRecord.java`: Ditto
### Why are the changes needed?
Our CI strictly verifies license headers of new files. I suppose many people copy and paste headers from existing files for convenience, so, I sometimes observe CI failing because of the format of licence headers.
As additional bonuses, consistent headers will reduce legal risks and future maintenance costs.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
I put the following configuration on `checkstyle/checkstyle.xml`, `standalone-metastore/checkstyle/checkstyle.xml`, and ``storage-api/checkstyle/checkstyle.xml. After that, I ran `mvn checkstyle:check`.
```xml
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.2//EN"
"http://www.puppycrawl.com/dtds/configuration_1_2.dtd">
<module name="Checker">
<module name="Header">
<property name="headerFile" value="${config_loc}/asf.header"/>
<property name="fileExtensions" value="java"/>
</module>
</module>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3388147950
@InvisibleProgrammer
My mistake is that I included whitespace changes and another ad-hoc change in a single pull request, and the ad-hoc styles are the headache to make the entire reformat hard. So, I tried to address them first. Please feel free to ask me if it is still huge.
https://github.com/apache/hive/pull/6125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3382886230
@okumin , it seems that the bottom part of license header is not indented:
For example in [AccumuloConnectionParameters.java](https://github.com/apache/hive/pull/6115/files#diff-e3e24edfaca3f85957562f0d5d55ed08bb26efa45e2a0f58dad550c29e5fa71a)
```
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
```
vs https://www.apache.org/legal/src-headers.html
```
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3384023450
@okumin, IMO we should change the expected file to be as same on asf website as mentioned above.
For example, iceberg project (failry new project) is using latest license header
https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/AncestorsOfProcedure.java
But flip side is number of files modified in this PR might substantially increase 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3370726960
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6115) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6115&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6115&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6115&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6115&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6115&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6115)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "InvisibleProgrammer (via GitHub)" <gi...@apache.org>.
InvisibleProgrammer commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3385553152
> If we switch to the following PR, is it possible to review it?
> https://github.com/apache/hive/pull/6122
TBH, I see no difference. The second PR has more than 700 file changes. And it doesn't even build due to the pom.xml of storage-api.
It contains 9 commits. And still, nothing shows what script modified the files.
Honestly, I love the concept of fixing those header files and forcing having the proper header files. But at this PR I don't see how the change can properly reviewed.
And also, the thing that actually happens, still hidden.
I mean, it is possible to spot the extra checks after the change. But it is not possible to spot how it changed. Is there a script that did the change.
I give you one example about what I'm thinking about:
Iceberg applied spotless back then. It ended up changing more than 3000 files. But it happened in only a few, individually understandable commits. And all the generated changes are in a single one commit. And this commit doesn't contain anything but a generated content.
In that way, it is easy to understand what happened and how happened. And also, it is easy to repeat the process if it is required.
[Ihttps://github.com/apache/iceberg/pull/5312/commits](https://github.com/apache/iceberg/pull/5312/commits)
At this PR, it is easy to understand the end goal. But I don't see the how.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3375047553
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6115) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6115&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6115&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6115&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6115&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6115&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6115)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3382978944
FYI,
I am checking using spotless plugin:
```
<plugin>
<groupId>com.diffplug.spotless</groupId>
<artifactId>spotless-maven-plugin</artifactId>
<version>2.46.1</version>
<configuration>
<java>
<licenseHeader>
<file>${maven.multiModuleProjectDirectory}/style/java-header</file>
</licenseHeader>
<excludes>
<exclude>**/LazyBinaryUnion.java</exclude>
<exclude>**/JsonSerde.java</exclude>
</excludes>
</java>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3384142383
@InvisibleProgrammer @Aggarwal-Raghav
If we switch to the following PR, is it possible to review it?
https://github.com/apache/hive/pull/6122
> @okumin, IMO we should change the expected file to be as same on asf website as mentioned above.
I personally think we can postpone the conclusion for three reasons. First, the current issue is the reviewers' cost in explaining why SonarQube complains about contributors' headers every time. Second, once we have completed normalizing all headers, a single script can rewrite all at once, and it is easy to write such a script, validate the script, or validate the result. Finally, ours is likely to be legal, hopefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3383669021
@InvisibleProgrammer
It is challenging to specify a single script because the header variations of the original code differ. As explained, I validated the final deliverable using Checkstyle. If we want more substantial evidence, I can split this PR into some PRs per a set of patterns.
### How was this patch tested?
I put the following configuration on `checkstyle/checkstyle.xml`, `standalone-metastore/checkstyle/checkstyle.xml`, and ``storage-api/checkstyle/checkstyle.xml. After that, I ran `mvn checkstyle:check`.
```xml
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.2//EN"
"http://www.puppycrawl.com/dtds/configuration_1_2.dtd">
<module name="Checker">
<module name="Header">
<property name="headerFile" value="${config_loc}/asf.header"/>
<property name="fileExtensions" value="java"/>
</module>
</module>
```
And I manually checked changes with `git diff master...HIVE-29245-asf-header`
@Aggarwal-Raghav
Just a clarification, does it mean we should update our expected files? Probably no.
- [storage-api](https://github.com/apache/hive/blob/master/storage-api/checkstyle/asf.header)
- [standalone-metastore](https://github.com/apache/hive/blob/master/standalone-metastore/checkstyle/asf.header)
- [Everything else](https://github.com/apache/hive/blob/master/checkstyle/asf.header)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3388082951
@InvisibleProgrammer
Thanks for the feedback. I added what commands are used or what are hand-maded.
```
commit c71e63c1d4b44137589519d13608eb197a03279b (HEAD -> HIVE-29245-regex-indent, origin/HIVE-29245-regex-indent)
Author: okumin <ok...@apache.org>
Date: Thu Oct 9 14:00:18 2025 +0900
Don't run RAT for checkstyle
commit 1292595d685be4056fc7ff9ad5d73a3ae6aaad1c
Author: okumin <ok...@apache.org>
Date: Thu Oct 9 13:40:43 2025 +0900
Add RegexpHeader
commit 76a30b762d486cc7d83235f2babcf0856a8fd844
Author: okumin <ok...@apache.org>
Date: Thu Oct 9 13:36:12 2025 +0900
Modify irregular files by hand
commit 0c349da0c8f3d5cd337d07613956c367ff2a2168
Author: okumin <ok...@apache.org>
Date: Thu Oct 9 13:19:59 2025 +0900
Add asterisk to some irregular files
By hand
commit 6a1b7ae9061cb61b15d7c75002e379014efbeaad
Author: okumin <ok...@apache.org>
Date: Thu Oct 9 13:11:45 2025 +0900
Normalize # of indents of irregular LLAP patterns
git ls-files '*.java' | grep -v 'src/gen/thrift' | grep -v '^iceberg/' \
| xargs perl -pi -e 's|^ \* you may not use this file except in compliance with the License\.$| * you may not use this file except in compliance with the License.|'
git ls-files '*.java' | grep -v 'src/gen/thrift' | grep -v '^iceberg/' \
| xargs perl -pi -e 's|^ \* You may obtain a copy of the License at$| * You may obtain a copy of the License at|'
commit f90704c64312f53ef980fb52822dc2c9e9acd5e2
Author: okumin <ok...@apache.org>
Date: Thu Oct 9 13:02:35 2025 +0900
Reduce extra indents, irregular pattern 1
git ls-files '*.java' | grep -v 'src/gen/thrift' | grep -v '^iceberg/' \
| xargs perl -pi -e 's|^ \* \* Licensed to the Apache Software Foundation \(ASF\) under one$| * * Licensed to the Apache Software Foundation (ASF) under one|'
git ls-files '*.java' | grep -v 'src/gen/thrift' | grep -v '^iceberg/' \
| xargs perl -pi -e 's|^ \* \* or more contributor license agreements\. See the NOTICE file$| * * or more contributor license agreements. See the NOTICE file|'
git ls-files '*.java' | grep -v 'src/gen/thrift' | grep -v '^iceberg/' \
| xargs perl -pi -e 's|^ \* \* distributed with this work for additional information$| * * distributed with this work for additional information|'
git ls-files '*.java' | grep -v 'src/gen/thrift' | grep -v '^iceberg/' \
| xargs perl -pi -e 's|^ \* \* regarding copyright ownership\. The ASF licenses this file$| * * regarding copyright ownership. The ASF licenses this file|'
```
And the diff is 500 lines. Is this still huge? In that case, I will split it more.
```
% git diff master...HIVE-29245-regex-indent -w | wc -l
555
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin closed pull request #6115: HIVE-29245: Normalize Java license headers
URL: https://github.com/apache/hive/pull/6115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29245: Normalize Java license headers [hive]
Posted by "InvisibleProgrammer (via GitHub)" <gi...@apache.org>.
InvisibleProgrammer commented on PR #6115:
URL: https://github.com/apache/hive/pull/6115#issuecomment-3381725531
This PR contains ~1400 file changes in 45 commits. Most of the changes are generated. Could you please clarify which files/commits worth reviewing?
Is there a script that generated the changes or a plugin execution? If yes, what was the tool or command that you ran?
TBH, I'm afraid, in that format the change will be squashed onto one single commit and later, it will be extremely hard to figure out what happened.
Is it possible to create two commits instead of 45? One commit for the changes around the rules, exceptions, generator script, etc. And one for the generated content. In that way the git history will be obvious.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org