You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jbonofre (via GitHub)" <gi...@apache.org> on 2023/12/13 14:23:14 UTC
[PR] GH-39214: [Java] Support reproducible build [arrow]
jbonofre opened a new pull request, #39215:
URL: https://github.com/apache/arrow/pull/39215
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
This PR add reproducible builds support.
### Are these changes tested?
I tested the builds are reproducible on `arrow-vector`, `arrow-memory`, ... using `mvn clean verify artifact:compare`.
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1884280923
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1884885503
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446979355
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
My question is, is this only enabled for specific build variants or options? Or would I get those fixed timestamps if I do a local Java build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875675843
@jbonofre the crossbow task uses * to match string patterns. You can use `*java*` and it will work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1442205290
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
The most important is to have the same output timestamp. I can use 1T 1970. I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1862850000
See https://github.com/apache/arrow/pull/39279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1862830036
https://maven.apache.org/plugins/maven-artifact-plugin/compare-mojo.html
> Compare current build output (from package) against reference either previously install-ed or downloaded from a remote repository: comparison results go to .buildcompare file.
So maybe we finally have a useful thing to do in the binary JAR verification step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446987158
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
It's only the archives/artifacts. And it's good practice generally speaking (we do that in most of Apache projects).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446988802
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
Fair enough. I was expecting this to be enabled selectively when building public packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1854631669
@lidavidm let me check why GH Action is not happy 😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1426151802
##########
java/format/pom.xml:
##########
@@ -23,6 +23,10 @@
<name>Arrow Format</name>
<description>Generated Java files from the IPC Flatbuffer definitions.</description>
+ <properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+ </properties>
Review Comment:
You are right for the check: mvn clean install and then mvn clean verify artifact:compare do a check.
This PR ensures we have reproducible build, not meaning that some module are not ok already (AFAIR, flight and algorithm were not reproducible).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1426870119
##########
java/pom.xml:
##########
@@ -28,6 +28,7 @@
<url>https://arrow.apache.org/</url>
<properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
Review Comment:
Good point. It makes sense. Let me update the PR accordingly. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1874222473
Revision: 6d91fddd6ce3bfa28cbb25421cb671b1b737147e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5e64edb2c5](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5e64edb2c5)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7387879193/job/20097514183)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7387879054/job/20097512103)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7387879086/job/20097512072)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7387879062/job/20097512052)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7387879061/job/20097512013)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7387879114/job/20097512480)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1863659173
How about this for "Dev /Source Release and Merge Script" failures?
```diff
diff --git a/dev/release/01-prepare-test.rb b/dev/release/01-prepare-test.rb
index 8fb23f45f0..f4bf01f1f6 100644
--- a/dev/release/01-prepare-test.rb
+++ b/dev/release/01-prepare-test.rb
@@ -264,18 +264,17 @@ class PrepareTest < Test::Unit::TestCase
end
Dir.glob("java/**/pom.xml") do |path|
- version = "<version>#{@snapshot_version}</version>"
- lines = File.readlines(path, chomp: true)
- target_lines = lines.grep(/#{Regexp.escape(version)}/)
- hunks = []
- target_lines.each do |line|
- new_line = line.gsub(@snapshot_version) do
- @release_version
+ hunks = generate_hunks(File.readlines(path, chomp: true)) do |line|
+ if line.include?("<version>#{@snapshot_version}</version>")
+ new_line = line.gsub(@snapshot_version) do
+ @release_version
+ end
+ [line, new_line]
+ elsif line.include?("<project.build.outputTimestamp>")
+ [line, normalize_pom_xml_output_timestamp(line)]
+ else
+ [nil, nil]
end
- hunks << [
- "-#{line}",
- "+#{new_line}",
- ]
end
expected_changes << {hunks: hunks, path: path}
end
diff --git a/dev/release/post-11-bump-versions-test.rb b/dev/release/post-11-bump-versions-test.rb
index 4b6933d610..08a900f71c 100644
--- a/dev/release/post-11-bump-versions-test.rb
+++ b/dev/release/post-11-bump-versions-test.rb
@@ -244,37 +244,19 @@ class PostBumpVersionsTest < Test::Unit::TestCase
end
import_path = "github.com/apache/arrow/go/v#{@snapshot_major_version}"
- hunks = []
if release_type == :major
- lines = File.readlines(path, chomp: true)
- target_lines = lines.each_with_index.select do |line, i|
- line.include?(import_path)
- end
- next if target_lines.empty?
- n_context_lines = 3 # The default of Git's diff.context
- target_hunks = [[target_lines.first[0]]]
- previous_i = target_lines.first[1]
- target_lines[1..-1].each do |line, i|
- if i - previous_i < n_context_lines
- target_hunks.last << line
- else
- target_hunks << [line]
- end
- previous_i = i
- end
- target_hunks.each do |lines|
- hunk = []
- lines.each do |line,|
- hunk << "-#{line}"
- end
- lines.each do |line|
+ hunks = generate_hunks(File.readlines(path, chomp: true)) do |line|
+ if line.include?(import_path)
new_line = line.gsub("v#{@snapshot_major_version}") do
"v#{@next_major_version}"
end
- hunk << "+#{new_line}"
+ [line, new_line]
+ else
+ [nil, nil]
end
- hunks << hunk
end
+ else
+ hunks = []
end
if path == "go/parquet/writer_properties.go"
hunks << [
@@ -287,18 +269,17 @@ class PostBumpVersionsTest < Test::Unit::TestCase
end
Dir.glob("java/**/pom.xml") do |path|
- version = "<version>#{@snapshot_version}</version>"
- lines = File.readlines(path, chomp: true)
- target_lines = lines.grep(/#{Regexp.escape(version)}/)
- hunks = []
- target_lines.each do |line|
- new_line = line.gsub(@snapshot_version) do
- @next_snapshot_version
+ hunks = generate_hunks(File.readlines(path, chomp: true)) do |line|
+ if line.include?("<version>#{@snapshot_version}</version>")
+ new_line = line.gsub(@snapshot_version) do
+ @next_snapshot_version
+ end
+ [line, new_line]
+ elsif line.include?("<project.build.outputTimestamp>")
+ [line, normalize_pom_xml_output_timestamp(line)]
+ else
+ [nil, nil]
end
- hunks << [
- "-#{line}",
- "+#{new_line}",
- ]
end
expected_changes << {hunks: hunks, path: path}
end
diff --git a/dev/release/test-helper.rb b/dev/release/test-helper.rb
index 3b2c3aa6e5..4238a534d1 100644
--- a/dev/release/test-helper.rb
+++ b/dev/release/test-helper.rb
@@ -83,15 +83,59 @@ module GitRunnable
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>1970-01-01T00:00:00Z<"
+ end
+ end
+
+ def normalize_added_line(line)
+ normalize_pom_xml_output_timestamp(line)
+ end
end
module VersionDetectable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1865766858
Thanks. Let me try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2502366954
We're moving `java/` to https://github.com/apache/arrow-java .
Could you open a new PR in apache/arrow-java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1865104091
> Yeah, as said on a comment, I fixed the maven enforcer issue, I'm not on the merge action issue (testing locally).
"the maven enforcer issue" is https://github.com/apache/arrow/pull/39215#issuecomment-1862842437 , right?
I'm focusing on "Dev / Source Release and Merge Script on ubuntu-latest" CI job https://github.com/apache/arrow/actions/runs/7263564652/job/19789165030?pr=39215 not "the maven enforcer issue".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1859940643
FYI, I'm resuming work on this PR:
- checking the build failure on CI
- checking to add a GH Action workflow to verify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1862839971
@lidavidm yup, it's what I plan to do (and I used while working on this PR 😄 ).
First I will fix the failing checks (AMD64 Conda Integration Test, etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1863622567
It seems that updated value is always the current time: https://github.com/mojohaus/versions/pull/522/files#diff-544db8b7e4906f57cb6c1bd43ba1100760f876da22b46b2ca381ab374cb169faR530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1863617944
When is `<project.build.outputTimestamp>` updated?
Does `mvn versions:set` update it?
It seems that our `dev/release/01-prepare.sh` updates it: https://github.com/apache/arrow/actions/runs/7263564652/job/19789165030?pr=39215#step:7:24478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2028798852
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1901721889
@kou hm weird asf membership is public for David so the bot should pick it up... if this keeps happening we may need to add a non gh api based method to identify committers (e.g. committer list, IIRC we have a yaml one around somehwere right?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1874218913
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1854636429
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1425927853
##########
java/format/pom.xml:
##########
@@ -23,6 +23,10 @@
<name>Arrow Format</name>
<description>Generated Java files from the IPC Flatbuffer definitions.</description>
+ <properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+ </properties>
Review Comment:
When I tried these commands, it finished ok with or without any changes:
````
$ mvn clean install
$ mvn clean verify artifact:compare
...
[INFO] Saved info on build to /Users/dsusanibar/fork/arrow/java/format/target/arrow-format-15.0.0-SNAPSHOT.buildinfo
[INFO] Checking against reference build from central...
[INFO] Reference buildinfo file not found: it will be generated from downloaded reference artifacts
[INFO] Reference build java.version: 11 (from MANIFEST.MF Build-Jdk-Spec)
[INFO] Reference build os.name: Unix (from pom.properties newline)
[INFO] Minimal buildinfo generated from downloaded artifacts: /Users/dsusanibar/fork/arrow/java/format/target/reference/arrow-format-15.0.0-SNAPSHOT.buildinfo
[INFO] Reproducible Build output summary: 5 files ok
[INFO] Reproducible Build output comparison saved to /Users/dsusanibar/fork/arrow/java/format/target/arrow-format-15.0.0-SNAPSHOT.buildcompare
...
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1427759426
##########
java/pom.xml:
##########
@@ -28,6 +28,7 @@
<url>https://arrow.apache.org/</url>
<properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
Review Comment:
Done, I include the timestamp property only in modules building jar (not pom).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1427138620
##########
java/format/pom.xml:
##########
@@ -23,6 +23,10 @@
<name>Arrow Format</name>
<description>Generated Java files from the IPC Flatbuffer definitions.</description>
+ <properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+ </properties>
Review Comment:
What we might want is to add this to CI somewhere (or possibly the release verification process) so we can make sure it stays working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1864091895
> It seems that updated value is always the current time: https://github.com/mojohaus/versions/pull/522/files#diff-544db8b7e4906f57cb6c1bd43ba1100760f876da22b46b2ca381ab374cb169faR530
On a specific version, the `outputTimestamp` has to be the same. So settings the `outputTimestamp` when using `verisons:set` is fine. The most important is to have a "stable" `outputTimestamp` across builds of the same version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1904157757
I fixed my membership after that. You may want to update the bot's error message to tell people what to do, I only knew because I happened to see someone else ask the same question in a private channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1444763418
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
I updated to use `1980-01-01T00:00:02Z` to avoid issue with `maven-assembly-plugin`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446467584
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
Does this force the output timestamp for all builds regardless of build options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1884283152
Revision: f109f8572cf95b82006671a5fa6b767cb3d52186
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5d0c1311bb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5d0c1311bb)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7471315606/job/20331386716)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7471315680/job/20331386405)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7471315631/job/20331386255)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7471315605/job/20331386128)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7471315820/job/20331386654)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7471315663/job/20331386353)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1426862129
##########
java/pom.xml:
##########
@@ -28,6 +28,7 @@
<url>https://arrow.apache.org/</url>
<properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
Review Comment:
Thank you:
````
$ mvn clean artifact:check-buildplan
[ERROR] project.build.outputTimestamp property should not be inherited but defined in POM /Users/dsusanibar/Documents/business/voltron/fork/arrow/java/format/pom.xml
````
If it is required on all projects, it will give us clarity if it is only defined on child projects and not parent project. What do you think about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1854639466
Revision: 502eb177c8265ff30879bd41db09380d6d52abe7
Submitted crossbow builds: [ursacomputing/crossbow @ actions-b2f5994f24](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b2f5994f24)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7200551614/job/19614749801)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7200551521/job/19614747974)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7200551542/job/19614747960)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7200551587/job/19614748296)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7200551525/job/19614747945)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7200551647/job/19614748698)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1426150367
##########
java/pom.xml:
##########
@@ -28,6 +28,7 @@
<url>https://arrow.apache.org/</url>
<properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
Review Comment:
It's possible but not recommended: if you do artifact:check-buildplan it will raise an error in the case it's inherited from parent pom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1427760531
##########
java/format/pom.xml:
##########
@@ -23,6 +23,10 @@
<name>Arrow Format</name>
<description>Generated Java files from the IPC Flatbuffer definitions.</description>
+ <properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+ </properties>
Review Comment:
@lidavidm good idea. Let me check to add a workflow in GH actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1863620913
https://maven.apache.org/guides/mini/guide-reproducible-builds.html#faq
> if you have a custom release process tooling, you'll need to add the feature to your release tooling. Notice that if you're using versions-maven-plugin in custom release scripts, starting with release 2.9.0, [versions:set goal updates the property](https://github.com/mojohaus/versions-maven-plugin/issues/453).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1862111613
How exactly does this work, it compares the current build against a reference build? How does that work for dev versions?
We do have binary verification scripts so it would make sense to add it there, that would automatically also run it. `dev/release/verify-release-candidate.sh` with `TEST_JARS=1' is run by the jar binary verification script (this downloads previously build nightly jars iirc) cc @raulcd who is a bit more familiar with the release scripts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446986163
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
Wouldn't it be confusing in some situations then, to have files dated to some random past value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1445710273
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
> 1. We need `project.build.outputTimestamp` in the `pom.xml` as it's used by maven plugins to ensure the build is reproducible.
Yes. I think so too.
> 2. `project.build.outputTimestamp` value has to be fixed (not changing at each build). The value doesn't matter if it's always the same. That's the purpose of the setting the value (it's what I did in all other Apache projects where I'm working on).
Yes. I think so too.
> 3. We can't use `1970-01-01` else maven-assembly-plugin complains, that's why I used the first allowed value in the range which is `1980-01-01T00:00:02Z`. So I used the same in the `test-helper.rb` to be in sync.
Yes and no. We don't need to use special timestamps such as `1970-...` and `1980-...` in `pom.xml`. `test-helper.rb` normalizes all timestamps such as `2023-12-13T00:00:00Z` in `pom.xml`. So we don't need to synchronize timestamps in `pom.xml` and `test-helper.rb`. So we can use `2023-12-13T00:00:00Z` in `pom.xml` like you did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446947595
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
Hey @pitrou . The `project.build.outputTimestamp` property is used by several Maven plugins to ensure the consistency of the generated artifacts (assembly, jar, source, ...). They "hook" in their goals on phases where it's required. So, yes, for instance, using `package` phase will set the output timestamp on the artifacts. Does it answer your question (not sure I understood your question correctly ;) ) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1877665409
I will complete the pr tomorrow (finishing Apache Karaf release right now ;)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1445549249
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
Hmm, it seems that we have miscommunication. Here are what I think:
* We need the `project.build.outputTimestamp` element in the `pom.xml`.
* But we don't need to use a special timestamp for the element in the `pom.xml` like you did in the first commit in this PR.
`dev/release/test-helper.rb` updates the element's timestamp because it runs partial our release process. And the process updates `project.build.outputTimestamp`'s timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1444776173
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
We don't need to use the special timestamp in `pom.xml`. (Timestamps in `pom.xml` are updated automatically in our release process, right? https://github.com/apache/arrow/pull/39215#issuecomment-1863620913 )
We normalize timestamps in `pom.xml` for testing by this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1444897408
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
I think it's better (for both visibility and testing) to have `project.build.outputTimestamp` in the `pom.xml`. Or at least, I can replace it with a comment documenting that `project.build.outputTimestamp` is populating by the `dev/release/test-helper.rb`. Thoughts ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1862842437
```
Error: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.4.1:enforce (enforce-maven-version) on project arrow-bom:
Error: Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion failed with message:
Error: Detected Maven Version: 3.5.0 is not in the allowed range [3.6.3,).
Error: -> [Help 1]
```
That's due to the Apache POM update. I will propose fix as part of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1456119954
##########
cpp/submodules/parquet-testing:
##########
Review Comment:
Ah yeah, possible indeed. Let me try thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2028821609
@lidavidm as I'm back on Arrow, I'm resuming several PRs/works I have in progress 😄 Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1439190982
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
My bad (bad vim configuration). Let me fix the indent. Sorry about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2143131667
@jbonofre need any help here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1901191614
@assignUser Could you take a look at https://github.com/apache/arrow/pull/39215#issuecomment-1900481478 and https://github.com/apache/arrow/pull/39215#issuecomment-1900482331 ?
#39610 is related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed pull request #39215: GH-39214: [Java] Support reproducible build
URL: https://github.com/apache/arrow/pull/39215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1900482331
```
Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7584811545
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1874366346
```
Error: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.6.0:single (default-cli) on project arrow-bom: Error reading assemblies: No assembly descriptors found. -> [Help 1]
```
hmm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1442204426
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
Ah ok. On most of other Apache projects I'm working we do a single commit to use simple merge. Sorry. I will fix indent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875671907
```
Unable to match any tasks for `java`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7400052099
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875670066
@lidavidm the assembly plugin failed on the BoM. I'm checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875678586
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1442171379
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
Oh, why...?
You broke it again...: https://github.com/apache/arrow/compare/6d91fddd6ce3bfa28cbb25421cb671b1b737147e..cb22662007ecada122cd6d441546e3385c31e906#diff-7655978b5793694e21d21f4c483847ed0004cd1035ebd91a87c862c246bbb83a
You don't need to squash all your commits in a PR because we use "squash & merge".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1881154580
Revision: 1d9640b5a61865c3ce340fcaf966fd619168dd35
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1b711b30db](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1b711b30db)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7449053702/job/20264793334)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7449053718/job/20264791727)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7449053634/job/20264791348)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7449053652/job/20264791339)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7449053822/job/20264791981)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7449053860/job/20264792331)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1425923128
##########
java/pom.xml:
##########
@@ -28,6 +28,7 @@
<url>https://arrow.apache.org/</url>
<properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
Review Comment:
The parameter 'project.build.outputTimestamp' is defined at the level of the parent project. Should it also be added to child projects?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2325816070
I'm back on Arrow. I will rebase and retest this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2501737071
:warning: GitHub issue #39214 **has no components**, please add labels for components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1860485287
> > checking to add a GH Action workflow to verify
>
> It might be best to add it to an existing workflow, or again, adding it to the release verification script even (what do you think @assignUser?)
@lidavidm it makes sense, I do like this. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1865613653
@kou Maven enforcer fix is part of this PR: https://github.com/apache/arrow/pull/39215/files#diff-d8225ebcfc11b480a6e4f54e183b67c3ead51635a167c106d928c2abf1f9ef66R31 and https://github.com/apache/arrow/pull/39215/files#diff-65ce66a6aeda30354aedbae081d0411a4ae6561f9d4745b86117ea7e3eb1839aR29
I'm on the "Release and Merge script" investigation right now 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1865675684
> I'm on the "Release and Merge script" investigation right now 😄
https://github.com/apache/arrow/pull/39215#issuecomment-1863659173 will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1446980679
##########
java/adapter/avro/pom.xml:
##########
@@ -25,6 +25,10 @@
<description>(Contrib/Experimental) A library for converting Avro data to Arrow data.</description>
<url>http://maven.apache.org</url>
+ <properties>
+ <project.build.outputTimestamp>1980-01-01T00:00:02Z</project.build.outputTimestamp>
Review Comment:
It applies to any build including your local one. So you can have a reproducible build locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1874326383
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1442171969
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
+ end
+ end
+
+ def normalize_added_line(line)
+ normalize_pom_xml_output_timestamp(line)
+ end
+
Review Comment:
```suggestion
```
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
Why did you change the normalized timestamp to `2023-12-13` from `1970-01-01`?
I think that special date (`1970-01-01` is a special date because UNIX time uses it as epoch) is better for easy to understand that we normalize our timestamps.
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
```suggestion
when /\A\+/
next unless in_hunk
diffs.last[:hunks].last << normalize_added_line(line.chomp)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1874383735
Weird. Let me check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1439225159
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
No problem. I've fixed the indent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875678064
@raulcd thanks, my bad (typo mistake 😄 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1900626626
@lidavidm yup, to be honest, I don't know why it's failing 😄 I will probably try on `main` branch to compare.
I will do a new pass again, maybe I missed something 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1900481478
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1910256113
I'm rebasing and doing new tests on this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1884889242
Revision: cb600e923b5aef9446726dd945de9c2e4a7aa498
Submitted crossbow builds: [ursacomputing/crossbow @ actions-cf01383961](https://github.com/ursacomputing/crossbow/branches/all?query=actions-cf01383961)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7475885590/job/20345018418)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7475885583/job/20345016750)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7475885581/job/20345016735)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7475885693/job/20345017470)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7475885677/job/20345017467)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7475885630/job/20345017233)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1883176629
Revision: d6175aa8137f8d1731ad51e740cd35b0e7f43e6d
Submitted crossbow builds: [ursacomputing/crossbow @ actions-2d8fd7d0c6](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2d8fd7d0c6)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7462570314/job/20305180492)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7462570228/job/20305178505)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7462570092/job/20305177929)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7462570177/job/20305178126)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7462570146/job/20305178073)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7462570185/job/20305178327)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1883172152
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1860453806
> checking to add a GH Action workflow to verify
It might be best to add it to an existing workflow, or again, adding it to the release verification script even (what do you think @assignUser?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1863242841
I fixed the enforcer range by setting `minimalMavenBuildVersion`.
Let me take a look on the Merge Script workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1426152169
##########
java/format/pom.xml:
##########
@@ -23,6 +23,10 @@
<name>Arrow Format</name>
<description>Generated Java files from the IPC Flatbuffer definitions.</description>
+ <properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+ </properties>
Review Comment:
You can also check with artifact:check-buildplan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1436724207
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
I'll push indent fix for `dev/release/*.rb` changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2028799478
Revision: e6e5d16702b8b51cdeb038874631a931c9f3fc6b
Submitted crossbow builds: [ursacomputing/crossbow @ actions-02510129a8](https://github.com/ursacomputing/crossbow/branches/all?query=actions-02510129a8)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/8499228119/job/23279803604)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/8499228075/job/23279803257)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/8499228064/job/23279803183)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/8499228079/job/23279803206)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/8499228043/job/23279803197)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/8499228041/job/23279803080)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1425925099
##########
java/format/pom.xml:
##########
@@ -23,6 +23,10 @@
<name>Arrow Format</name>
<description>Generated Java files from the IPC Flatbuffer definitions.</description>
+ <properties>
+ <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+ </properties>
Review Comment:
Can you provide some commands or steps for validating a reproducible build, such as in the Arrow Format module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1439659286
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
Review Comment:
I pushed the fix as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1896192715
@lidavidm do you have an idea why `C++ / AMD64 Conda C++ AVX2`, `Java JNI / AMD64 manylinux2014 Kava JNI`, etc jobs are failing ?
It looks weird to me as it seems related to docker container execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1456116318
##########
cpp/submodules/parquet-testing:
##########
Review Comment:
Can you revert this change? That might be why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-2143375399
I can resume on this one yeah :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1854011477
:warning: GitHub issue #39214 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875681884
Revision: 0f44b8722769104bc9ed5c2cb559cc4d395bec4c
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1e2e249f36](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1e2e249f36)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7400121082/job/20133089629)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7400121057/job/20133088244)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7400121132/job/20133088860)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7400121097/job/20133088360)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7400121064/job/20133088275)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7400121133/job/20133088864)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1874328847
Revision: cb22662007ecada122cd6d441546e3385c31e906
Submitted crossbow builds: [ursacomputing/crossbow @ actions-0a53bd923d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-0a53bd923d)
|Task|Status|
|----|------|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions/runs/7388697244/job/20100036961)|
|verify-rc-source-java-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7388697158/job/20100035633)|
|verify-rc-source-java-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7388697242/job/20100036093)|
|verify-rc-source-java-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7388697208/job/20100035833)|
|verify-rc-source-java-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7388697214/job/20100035835)|
|verify-rc-source-java-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7388697184/job/20100035617)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1875670515
@github-actions crossbow submit java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1864089452
> When is `<project.build.outputTimestamp>` updated? Does `mvn versions:set` update it?
>
> It seems that our `dev/release/01-prepare.sh` updates it: https://github.com/apache/arrow/actions/runs/7263564652/job/19789165030?pr=39215#step:7:24478
`outputTimestamp` should be always the same to guarantee the same artifact accross build. I'm doing a pass on actions to be sure we are clean here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1864092947
> How about this for "Dev /Source Release and Merge Script" failures?
>
> ```diff
> diff --git a/dev/release/01-prepare-test.rb b/dev/release/01-prepare-test.rb
> index 8fb23f45f0..f4bf01f1f6 100644
> --- a/dev/release/01-prepare-test.rb
> +++ b/dev/release/01-prepare-test.rb
> @@ -264,18 +264,17 @@ class PrepareTest < Test::Unit::TestCase
> end
>
> Dir.glob("java/**/pom.xml") do |path|
> - version = "<version>#{@snapshot_version}</version>"
> - lines = File.readlines(path, chomp: true)
> - target_lines = lines.grep(/#{Regexp.escape(version)}/)
> - hunks = []
> - target_lines.each do |line|
> - new_line = line.gsub(@snapshot_version) do
> - @release_version
> + hunks = generate_hunks(File.readlines(path, chomp: true)) do |line|
> + if line.include?("<version>#{@snapshot_version}</version>")
> + new_line = line.gsub(@snapshot_version) do
> + @release_version
> + end
> + [line, new_line]
> + elsif line.include?("<project.build.outputTimestamp>")
> + [line, normalize_pom_xml_output_timestamp(line)]
> + else
> + [nil, nil]
> end
> - hunks << [
> - "-#{line}",
> - "+#{new_line}",
> - ]
> end
> expected_changes << {hunks: hunks, path: path}
> end
> diff --git a/dev/release/post-11-bump-versions-test.rb b/dev/release/post-11-bump-versions-test.rb
> index 4b6933d610..08a900f71c 100644
> --- a/dev/release/post-11-bump-versions-test.rb
> +++ b/dev/release/post-11-bump-versions-test.rb
> @@ -244,37 +244,19 @@ class PostBumpVersionsTest < Test::Unit::TestCase
> end
>
> import_path = "github.com/apache/arrow/go/v#{@snapshot_major_version}"
> - hunks = []
> if release_type == :major
> - lines = File.readlines(path, chomp: true)
> - target_lines = lines.each_with_index.select do |line, i|
> - line.include?(import_path)
> - end
> - next if target_lines.empty?
> - n_context_lines = 3 # The default of Git's diff.context
> - target_hunks = [[target_lines.first[0]]]
> - previous_i = target_lines.first[1]
> - target_lines[1..-1].each do |line, i|
> - if i - previous_i < n_context_lines
> - target_hunks.last << line
> - else
> - target_hunks << [line]
> - end
> - previous_i = i
> - end
> - target_hunks.each do |lines|
> - hunk = []
> - lines.each do |line,|
> - hunk << "-#{line}"
> - end
> - lines.each do |line|
> + hunks = generate_hunks(File.readlines(path, chomp: true)) do |line|
> + if line.include?(import_path)
> new_line = line.gsub("v#{@snapshot_major_version}") do
> "v#{@next_major_version}"
> end
> - hunk << "+#{new_line}"
> + [line, new_line]
> + else
> + [nil, nil]
> end
> - hunks << hunk
> end
> + else
> + hunks = []
> end
> if path == "go/parquet/writer_properties.go"
> hunks << [
> @@ -287,18 +269,17 @@ class PostBumpVersionsTest < Test::Unit::TestCase
> end
>
> Dir.glob("java/**/pom.xml") do |path|
> - version = "<version>#{@snapshot_version}</version>"
> - lines = File.readlines(path, chomp: true)
> - target_lines = lines.grep(/#{Regexp.escape(version)}/)
> - hunks = []
> - target_lines.each do |line|
> - new_line = line.gsub(@snapshot_version) do
> - @next_snapshot_version
> + hunks = generate_hunks(File.readlines(path, chomp: true)) do |line|
> + if line.include?("<version>#{@snapshot_version}</version>")
> + new_line = line.gsub(@snapshot_version) do
> + @next_snapshot_version
> + end
> + [line, new_line]
> + elsif line.include?("<project.build.outputTimestamp>")
> + [line, normalize_pom_xml_output_timestamp(line)]
> + else
> + [nil, nil]
> end
> - hunks << [
> - "-#{line}",
> - "+#{new_line}",
> - ]
> end
> expected_changes << {hunks: hunks, path: path}
> end
> diff --git a/dev/release/test-helper.rb b/dev/release/test-helper.rb
> index 3b2c3aa6e5..4238a534d1 100644
> --- a/dev/release/test-helper.rb
> +++ b/dev/release/test-helper.rb
> @@ -83,15 +83,59 @@ module GitRunnable
> when /\A@@/
> in_hunk = true
> diffs.last[:hunks] << []
> - when /\A[-+]/
> + when /\A-/
> next unless in_hunk
> diffs.last[:hunks].last << line.chomp
> + when /\A\+/
> + next unless in_hunk
> + diffs.last[:hunks].last << normalize_added_line(line.chomp)
> end
> end
> diffs.sort_by do |diff|
> diff[:path]
> end
> end
> +
> + def generate_hunks(lines)
> + git_diff_context = 3 # The default of Git's diff.context
> + max_lines_for_same_hunk = git_diff_context * 2 + 1
> + previous_i = nil
> + grouped_change_blocks = []
> + lines.each_with_index do |line, i|
> + deleted, added = yield(line)
> + next if deleted.nil? and added.nil?
> + if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
> + grouped_change_blocks << []
> + end
> + if i - 1 != previous_i
> + grouped_change_blocks.last << []
> + end
> + grouped_change_blocks.last.last << [deleted, added]
> + previous_i = i
> + end
> + grouped_change_blocks.collect do |change_blocks|
> + hunk = []
> + change_blocks.each do |continuous_changes|
> + continuous_changes.each do |deleted, _|
> + hunk << "-#{deleted}" if deleted
> + end
> + continuous_changes.each do |_, added|
> + hunk << "+#{added}" if added
> + end
> + end
> + hunk
> + end
> + end
> +
> + def normalize_pom_xml_output_timestamp(line)
> + line.gsub(/<project\.build\.outputTimestamp>.+?</) do
> + "<project.build.outputTimestamp>1970-01-01T00:00:00Z<"
> + end
> + end
> +
> + def normalize_added_line(line)
> + normalize_pom_xml_output_timestamp(line)
> + end
> end
>
> module VersionDetectable
> ```
Yeah, as said on a comment, I fixed the maven enforcer issue, I'm not on the merge action issue (testing locally).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1881150271
@github-actions crossbow submit *java*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1881525755
OK, I understand what happens with `ci/scripts/java_full_build.sh` script: it explicitly call `assembly:single` goal. I'm fixing that :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #39215:
URL: https://github.com/apache/arrow/pull/39215#discussion_r1445680391
##########
dev/release/test-helper.rb:
##########
@@ -83,15 +83,60 @@ def parse_patch(patch)
when /\A@@/
in_hunk = true
diffs.last[:hunks] << []
- when /\A[-+]/
+ when /\A-/
next unless in_hunk
diffs.last[:hunks].last << line.chomp
+ when /\A\+/
+ next unless in_hunk
+ diffs.last[:hunks].last << normalize_added_line(line.chomp)
end
end
diffs.sort_by do |diff|
diff[:path]
end
end
+
+ def generate_hunks(lines)
+ git_diff_context = 3 # The default of Git's diff.context
+ max_lines_for_same_hunk = git_diff_context * 2 + 1
+ previous_i = nil
+ grouped_change_blocks = []
+ lines.each_with_index do |line, i|
+ deleted, added = yield(line)
+ next if deleted.nil? and added.nil?
+ if previous_i.nil? or (i - previous_i) > max_lines_for_same_hunk
+ grouped_change_blocks << []
+ end
+ if i - 1 != previous_i
+ grouped_change_blocks.last << []
+ end
+ grouped_change_blocks.last.last << [deleted, added]
+ previous_i = i
+ end
+ grouped_change_blocks.collect do |change_blocks|
+ hunk = []
+ change_blocks.each do |continuous_changes|
+ continuous_changes.each do |deleted, _|
+ hunk << "-#{deleted}" if deleted
+ end
+ continuous_changes.each do |_, added|
+ hunk << "+#{added}" if added
+ end
+ end
+ hunk
+ end
+ end
+
+ def normalize_pom_xml_output_timestamp(line)
+ line.gsub(/<project\.build\.outputTimestamp>.+?</) do
+ "<project.build.outputTimestamp>2023-12-13T00:00:00Z<"
Review Comment:
To summarise:
1. We need `project.build.outputTimestamp` in the `pom.xml` as it's used by maven plugins to ensure the build is reproducible.
2. `project.build.outputTimestamp` value has to be fixed (not changing at each build). The value doesn't matter if it's always the same. That's the purpose of the setting the value (it's what I did in all other Apache projects where I'm working on).
3. We can't use `1970-01-01` else maven-assembly-plugin complains, that's why I used the first allowed value in the range which is `1980-01-01T00:00:02Z`. So I used the same in the `test-helper.rb` to be in sync.
Is it OK for you ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1900590304
Hmm well Archery doesn't like me but it seems CI still has a few issues even after a re-run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39214: [Java] Support reproducible build [arrow]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #39215:
URL: https://github.com/apache/arrow/pull/39215#issuecomment-1887136478
@lidavidm I'm investigating why GH Actions are not happy now (weird). Sorry for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org