You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2023/01/05 07:29:45 UTC

[GitHub] [spark] dongjoon-hyun opened a new pull request, #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

dongjoon-hyun opened a new pull request, #39401:
URL: https://github.com/apache/spark/pull/39401

   ### What changes were proposed in this pull request?
   
   This PR aims to publish `SBOM` artifacts.
   
   ### Why are the changes needed?
   
   Here is an article to give some context.
   - https://www.activestate.com/blog/why-the-us-government-is-mandating-software-bill-of-materials-sbom/
   
   Software Bill of Materials (SBOM) are additional artifacts containing the aggregate of all direct and transitive dependencies of a project. The US Government (based on NIST recommendations) currently accepts only the three most popular SBOM standards as valid, namely: [CycloneDX](https://cyclonedx.org/), [Software Identification (SWID) tag](https://csrc.nist.gov/projects/Software-Identification-SWID), [Software Package Data Exchange® (SPDX)](https://spdx.dev/).
   
   This PR uses [CycloneDX maven plugin](https://github.com/CycloneDX/cyclonedx-maven-plugin), a lightweight software bill of materials (SBOM) standard designed for use in application security contexts and supply chain component analysis.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, but dev-only changes.
   
   ### How was this patch tested?
   
   Manually test.
   ```
   $ mvn install -DskipTests
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372967599

   Thank you all. Let me merge this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372770976

   Yes, right. Thank you, @viirya .
   > So it basically generates aggregate of dependencies as xml and json files and attaches into jar files, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1373006207

   We can see the published SBOM tomorrow after tomorrow's snapshot publishing.
   - https://github.com/apache/spark/actions/workflows/publish_snapshot.yml


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts
URL: https://github.com/apache/spark/pull/39401


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1374367656

   Thanks @dongjoon-hyun !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372460399

   Thank you, @srowen . 
   1. It goes to Maven Central. I did this at the Apache ORC project first via ORC-1342. Here is the published SBOM on the snapshot versions after merging this.
      - https://github.com/apache/orc/pull/1353
   
   ```
   **1.9.0**
   - https://repository.apache.org/content/repositories/snapshots/org/apache/orc/orc-core/1.9.0-SNAPSHOT/orc-core-1.9.0-20230105.074036-185-cyclonedx.xml
   - https://repository.apache.org/content/repositories/snapshots/org/apache/orc/orc-core/1.9.0-SNAPSHOT/orc-core-1.9.0-20230105.074036-185-cyclonedx.json
   
   **1.8.2**
   - https://repository.apache.org/content/repositories/snapshots/org/apache/orc/orc-core/1.8.2-SNAPSHOT/orc-core-1.8.2-20230105.074040-16-cyclonedx.xml
   - https://repository.apache.org/content/repositories/snapshots/org/apache/orc/orc-core/1.8.2-SNAPSHOT/orc-core-1.8.2-20230105.074040-16-cyclonedx.json
   
   **1.7.8**
   - https://repository.apache.org/content/repositories/snapshots/org/apache/orc/orc-core/1.7.8-SNAPSHOT/orc-core-1.7.8-20230105.074050-2-cyclonedx.xml
   - https://repository.apache.org/content/repositories/snapshots/org/apache/orc/orc-core/1.7.8-SNAPSHOT/orc-core-1.7.8-20230105.074050-2-cyclonedx.json
   ```
   
   2. While `deps` files are only for Spark binary distributions, this is a jar-level manifest. For example, `spark-core_2.12-3.4.0-SNAPSHOT-cyclonedx.xml` shows its dependency only without Kubernetes dependencies.
   
   ```
   $ cat ~/.m2/repository/org/apache/spark/spark-core_2.12/3.4.0-SNAPSHOT/spark-core_2.12-3.4.0-SNAPSHOT-cyclonedx.json  | jq .components | grep \"name\" | head -n5
       "name": "avro",
       "name": "jackson-core",
       "name": "commons-compress",
       "name": "avro-mapred",
       "name": "avro-ipc",
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1371876411

   cc @srowen and @HyukjinKwon 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
viirya commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372760984

   Looks good but maybe wait for a while for others to chime in if they have some opinions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372018391

   The PR is ready for review now. Could you review when you have some time?
   @srowen , @HyukjinKwon , @cloud-fan , @viirya , @sunchao , @huaxingao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1371884646

   Ah, it seems that I missed some failures. I convert this as `Draft`. Let me dig this.
   ```
   [WARNING] An unexpected issue occurred attempting to resolve the effective pom for  org.xerial.snappy:snappy-java:1.1.8.4
   org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
   [ERROR] Unknown packaging: bundle @ line 6, column 16
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
srowen commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372218017

   Seems fine to me. I'm not sure if the maven release plugin will also push this to Maven Central, but maybe that's not essential. Do the files look plausible, like they appear to contain the transitive dependencies and more or less match what's in the "deps" files in the repo?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1372928635

   Thank you, @sunchao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39401: [SPARK-41893][BUILD] Publish SBOM artifacts

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39401:
URL: https://github.com/apache/spark/pull/39401#issuecomment-1374366502

   Here is the published SBOM
   - https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-parent_2.12/3.4.0-SNAPSHOT/spark-parent_2.12-3.4.0-20230107.001126-296-cyclonedx.xml


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org