Re: [I] Airflow 2 to 3 Rest API Changes [airflow]

["sunank200 (via GitHub)" <gi...@apache.org>]

sunank200 commented on issue #43378:
URL: https://github.com/apache/airflow/issues/43378#issuecomment-2787798694

   I’ve written a Python script to automatically compare the OpenAPI specifications for Airflow 2.10.5 (v1) and Airflow 3.0.0 (v2) to identify breaking changes. In our context, a **breaking change** includes:
   
   - Removal of an entire endpoint or an HTTP method.
   - Changes in endpoint definitions that modify required parameters, request bodies, or response schemas.
   
   ### How to Run the Script
   
   1. **Install dependencies:**
   
      ```bash
      pip install pyyaml requests deepdiff
      ```
   
   2. **Use the raw URL for the v1 spec:**
   
      - v1: `https://raw.githubusercontent.com/apache/airflow/2.10.5/airflow/api_connexion/openapi/v1.yaml`
      - v2 (local): `http://localhost:28080/openapi.json`
   
   3. **Run the script with:**
   
      ```bash
      python compare_openapi.py https://raw.githubusercontent.com/apache/airflow/2.10.5/airflow/api_connexion/openapi/v1.yaml http://localhost:28080/openapi.json
      ```
   
      The script will print the detected breaking changes and write a report to `report.txt`.
   
   ### Final Script
   
   ```python
   import sys
   import json
   import yaml
   import requests
   from deepdiff import DeepDiff
   import pprint
   
   def load_spec_from_file(filepath):
       with open(filepath, "r") as f:
           if filepath.endswith(".json"):
               return json.load(f)
           else:
               return yaml.safe_load(f)
   
   def load_spec_from_url(url):
       response = requests.get(url)
       response.raise_for_status()
       if url.endswith(".json"):
           return response.json()
       else:
           return yaml.safe_load(response.text)
   
   def normalize_spec(spec, version_prefixes=("/api/v1", "/api/v2")):
       new_paths = {}
       for path, methods in spec.get("paths", {}).items():
           for prefix in version_prefixes:
               if path.startswith(prefix):
                   path = path[len(prefix):]
           new_paths[path or "/"] = methods
       spec["paths"] = new_paths
       spec.pop("servers", None)
       return spec
   
   def compare_specs(spec1, spec2):
       breaking_changes = []
       notes = []
       pp = pprint.PrettyPrinter(indent=2)
       paths1 = spec1.get("paths", {})
       paths2 = spec2.get("paths", {})
       for path in sorted(paths1.keys()):
           if path not in paths2:
               breaking_changes.append(f"Endpoint '{path}' removed in new spec")
           else:
               for method in sorted(paths1[path].keys()):
                   if method not in paths2[path]:
                       breaking_changes.append(f"Method '{method.upper()}' for endpoint '{path}' removed in new spec")
                   else:
                       diff = DeepDiff(paths1[path][method], paths2[path][method], ignore_order=True)
                       if diff:
                           formatted_diff = pp.pformat(diff)
                           breaking_changes.append(f"Method '{method.upper()}' for endpoint '{path}' changed:\n{formatted_diff}")
       for path in sorted(paths2.keys()):
           if path not in paths1:
               notes.append(f"New endpoint '{path}' added in new spec")
           else:
               for method in sorted(paths2[path].keys()):
                   if method not in paths1[path]:
                       notes.append(f"New method '{method.upper()}' for endpoint '{path}' added in new spec")
       return breaking_changes, notes
   
   if __name__ == "__main__":
       if len(sys.argv) < 3:
           print("Usage: python compare_openapi.py <spec1_path_or_url> <spec2_path_or_url>")
           sys.exit(1)
       spec1_source = sys.argv[1]
       spec2_source = sys.argv[2]
       try:
           if spec1_source.startswith("http"):
               spec1 = load_spec_from_url(spec1_source)
           else:
               spec1 = load_spec_from_file(spec1_source)
       except Exception as e:
           print(f"Failed to load spec from {spec1_source}: {e}")
           sys.exit(1)
       try:
           if spec2_source.startswith("http"):
               spec2 = load_spec_from_url(spec2_source)
           else:
               spec2 = load_spec_from_file(spec2_source)
       except Exception as e:
           print(f"Failed to load spec from {spec2_source}: {e}")
           sys.exit(1)
       spec1 = normalize_spec(spec1)
       spec2 = normalize_spec(spec2)
       breaking, notes = compare_specs(spec1, spec2)
       report_lines = []
       report_lines.append("API Comparison Report:")
       report_lines.append("======================\n")
       if notes:
           report_lines.append("Additional Notes (Non-Breaking Changes):")
           for note in notes:
               report_lines.append(" - " + note)
           report_lines.append("")
       if breaking:
           report_lines.append("Breaking Changes Detected:")
           for change in breaking:
               report_lines.append("\n - " + change)
       else:
           report_lines.append("No breaking changes detected.")
       report_content = "\n".join(report_lines)
       print(report_content)
       output_filename = "report.txt"
       try:
           with open(output_filename, "w") as f:
               f.write(report_content)
           print(f"\nReport written to {output_filename}")
       except Exception as e:
           print(f"Failed to write report to {output_filename}: {e}")
   ```
   
   Following is the report.txt I got from the script: 
   
   [report.txt](https://github.com/user-attachments/files/19657773/report.txt)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org