You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sjyangkevin (via GitHub)" <gi...@apache.org> on 2025/05/18 13:27:14 UTC

[PR] Issues/42568/weaviate batch create links [airflow]

sjyangkevin opened a new pull request, #50750:
URL: https://github.com/apache/airflow/pull/50750

   ### Motivation
   Users can create links between objects through cross-references using this method.
   
   Close #42568 
   
   ### Test
   Add the following 2 test cases
   * Ensure `add_reference` is called the right number of times.
   * Ensure the retry mechanism is working
   * A DAG is created to connect to Weaviate and links existing objects in two different collections.
   
   ### Testing DAG code
   ```python
   @task()
       def batch_create_links():
           from airflow.providers.weaviate.hooks.weaviate import WeaviateHook
           from weaviate.classes.query import QueryReference
           import pandas as pd
   
           weaviate_hook = WeaviateHook()
   
           weaviate_hook.batch_create_links(
               collection_name="JeopardyQuestion",
               data=pd.DataFrame.from_dict(
                   {
                       "from_uuid": ["5b118d91-7006-444f-a028-08372f1e7356", "00d38b07-ebd7-484a-a75c-2fa37ecfb29d"],
                       "to_uuid": ["b801e25e-6c86-4ad6-a5c9-d14994375a71", "b0c80fc4-cc87-42a6-9324-f2e663d67266"],
                       "from_property": ["hasCategory", "hasCategory"]
                   }
               ),
               from_property_col="from_property",
               from_uuid_col="from_uuid",
               to_uuid_col="to_uuid"
           )
   
           collection = weaviate_hook.get_collection("JeopardyQuestion")
   
           response = collection.query.fetch_objects(
               return_references=QueryReference(
                   link_on="hasCategory",
                   return_properties=["category"]
               )
           )
           print(len(response.objects))
           print(response)
   ```
   
   ### Test Output
   ![Screenshot from 2025-05-18 09-01-48](https://github.com/user-attachments/assets/4c892ce5-0999-4722-9afd-72ae5f51d94b)
   
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add batch_create_links method in weaviate hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #50750:
URL: https://github.com/apache/airflow/pull/50750


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add batch_create_links method in weaviate hook [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #50750:
URL: https://github.com/apache/airflow/pull/50750#discussion_r2094599220


##########
providers/weaviate/src/airflow/providers/weaviate/hooks/weaviate.py:
##########
@@ -268,6 +269,61 @@ def _convert_dataframe_to_list(data: list[dict[str, Any]] | pd.DataFrame | None)
                 data = json.loads(data.to_json(orient="records"))
         return cast("list[dict[str, Any]]", data)
 
+    def batch_create_links(
+        self,
+        collection_name: str,
+        data: list[dict[str, Any]] | pd.DataFrame | None,
+        from_property_col: str = "from_property",
+        from_uuid_col: str = "from_uuid",
+        to_uuid_col: str = "to",
+        retry_attempts_per_object: int = 5,
+    ) -> list[ErrorReference] | None:
+        """
+        Batch create links from an object to another other object through cross-references (https://weaviate.io/developers/weaviate/manage-data/import#import-with-references).
+
+        :param collection_name: The name of the collection containing the source objects.
+        :param data: list or dataframe of objects we want to create links.
+        :param from_property_col: name of the reference property column.
+        :param from_uuid_col: Name of the column containing the from UUID.
+        :param to_uuid_col: Name of the column containing the target UUID.
+        :param retry_attempts_per_object: number of time to try in case of failure before giving up.
+        """
+        converted_data = self._convert_dataframe_to_list(data)
+        collection = self.get_collection(collection_name)
+
+        with collection.batch.dynamic() as batch:
+            # Batch create links
+            for data_obj in converted_data:
+                for attempt in Retrying(
+                    stop=stop_after_attempt(retry_attempts_per_object),
+                    retry=(
+                        retry_if_exception(lambda exc: check_http_error_is_retryable(exc))
+                        | retry_if_exception_type(REQUESTS_EXCEPTIONS_TYPES)
+                    ),
+                ):
+                    with attempt:
+                        from_property = data_obj.pop(from_property_col, None)
+                        from_uuid = data_obj.pop(from_uuid_col, None)
+                        to_uuid = data_obj.pop(to_uuid_col, None)
+                        self.log.debug(
+                            "Attempt %s of create links between %s and %s using reference property %s",
+                            attempt.retry_state.attempt_number,
+                            from_uuid,
+                            to_uuid,
+                            from_property,
+                        )
+                        batch.add_reference(
+                            from_property=from_property,
+                            from_uuid=from_uuid,
+                            to=to_uuid,
+                        )
+
+        failed_references = collection.batch.failed_references
+        if failed_references:
+            print(f"Number of failed imports: {len(failed_references)}")

Review Comment:
   Don't we want to raise an exception in this case?
   If not, I think that logging fits better in this case:
   
   ```suggestion
               self.log.error("Number of failed imports: %s", len(failed_references))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add batch_create_links method in weaviate hook [airflow]

Posted by "sjyangkevin (via GitHub)" <gi...@apache.org>.
sjyangkevin commented on code in PR #50750:
URL: https://github.com/apache/airflow/pull/50750#discussion_r2094602774


##########
providers/weaviate/src/airflow/providers/weaviate/hooks/weaviate.py:
##########
@@ -268,6 +269,61 @@ def _convert_dataframe_to_list(data: list[dict[str, Any]] | pd.DataFrame | None)
                 data = json.loads(data.to_json(orient="records"))
         return cast("list[dict[str, Any]]", data)
 
+    def batch_create_links(
+        self,
+        collection_name: str,
+        data: list[dict[str, Any]] | pd.DataFrame | None,
+        from_property_col: str = "from_property",
+        from_uuid_col: str = "from_uuid",
+        to_uuid_col: str = "to",
+        retry_attempts_per_object: int = 5,
+    ) -> list[ErrorReference] | None:
+        """
+        Batch create links from an object to another other object through cross-references (https://weaviate.io/developers/weaviate/manage-data/import#import-with-references).
+
+        :param collection_name: The name of the collection containing the source objects.
+        :param data: list or dataframe of objects we want to create links.
+        :param from_property_col: name of the reference property column.
+        :param from_uuid_col: Name of the column containing the from UUID.
+        :param to_uuid_col: Name of the column containing the target UUID.
+        :param retry_attempts_per_object: number of time to try in case of failure before giving up.
+        """
+        converted_data = self._convert_dataframe_to_list(data)
+        collection = self.get_collection(collection_name)
+
+        with collection.batch.dynamic() as batch:
+            # Batch create links
+            for data_obj in converted_data:
+                for attempt in Retrying(
+                    stop=stop_after_attempt(retry_attempts_per_object),
+                    retry=(
+                        retry_if_exception(lambda exc: check_http_error_is_retryable(exc))
+                        | retry_if_exception_type(REQUESTS_EXCEPTIONS_TYPES)
+                    ),
+                ):
+                    with attempt:
+                        from_property = data_obj.pop(from_property_col, None)
+                        from_uuid = data_obj.pop(from_uuid_col, None)
+                        to_uuid = data_obj.pop(to_uuid_col, None)
+                        self.log.debug(
+                            "Attempt %s of create links between %s and %s using reference property %s",
+                            attempt.retry_state.attempt_number,
+                            from_uuid,
+                            to_uuid,
+                            from_property,
+                        )
+                        batch.add_reference(
+                            from_property=from_property,
+                            from_uuid=from_uuid,
+                            to=to_uuid,
+                        )
+
+        failed_references = collection.batch.failed_references
+        if failed_references:
+            print(f"Number of failed imports: {len(failed_references)}")

Review Comment:
   Thank you for pointing out. I agree that logging fits better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org