You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vincbeck (via GitHub)" <gi...@apache.org> on 2023/05/17 15:42:38 UTC

[GitHub] [airflow] vincbeck opened a new pull request, #31362: Fix AWS system test example_dynamodb_to_s3

vincbeck opened a new pull request, #31362:
URL: https://github.com/apache/airflow/pull/31362

   The system test `example_dynamodb_to_s3` is currently failing for multiple reasons:
   - The point in time recovery must be enabled in the table
   - Once enabled, you need to get a valid export time. You can get such information by calling `describe_continuous_backups` API
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ferruzzi commented on a diff in pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #31362:
URL: https://github.com/apache/airflow/pull/31362#discussion_r1196790955


##########
tests/system/providers/amazon/aws/example_dynamodb_to_s3.py:
##########
@@ -56,9 +56,24 @@ def set_up_table(table_name: str):
     boto3.client("dynamodb").get_waiter("table_exists").wait(
         TableName=table_name, WaiterConfig={"Delay": 10, "MaxAttempts": 10}
     )
+    boto3.client("dynamodb").update_continuous_backups(
+        TableName=table_name,
+        PointInTimeRecoverySpecification={
+            "PointInTimeRecoveryEnabled": True,
+        },
+    )
     table.put_item(Item={"ID": "123", "Value": "Testing"})
 
 
+@task
+def get_export_time(table_name: str):
+    r = boto3.client("dynamodb").describe_continuous_backups(
+        TableName=table_name,
+    )
+
+    return r["ContinuousBackupsDescription"]["PointInTimeRecoveryDescription"]["EarliestRestorableDateTime"]

Review Comment:
   Oh, that's nicer than my created time idea.  Nice find.



##########
airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py:
##########
@@ -163,7 +168,7 @@ def _export_table_to_point_in_time(self):
 
     def _export_entire_data(self):
         """Export all data from the table."""
-        table = self.hook.get_conn().Table(self.dynamodb_table_name)
+        table = self.hook.conn.Table(self.dynamodb_table_name)

Review Comment:
   Thanks.



##########
airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py:
##########
@@ -97,9 +97,14 @@ class DynamoDBToS3Operator(AwsToAwsBaseOperator):
 
     template_fields: Sequence[str] = (
         *AwsToAwsBaseOperator.template_fields,
+        "dynamodb_table_name",
         "s3_bucket_name",
+        "file_size",
+        "dynamodb_scan_kwargs",
         "s3_key_prefix",
-        "dynamodb_table_name",
+        "process_func",
+        "export_time",
+        "export_format",

Review Comment:
   Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vincbeck commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1551827826

   All green :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vincbeck commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1552074898

   :D
   
   ![Screenshot 2023-05-17 at 4 56 57 PM](https://github.com/apache/airflow/assets/97131062/9a9d646b-5333-4fc4-ae65-e19c95873e09)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1552089826

   🎉 🎉 🎉 🎉 🎉 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1551731992

   @vincbeck Thanks for handling the timezone scenario as well. LGTM :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1551746042

   (Approved - pending the unit-test fix :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vincbeck commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1551636317

   cc @utkarsharma2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #31362:
URL: https://github.com/apache/airflow/pull/31362#discussion_r1196795581


##########
tests/system/providers/amazon/aws/example_dynamodb_to_s3.py:
##########
@@ -56,9 +56,24 @@ def set_up_table(table_name: str):
     boto3.client("dynamodb").get_waiter("table_exists").wait(
         TableName=table_name, WaiterConfig={"Delay": 10, "MaxAttempts": 10}
     )
+    boto3.client("dynamodb").update_continuous_backups(
+        TableName=table_name,
+        PointInTimeRecoverySpecification={
+            "PointInTimeRecoveryEnabled": True,
+        },
+    )
     table.put_item(Item={"ID": "123", "Value": "Testing"})
 
 
+@task
+def get_export_time(table_name: str):
+    r = boto3.client("dynamodb").describe_continuous_backups(
+        TableName=table_name,
+    )
+
+    return r["ContinuousBackupsDescription"]["PointInTimeRecoveryDescription"]["EarliestRestorableDateTime"]

Review Comment:
   This looks good! thanks for taking care of this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ferruzzi commented on pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #31362:
URL: https://github.com/apache/airflow/pull/31362#issuecomment-1551726875

   One of your unit tests are failing, bit otherwise LGTM.  Some nice nitpick catches in there as well, thanks for those.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #31362: Fix AWS system test example_dynamodb_to_s3

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #31362:
URL: https://github.com/apache/airflow/pull/31362


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org