You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/29 16:18:18 UTC
[GitHub] [airflow] bbovenzi opened a new pull request, #25403: Show upstream/downstreams from a dataset
bbovenzi opened a new pull request, #25403:
URL: https://github.com/apache/airflow/pull/25403
Datasets list: show a count of upstreams and downstreams for a dataset
Dataset details: show a full list of upstreams and downstreams for the dataset, and each item links to the DAG's grid view
<img width="741" alt="Screen Shot 2022-07-29 at 5 11 33 PM" src="https://user-images.githubusercontent.com/4600967/181801438-e344fa24-2c66-43fd-8a6d-6aecbfc913ee.png">
<img width="1368" alt="Screen Shot 2022-07-29 at 5 11 46 PM" src="https://user-images.githubusercontent.com/4600967/181801447-62a08e2f-9008-4103-95f0-ba930a73038e.png">
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #25403: Show upstream/downstreams from a dataset
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #25403:
URL: https://github.com/apache/airflow/pull/25403#issuecomment-1202291405
Thought/idea: Rather than "upstream" and "downstream" how about we use "producing" and "consuming"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi merged pull request #25403: Show upstream/downstreams from a dataset
Posted by GitBox <gi...@apache.org>.
bbovenzi merged PR #25403:
URL: https://github.com/apache/airflow/pull/25403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25403: Show upstream/downstreams from a dataset
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25403:
URL: https://github.com/apache/airflow/pull/25403#issuecomment-1202368058
> Thought/idea: Rather than "upstream" and "downstream" how about we use "producing" and "consuming"?
I very much like this proposal. This is so much more "real-live" than downstream/upstream. Maybe it's the non-native English kicking-in, but upstream/downstream has no 'natural" mental model for me when it comes to data flows (counterintuitvely you could say).
I have to always think twice what it really means, Producer/Consumer is very straightforward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on a diff in pull request #25403: Show upstream/downstreams from a dataset
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on code in PR #25403:
URL: https://github.com/apache/airflow/pull/25403#discussion_r934292029
##########
airflow/www/static/js/datasets/Details.tsx:
##########
@@ -30,12 +30,74 @@ import {
Table, TimeCell, CodeCell, TaskInstanceLink,
} from 'src/components/Table';
import { ClipboardButton } from 'src/components/Clipboard';
+import type { API } from 'src/types';
+import InfoTooltip from 'src/components/InfoTooltip';
interface Props {
datasetId: string;
onBack: () => void;
}
+const Details = ({
+ dataset: {
+ createdAt,
+ updatedAt,
+ uri,
+ extra,
+ upstreamTaskReferences,
+ downstreamDagReferences,
+ },
+}: { dataset: API.Dataset }) => (
+ <Box>
+ <Heading my={2} fontWeight="normal">
+ Dataset:
+ {' '}
+ {uri}
+ <ClipboardButton value={uri} iconOnly ml={2} />
+ </Heading>
+ {!!extra && (
+ <Flex>
+ <Text mr={1}>Extra:</Text>
+ <Code>{JSON.stringify(extra)}</Code>
+ </Flex>
+ )}
+ <Flex my={2}>
+ <Text mr={1}>Updated At:</Text>
+ <Time dateTime={updatedAt} />
+ </Flex>
+ <Flex my={2}>
+ <Text mr={1}>Created At:</Text>
+ <Time dateTime={createdAt} />
+ </Flex>
Review Comment:
Oh that isn't the case, its just when the db entry was updated? Ok. I'll cut that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25403: Show upstream/downstreams from a dataset
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25403:
URL: https://github.com/apache/airflow/pull/25403#discussion_r933470703
##########
airflow/www/static/js/datasets/Details.tsx:
##########
@@ -30,12 +30,74 @@ import {
Table, TimeCell, CodeCell, TaskInstanceLink,
} from 'src/components/Table';
import { ClipboardButton } from 'src/components/Clipboard';
+import type { API } from 'src/types';
+import InfoTooltip from 'src/components/InfoTooltip';
interface Props {
datasetId: string;
onBack: () => void;
}
+const Details = ({
+ dataset: {
+ createdAt,
+ updatedAt,
+ uri,
+ extra,
+ upstreamTaskReferences,
+ downstreamDagReferences,
+ },
+}: { dataset: API.Dataset }) => (
+ <Box>
+ <Heading my={2} fontWeight="normal">
+ Dataset:
+ {' '}
+ {uri}
+ <ClipboardButton value={uri} iconOnly ml={2} />
+ </Heading>
+ {!!extra && (
+ <Flex>
+ <Text mr={1}>Extra:</Text>
+ <Code>{JSON.stringify(extra)}</Code>
+ </Flex>
+ )}
+ <Flex my={2}>
+ <Text mr={1}>Updated At:</Text>
+ <Time dateTime={updatedAt} />
+ </Flex>
+ <Flex my={2}>
+ <Text mr={1}>Created At:</Text>
+ <Time dateTime={createdAt} />
+ </Flex>
+ {upstreamTaskReferences && !!upstreamTaskReferences.length && (
+ <Box mb={2}>
+ <Flex alignItems="center">
+ <Heading size="md" fontWeight="normal">Upstream Tasks</Heading>
+ <InfoTooltip label="Tasks that will update this dataset." size={14} />
+ </Flex>
+ {upstreamTaskReferences.map(({ dagId, taskId }) => (
+ <Link key={`${dagId}.${taskId}`} color="blue.600" href={`/dags/${dagId}/grid`}>
+ {`${dagId}.${taskId}`}
+ </Link>
+ ))}
+ </Box>
+ )}
+ {downstreamDagReferences && !!downstreamDagReferences.length && (
+ <Box>
+ <Flex alignItems="center">
+ <Heading size="md" fontWeight="normal">Downstream DAGs</Heading>
Review Comment:
i think you might need some `<br />` tags somewhere ... they are all on same line
<img width="888" alt="image" src="https://user-images.githubusercontent.com/15932138/181810333-311d843c-c0bd-4a6a-9725-326b76c1f0a2.png">
##########
airflow/www/static/js/datasets/Details.tsx:
##########
@@ -85,32 +147,11 @@ const DatasetDetails = ({ datasetId, onBack }: Props) => {
<Box mt={[6, 3]} maxWidth="1500px">
<Button onClick={onBack}>See all datasets</Button>
{isLoading && <Spinner display="block" />}
- {!!dataset && (
- <Box>
- <Heading my={2} fontWeight="normal">
- Dataset:
- {' '}
- {dataset.uri}
- <ClipboardButton value={dataset.uri} iconOnly ml={2} />
- </Heading>
- {!!dataset.extra && (
- <Flex>
- <Text mr={1}>Extra:</Text>
- <Code>{JSON.stringify(dataset.extra)}</Code>
- </Flex>
- )}
- <Flex my={2}>
- <Text mr={1}>Updated At:</Text>
- <Time dateTime={dataset.updatedAt} />
- </Flex>
- <Flex my={2}>
- <Text mr={1}>Created At:</Text>
- <Time dateTime={dataset.createdAt} />
- </Flex>
- </Box>
- )}
- <Heading size="lg" mt={3} mb={2} fontWeight="normal">Upstream Events</Heading>
- <Text>Whenever a DAG has updated this dataset.</Text>
+ {!!dataset && (<Details dataset={dataset} />)}
+ <Flex alignItems="center">
+ <Heading size="lg" mt={3} mb={2} fontWeight="normal">Upstream Events</Heading>
Review Comment:
```suggestion
<Heading size="lg" mt={3} mb={2} fontWeight="normal">Events</Heading>
```
they are just events, in this context.
##########
airflow/www/static/js/datasets/Details.tsx:
##########
@@ -30,12 +30,74 @@ import {
Table, TimeCell, CodeCell, TaskInstanceLink,
} from 'src/components/Table';
import { ClipboardButton } from 'src/components/Clipboard';
+import type { API } from 'src/types';
+import InfoTooltip from 'src/components/InfoTooltip';
interface Props {
datasetId: string;
onBack: () => void;
}
+const Details = ({
+ dataset: {
+ createdAt,
+ updatedAt,
+ uri,
+ extra,
+ upstreamTaskReferences,
+ downstreamDagReferences,
+ },
+}: { dataset: API.Dataset }) => (
+ <Box>
+ <Heading my={2} fontWeight="normal">
+ Dataset:
+ {' '}
+ {uri}
+ <ClipboardButton value={uri} iconOnly ml={2} />
+ </Heading>
+ {!!extra && (
+ <Flex>
+ <Text mr={1}>Extra:</Text>
+ <Code>{JSON.stringify(extra)}</Code>
+ </Flex>
+ )}
+ <Flex my={2}>
+ <Text mr={1}>Updated At:</Text>
+ <Time dateTime={updatedAt} />
+ </Flex>
+ <Flex my={2}>
+ <Text mr={1}>Created At:</Text>
+ <Time dateTime={createdAt} />
+ </Flex>
Review Comment:
```suggestion
```
In this context, adding updated and created at are maybe a bit confusing. Cus "updated at" could indicated "the dataset was _updated_" i.e. in an event.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org