You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Gwen Shapira (JIRA)" <ji...@apache.org> on 2014/12/22 22:01:13 UTC
[jira] [Commented] (SQOOP-1804) Respository Structure + API:
Storing/Retrieving the From/To state of the incremental read/ write
[ https://issues.apache.org/jira/browse/SQOOP-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256187#comment-14256187 ]
Gwen Shapira commented on SQOOP-1804:
-------------------------------------
Looks good.
Two questions for consideration:
1. Do we want to have a separate table for keys? This will allow us to have an API to list all available outputs (for GUI mostly), without expensive "select distinct" on what can be a largish table.
2. Do we need to separate outputs and counters? Perhaps they are both metadata and we can roll counters into the new table? Ryan Blue may have an opinion here, since we are touching on Metadata management.
In my experience, customers will be interested in information like:
- Last value
- How was the data split between mappers (#rows per mapper)
- How many rows total
- Min / Max / Avg of specific columns in the data
- How long it took to Sqoop (end to end)
- Number of errors
Are these metrics? counters? outputs? is there a need to store them separately?
Interested in your thoughts here.
> Respository Structure + API: Storing/Retrieving the From/To state of the incremental read/ write
> ------------------------------------------------------------------------------------------------
>
> Key: SQOOP-1804
> URL: https://issues.apache.org/jira/browse/SQOOP-1804
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> Details of this proposal are in the wiki.
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Wheretostoretheoutputinsqoop?
> Will use this ticket for more detailed discussion on storage options for the output from connectors
> 1.
> {code}
> // will have FK to submission
> public static final String QUERY_CREATE_TABLE_SQ_JOB_OUTPUT_SUBMISSION =
> "CREATE TABLE " + TABLE_SQ_JOB_OUTPUT + " ("
> + COLUMN_SQ_JOB_OUT_ID + " BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1), "
> + COLUMN_SQ_JOB_OUT_KEY + " VARCHAR(32), "
> + COLUMN_SQ_JOB_OUT_VALUE + " LONG VARCHAR,"
> + COLUMN_SQ_JOB_OUT_TYPE + " VARCHAR(32),"
> + COLUMN_SQD_ID + " VARCHAR(32)," // FK to the direction table, since this allows to distinguish output from FROM/ TO part of the job
> + COLUMN_SQRS_SUBMISSION + " BIGINT, "
> + "CONSTRAINT " + CONSTRAINT_SQRS_SQS + " "
> + "FOREIGN KEY (" + COLUMN_SQRS_SUBMISSION + ") "
> + "REFERENCES " + TABLE_SQ_SUBMISSION + "(" + COLUMN_SQS_ID + ") ON DELETE CASCADE "
> {code}
> 2.
> At the code level, we will define MOutputType, one of the types can be BLOB as well, if a connector decides to store the value as a BLOB
> {code}
> class JobOutput {
> String key;
> Object value;
> MOutputType type;
> }
> {code}
> 3.
> At the repository API, add a new API to get job output for a particular submission Id and allow updates on values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)