Importer framework

[Cory Johns <cj...@slashdotmedia.com>]

We're beginning to focus on improving and expanding our support for
importing and exporting project data to and from Allura, and as part of
that we need to develop an extensible framework for adding new import
mechanisms to Allura and exposing them in the UI.

New importers will need to be able to advertise both what tool they are
able to import data for as well as what external source they will import
their data from.  The discovery mechanism will need to be able to list the
importers available by tool type, so that a given tool can provide a UI to
import from any supported source, or by source, so that we can provide an
integrated mechanism to import an entire project and all supported data
from a source.

Entry points are an obvious choice for discovering importers, whether they
are provided by the tool itself or a third-party library or Application to
extend the tool's ability to import.  My thought is to have tools or
libraries add entry points to the "allura.importers" group, with each entry
point class providing import capability for a single tool from a single
source.  Each importer would then have attributes indicating the tool it
applies to, the source it imports from, and the controller for its UI.

Here is an example importer stub with the attributes I'm thinking of:

    class GoogleTrackerImporter(class):
        target_app = forgetracker.tracker_main.ForgeTrackerApp
        source = 'Google'
        controller = forgeimporters.google.TrackerImportController

To support full-project imports, a library could expose an entry point in
the "allura.project_importers" group, which would point to a controller
that coordinates importing an entire project from an external source.

I believe that will allow us to tie together importers in an extensible
way, yet give each importer flexibility in how it implements its import
logic.

What do you think?


- Cory

Re: Importer framework

[Cory Johns <cj...@slashdotmedia.com>]

The main use for the source, other than display purposes, is to group
importers for something like a full-project import.  In that case, the
full-project import will likely have all or most of the importers in a
single library, establishing a "canonical" value for that source.  The case
of importers growing organically with a full-project import being created
post-hoc seems less likely, though certainly possible, and anyway, the
full-project import could use the most common value if there's disagreement
and just require the differing ones to update to be included.  Or create
wrappers or something.  I assumed the values to be used for display, but
maybe we should have a separate title property and recommend that the
source property match the entry-point name for the full-project importer.

As for the status, is expecting a get_status(project) method on the
importer sufficient?  As for the return value, I don't expect that
partial-progress status information will be easy to get, but we could
define the get_status() return value to be a 0.0 to 1.0 numeric value to
support it, with 0.0 being not started, 0.5 being in-progress, and 1.0
being complete, and any further resolution in between a (rough) percentage.


On Thu, Jul 11, 2013 at 6:26 PM, Dave Brondsema <
dbrondsema@slashdotmedia.com> wrote:

> Sounds pretty good to me.  It'll also enable us to more easily move the
> MediaWiki importer into a separate package.  It depends on some GPL code so
> it's already optional, but would be cleaner to have it completely separate.
>  Right now it's just a paster script, I think.
>
> Do we want to have anything more formal than a string for the source?  As
> long as all the importers follow the same conventions (Google? GoogleCode?)
> it'll probably be fine.
>
> The backend flexibility for each importer will be good.  It might be nice
> to have some lightweight integration points there too, as we build more
> features into them.  For example, we'll probably want to show the status of
> any import.  If we have a generic way to do reference that (e.g. a task id,
> or something fancier for incremental progress within a task), then we could
> show the status for any type of import.  That might actually be even more
> generally useful than imports -- I'm thinking it'd be nice for the data
> export/backup tasks too.  Anyway, it'll depend on how implementation goes
> and how the details get worked out.  Can't say yet exactly what it'd look
> like.
>
>
> On Thu, Jul 11, 2013 at 2:43 PM, Cory Johns <cjohns@slashdotmedia.com
> >wrote:
>
> > We're beginning to focus on improving and expanding our support for
> > importing and exporting project data to and from Allura, and as part of
> > that we need to develop an extensible framework for adding new import
> > mechanisms to Allura and exposing them in the UI.
> >
> > New importers will need to be able to advertise both what tool they are
> > able to import data for as well as what external source they will import
> > their data from.  The discovery mechanism will need to be able to list
> the
> > importers available by tool type, so that a given tool can provide a UI
> to
> > import from any supported source, or by source, so that we can provide an
> > integrated mechanism to import an entire project and all supported data
> > from a source.
> >
> > Entry points are an obvious choice for discovering importers, whether
> they
> > are provided by the tool itself or a third-party library or Application
> to
> > extend the tool's ability to import.  My thought is to have tools or
> > libraries add entry points to the "allura.importers" group, with each
> entry
> > point class providing import capability for a single tool from a single
> > source.  Each importer would then have attributes indicating the tool it
> > applies to, the source it imports from, and the controller for its UI.
> >
> > Here is an example importer stub with the attributes I'm thinking of:
> >
> >     class GoogleTrackerImporter(class):
> >         target_app = forgetracker.tracker_main.ForgeTrackerApp
> >         source = 'Google'
> >         controller = forgeimporters.google.TrackerImportController
> >
> > To support full-project imports, a library could expose an entry point in
> > the "allura.project_importers" group, which would point to a controller
> > that coordinates importing an entire project from an external source.
> >
> > I believe that will allow us to tie together importers in an extensible
> > way, yet give each importer flexibility in how it implements its import
> > logic.
> >
> > What do you think?
> >
> >
> > - Cory
> >
>
>
>
> --
> Dave Brondsema
> Principal Software Engineer - sourceforge.net
> Dice Holdings, Inc.
>

Re: Importer framework

[Dave Brondsema <db...@slashdotmedia.com>]

Sounds pretty good to me.  It'll also enable us to more easily move the
MediaWiki importer into a separate package.  It depends on some GPL code so
it's already optional, but would be cleaner to have it completely separate.
 Right now it's just a paster script, I think.

Do we want to have anything more formal than a string for the source?  As
long as all the importers follow the same conventions (Google? GoogleCode?)
it'll probably be fine.

The backend flexibility for each importer will be good.  It might be nice
to have some lightweight integration points there too, as we build more
features into them.  For example, we'll probably want to show the status of
any import.  If we have a generic way to do reference that (e.g. a task id,
or something fancier for incremental progress within a task), then we could
show the status for any type of import.  That might actually be even more
generally useful than imports -- I'm thinking it'd be nice for the data
export/backup tasks too.  Anyway, it'll depend on how implementation goes
and how the details get worked out.  Can't say yet exactly what it'd look
like.


On Thu, Jul 11, 2013 at 2:43 PM, Cory Johns <cj...@slashdotmedia.com>wrote:

> We're beginning to focus on improving and expanding our support for
> importing and exporting project data to and from Allura, and as part of
> that we need to develop an extensible framework for adding new import
> mechanisms to Allura and exposing them in the UI.
>
> New importers will need to be able to advertise both what tool they are
> able to import data for as well as what external source they will import
> their data from.  The discovery mechanism will need to be able to list the
> importers available by tool type, so that a given tool can provide a UI to
> import from any supported source, or by source, so that we can provide an
> integrated mechanism to import an entire project and all supported data
> from a source.
>
> Entry points are an obvious choice for discovering importers, whether they
> are provided by the tool itself or a third-party library or Application to
> extend the tool's ability to import.  My thought is to have tools or
> libraries add entry points to the "allura.importers" group, with each entry
> point class providing import capability for a single tool from a single
> source.  Each importer would then have attributes indicating the tool it
> applies to, the source it imports from, and the controller for its UI.
>
> Here is an example importer stub with the attributes I'm thinking of:
>
>     class GoogleTrackerImporter(class):
>         target_app = forgetracker.tracker_main.ForgeTrackerApp
>         source = 'Google'
>         controller = forgeimporters.google.TrackerImportController
>
> To support full-project imports, a library could expose an entry point in
> the "allura.project_importers" group, which would point to a controller
> that coordinates importing an entire project from an external source.
>
> I believe that will allow us to tie together importers in an extensible
> way, yet give each importer flexibility in how it implements its import
> logic.
>
> What do you think?
>
>
> - Cory
>



-- 
Dave Brondsema
Principal Software Engineer - sourceforge.net
Dice Holdings, Inc.