You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bloodhound.apache.org by Apache Bloodhound <bl...@incubator.apache.org> on 2012/12/06 09:31:34 UTC
[Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
#288: Create prototype for legacy database schema proxy
----------------------------------+--------------------
Reporter: jure | Owner: nobody
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Keywords: bep-003 multiproduct |
----------------------------------+--------------------
As discussed on the dev mailing list and documented in
[https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0003#database-
schema-changes], we want to make product a first class citizen in
Bloodhound. This would be (as proposed) be implemented by introducing new
column (product) to tables that hold per-product resources (milestones,
components, versions,...). As this would break existing trac/plugin
compatibility, the idea is to produce a database proxy component
(prototype for starters) that would 'install' itself directly above the
database, parse the SQL statements and modify them in such a way that
would limit the scope (of the SQL statements) to the product in current
scope. Using this approach (if it turns out to be feasible at all) we
would keep compatibility with both trac and plugins and still be able to
change the database schema as proposed and at the same time allow
multiproduct aware code/plugins to leverage the new database schema.
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
I'd like to propose some features related to #288 for you to consider .
These are aimed at succeeding with tests cases for permissions , and
ticket API . I've ben working the last few days towards that goal . The
proposal is presented in the form of two patches .
[attachment:t288_r1438538_sql_translate_global_env.diff The first one]
incorporates the following features :
1. Add a context manager for switching product environments
with the goal in mind of better tracking of active environment
when using nested transactions . JFTR notice that «active
environment»
is stored in TLS .
2. Add two methods `db_direct_query` and `db_direct_transaction`
providing direct DB access (i.e. no product prefix translation).
3. Default product prefix will be '' rather than 'default' and will
be reserved for global environment (i.e. `trac.env.Environment`).
4. `ProductEnvironment(env, None) is env` and
`ProductEnvironment(env, '') is env` in order to be consistent with
(3) above.
Up to this point patch order is as follows ...
{{{
#!sh
$ hg qapplied
t288/t288_r1438538_sql_translate_global_env.diff
t355/t355_r1437383_trac_test_perm.diff
t355/t355_r1437383_trac_test_ticket_api.diff
}}}
... and everything is still looking ok and all tests (except those in
tests.perm and tests.ticket.api) succeed . The next logical step forward
is to implement the actual translation for the global environment with the
help of (1) and (2) above.
In the mechanism mentioned in (2) both , aforementioned context manager
and TLS variable participate following the following sequence
* context manager sets env in TLS
* iterable cursor checks env var in TLS .
- if set to product env , translate SQL considering product = prefix
- if set to global env , translate SQL considering product = ''
- if set to `None` (e.g. after invoking `db_direct_query`
and `db_direct_transaction` ) then do not translate SQL
query.
... at least that's the idea I tried to sketch in
[attachment:t288_r1438538_sql_translate_global_env_pending.diff the second
patch] , but up to that point tests won't work anymore :'(
When patches applied this way
{{{
#!sh
$ hg qapplied
t288/t288_r1438538_sql_translate_global_env.diff
t288/t288_r1438538_sql_translate_global_env_pending.diff
t355/t355_r1437383_trac_test_perm.diff
t355/t355_r1437383_trac_test_ticket_api.diff
}}}
... this is what I get :
{{{
#!sh
$ python setup.py test -m tests.env
[...]
Testing env.get_known_users ... ERROR
Testing env.get_version ... ERROR
Testing env.__getattr__ ... ok
Testing env.is_component_enabled ... ok
Testing env.path ... ok
Testing env.__init__ ... ok
======================================================================
ERROR: Testing env.get_known_users
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/bloodhound/bloodhound_multiproduct/tests/env.py", line
182, in setUp
EnvironmentTestCase.setUp(self)
File "/path/to/bloodhound/trac/trac/tests/env.py", line 16, in setUp
self.env = Environment(env_path, create=True)
File "/path/to/bloodhound/trac/trac/core.py", line 140, in __call__
self.__init__(*args, **kwargs)
File "/path/to/bloodhound/trac/trac/core.py", line 107, in new_init
original_init(self, *args, **kwargs)
File "/path/to/bloodhound/bloodhound_multiproduct/multiproduct/env.py",
line 54, in __init__
super(Environment, self).__init__(path, create=create,
options=options)
File "/path/to/bloodhound/trac/trac/core.py", line 107, in new_init
original_init(self, *args, **kwargs)
File "/path/to/bloodhound/trac/trac/env.py", line 289, in __init__
setup_participant.environment_created()
File "/path/to/bloodhound/trac/trac/env.py", line 743, in
environment_created
vals)
File "/path/to/bloodhound/trac/trac/db/util.py", line 139, in
executemany
cursor.executemany(query, params)
File
"/path/to/bloodhound/bloodhound_multiproduct/multiproduct/dbcursor.py",
line 79, in executemany
return super(BloodhoundIterableCursor,
self).executemany(self._translate_sql(sql), args=args)
File "/path/to/bloodhound/trac/trac/db/util.py", line 85, in executemany
return self.cursor.executemany(sql_escape_percent(sql), args)
File "/path/to/bloodhound/trac/trac/db/sqlite_backend.py", line 62, in
executemany
args)
File "/path/to/bloodhound/trac/trac/db/sqlite_backend.py", line 48, in
_rollback_on_error
return function(self, *args, **kwargs)
OperationalError: table component has no column named product
[...]
}}}
So I guess something in there is breaking multi-product upgrades . It
seems to be related to the translation of CREATE and/or ALTER statements ,
but I'm not 100% sure . Anyway ...
@jure : could you please take a look and review ?
I'll continue trying to figure out what else needs to be done , hoping
I'll find the root cause , but I share the code soon looking forward to
your feedback .
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:18>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: assigned
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Changes (by jure):
* keywords: bep-003 multiproduct => bep-0003 multiproduct
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:2>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Replying to [comment:8 olemis]:
> AFAICS the issue is related to this ticket .
It was, permission table did not get translated, in r1438011 the table has
been added into translated tables list.
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:11>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: closed
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: fixed | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Changes (by jure):
* status: accepted => closed
* resolution: => fixed
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:7>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
ok . I'll check once again asap . The good news is that we are moving
towards getting this done and everything seems to be on the right track ,
at least considering the development stage we are carrying out now . Your
enhancements are really working much better than my initial solution did .
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:21>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Working `get_read_db` and `get_db_cnx` available in r1446579.
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:25>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: closed
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: fixed | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Changes (by jure):
* status: new => closed
* resolution: => fixed
Comment:
No major issues found in the last month, resolving the ticket...
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:26>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: closed
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: fixed | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
Shall we reopen it ?
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:9>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
At some point in time while working on #355 I changed product
environment's `get_read_db` method to always raise `NotImplemented`
exception with deprecation warning message . Now , while translating test
cases for ticket query module all test cases but one will fail because
that subsystem still makes use of aforementioned method .
@jure : could you please provide in `bep_0003_multiproduct` a (preferently
tested ;) implementation of `ProductEnvironment.get_read_db` consistent
with latest API and the underlying SQL translation mechanism ?
PS: In the meantime I'll be able to continue with other pending test case
migration tasks . Thanks in advance .
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:24>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: assigned
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-003 multiproduct
---------------------------+----------------------------------
Changes (by jure):
* owner: nobody => jure
* status: new => assigned
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:1>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
Replying to [comment:12 olemis]:
> Replying to [comment:11 jure]:
> > Replying to [comment:8 olemis]:
> >
> > > AFAICS the issue is related to this ticket .
> >
> > It was, permission table did not get translated, in r1438011 the table
has been added into translated tables list.
>
> Thanks very much .
>
... but ... there still some issues identified while running test cases
for dafault permissions store . The most critical being
{{{
======================================================================
ERROR: test_env_isolation
(tests.perm.ProductDefaultPermissionStoreTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/bloodhound/bloodhound_multiproduct/tests/perm.py", line
58, in test_env_isolation
('john', 'dev')])
File "/path/to/bloodhound/trac/trac/db/api.py", line 123, in executemany
return db.executemany(query, params)
File "/path/to/bloodhound/trac/trac/db/util.py", line 139, in
executemany
cursor.executemany(query, params)
File
"/path/to/bloodhound/bloodhound_multiproduct/multiproduct/dbcursor.py",
line 65, in executemany
return super(BloodhoundIterableCursor,
self).executemany(self._translate_sql(sql), args=args)
File "/path/to/bloodhound/trac/trac/db/util.py", line 77, in executemany
args)
File "/path/to/bloodhound/trac/trac/db/sqlite_backend.py", line 62, in
executemany
args)
File "/path/to/bloodhound/trac/trac/db/sqlite_backend.py", line 48, in
_rollback_on_error
return function(self, *args, **kwargs)
OperationalError: table permission has 3 columns but 2 values were
supplied
}}}
@jure : could you please take a look ?
Anyway , I'll continue working on #355 . By the time this will be fixed
I'll come back to this once again .
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:14>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
t288_r1438538_sql_translate_global_env.diff applied in r1440346 with some
modifications, most important being implementation of `execute` and
`executemany` on `ProductEnvContextManager` and modification of the
`__call__` method ... all three now properly set environment on the
iterable cursor to properly scope SQL ...
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:19>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: accepted
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
This has been implemented on bep_0003_multiproduct branch. The basic
functionality is in place. New tickets will be entered if there are issues
found with the implementation.
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:6>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: assigned
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Changes (by olemis):
* cc: olemis+bh@… (added)
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:3>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: closed
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: fixed | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
After running the [comment:8:ticket:355 test suite for default permission
store] I've detected this failure :
{{{
#!sh
$ /srv/venv/python/trac/0.13/bin/python setup.py test -m tests.perm
running test
running egg_info
writing BloodhoundMultiProduct.egg-info/PKG-INFO
writing top-level names to BloodhoundMultiProduct.egg-info/top_level.txt
writing dependency_links to BloodhoundMultiProduct.egg-
info/dependency_links.txt
writing entry points to BloodhoundMultiProduct.egg-info/entry_points.txt
writing BloodhoundMultiProduct.egg-info/PKG-INFO
writing top-level names to BloodhoundMultiProduct.egg-info/top_level.txt
writing dependency_links to BloodhoundMultiProduct.egg-
info/dependency_links.txt
writing entry points to BloodhoundMultiProduct.egg-info/entry_points.txt
reading manifest file 'BloodhoundMultiProduct.egg-info/SOURCES.txt'
writing manifest file 'BloodhoundMultiProduct.egg-info/SOURCES.txt'
running build_ext
test_builtin_groups (tests.perm.ProductDefaultPermissionStoreTestCase) ...
ok
test_env_isolation (tests.perm.ProductDefaultPermissionStoreTestCase) ...
ERROR
test_get_all_permissions
(tests.perm.ProductDefaultPermissionStoreTestCase) ... ok
test_mixed_case_group (tests.perm.ProductDefaultPermissionStoreTestCase)
... ok
test_nested_groups (tests.perm.ProductDefaultPermissionStoreTestCase) ...
ok
test_simple_actions (tests.perm.ProductDefaultPermissionStoreTestCase) ...
ok
test_simple_group (tests.perm.ProductDefaultPermissionStoreTestCase) ...
ok
======================================================================
ERROR: test_env_isolation
(tests.perm.ProductDefaultPermissionStoreTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/bloodhound/bloodhound_multiproduct/tests/perm.py", line
63, in test_env_isolation
('john', 'dev')])
File "/path/to/bloodhound/trac/trac/db/api.py", line 123, in executemany
return db.executemany(query, params)
File "/path/to/bloodhound/trac/trac/db/util.py", line 139, in
executemany
cursor.executemany(query, params)
File
"/path/to/bloodhound/bloodhound_multiproduct/multiproduct/dbcursor.py",
line 65, in executemany
return super(BloodhoundIterableCursor,
self).executemany(self._translate_sql(sql), args=args)
File "/path/to/bloodhound/trac/trac/db/util.py", line 77, in executemany
args)
File "/path/to/bloodhound/trac/trac/db/sqlite_backend.py", line 62, in
executemany
args)
File "/path/to/bloodhound/trac/trac/db/sqlite_backend.py", line 48, in
_rollback_on_error
return function(self, *args, **kwargs)
IntegrityError: columns username, action are not unique
----------------------------------------------------------------------
Ran 7 tests in 0.193s
FAILED (errors=1)
}}}
AFAICS the issue is related to this ticket .
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:8>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Replying to [comment:22 olemis]:
> [attachment:t288_r1441419_sql_translate_cursor.diff Attached patch] is
related to comment:18:ticket:355
Patch applied in r1442889
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:23>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Changes (by jure):
* status: closed => new
* resolution: fixed =>
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:10>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Replying to [comment:15 jure]:
> Replying to [comment:14 olemis]:
> >
> > [...]
> >
> > @jure : could you please take a look ?
> >
>
> Yes, that's what I'm looking into atm, the problem are INSERTs w/o
specifying column names as we get column mismatches. I plan to have a
patch for this later today.
>
Fix available in r1438538.
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:16>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
[BEP-0003] Database schema schanges (Was: [Apache Bloodhound] #288:
Create prototype for legacy database schema proxy)
Posted by Jure Zitnik <ju...@digiverse.si>.
Hi,
as described in ticket #288, I produced an (experimental!) patch
(attached to the ticket and commited on a multiproduct branch - see
below) that provides functionalities as described in the ticket description:
- extends enum, component, version, milestone and wiki tables with
product column
- upgrades existing database to a new schema by properly migrating
existing configuration to a default product and multiplying the
configuration for all products defined
- in runtime:
- introduces BloodhoundIterableCursor that translates all SQLs
targeted at the affected tables
- introduces BloodhoundEnvironment skeleton that replaces trac
Environment with product specific functionality (awareness, scope)
- all SQLs targeted at the database are 'scoped' to a product defined
in the environment
- the only drawback is that there is currently no UI way of changing
product scope, at this point in time it can only be changed by modifying
DEFAULT_PRODUCT in bloodhound.py and restarting/reloading bloodhound runtime
What this mean in practice is that through controlling product scope
within BloodhoundEnvironment, one can (in runtime) easily change product
scope and in turn affect all queries targeted at the resources that we
want to be product specific. For now these resources include milestones,
versions, components, enums (resolutions, priorities, ticket types).
Please test the patch and let me know if you run into any issues. Note
that sqlparse (https://code.google.com/p/python-sqlparse/) is a
prerequisite for the patch to work.
I also created a Bloodhound branch that we will use for the BEP-0003
multiproduct development. I applied the patch from ticket #288 and we
can go from there to implementing other BEP-0003 functionalities. I
think that multi product branch should be separated as it affect
basically everything from the UI to the database and will take some time
before we have a 'release ready' solution.
Cheers,
Jure
On 12/11/12 11:00 AM, Apache Bloodhound wrote:
> #288: Create prototype for legacy database schema proxy
> ---------------------------+-----------------------------------
> Reporter: jure | Owner: jure
> Type: enhancement | Status: assigned
> Priority: major | Milestone:
> Component: multiproduct | Version:
> Resolution: | Keywords: bep-0003 multiproduct
> ---------------------------+-----------------------------------
>
> Comment (by jure):
>
> As described in ticket description, attached patch
> (ticket288_r1418224_MultiproductSQLProxy.diff) is an initial
> implementation of SQL proxy with the fore-mentioned functionality. It
> monkey-patches trac to install BloodhoundIterableCursor and
> BloodhoundEnvironment to capture all SQLs targeted at the database and
> provides product aware/scope selection.
>
> The patch does the following:
> - introduces modified database schema by extending enum, component,
> milestone, version and wiki tables with additional column named product
> - database upgrade
> - migrates existing enums, components, milestones, versions to a default
> product and, in addition to that, multiplies the configuration for each
> defined product
> - tickets w/o product are migrated to default product
> - in runtime, the BloodhoundIterableCursor translates SQLs targeted at the
> tables mentioned above to the scope of the product defined in
> bloodhound.py (DEFAULT_PRODUCT). This effectively scopes all SQLs targeted
> to translated tables to that product
> - currently, as there is no obvious UI way of chaning product scope,
> currently selected product scope can be changed by modifying the
> bloodhound.py file and chaning the DEFAULT_PRODUCT...
>
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: assigned
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
As described in ticket description, attached patch
(ticket288_r1418224_MultiproductSQLProxy.diff) is an initial
implementation of SQL proxy with the fore-mentioned functionality. It
monkey-patches trac to install BloodhoundIterableCursor and
BloodhoundEnvironment to capture all SQLs targeted at the database and
provides product aware/scope selection.
The patch does the following:
- introduces modified database schema by extending enum, component,
milestone, version and wiki tables with additional column named product
- database upgrade
- migrates existing enums, components, milestones, versions to a default
product and, in addition to that, multiplies the configuration for each
defined product
- tickets w/o product are migrated to default product
- in runtime, the BloodhoundIterableCursor translates SQLs targeted at the
tables mentioned above to the scope of the product defined in
bloodhound.py (DEFAULT_PRODUCT). This effectively scopes all SQLs targeted
to translated tables to that product
- currently, as there is no obvious UI way of chaning product scope,
currently selected product scope can be changed by modifying the
bloodhound.py file and chaning the DEFAULT_PRODUCT...
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:4>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
[attachment:t288_r1441419_sql_translate_cursor.diff Attached patch] is
related to comment:18:ticket:355
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:22>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Olemis Lang <ol...@gmail.com>.
On 1/28/13, Jure Zitnik <ju...@digiverse.si> wrote:
> On 1/25/13 7:05 PM, Apache Bloodhound wrote:
>> > One note though, the permission test should include product column
>> value
>> when inserting into global environment. But you might wait with that
>> till
>> after my patch...
>>
>> IMO it's quite important to be consistent with the previous approach
>> (i.e.
>> no explicit reference to products in SQL queries ). All this should be
>> encapsulated e.g. the proxy has to detect whether a product is not set
>> in
>> context and still translate the query considering `product = NULL` (
>> `product = ''` ? ) . All instances of `product = NULL` will refer to
>> resources in the global environment . That would be enough to keep the
>> interface exactly the same as before .
>>
>> I don't know how much complicated this will be for you to add this
>> feature
>> , but IMO it's a **MUST** . Is it feasible up to this point ?
>>
>> Everything **must** be just like before for both product envs and
>> global
>> env . That's exactly the point .
>> ''';)'''
>
> I don't find 'product detection' code very appealing. Assuming that
> detecting the product column within the INSERT statement would be
> relatively easy to do, detecting WHERE conditions is not that trivial.
>
I'm aware that there are some subtle details to consider , yes ...
> Therefor I would suggest the following solution - the code should always
> run within ProductEnvironment (and never global Environment).
-1 ... if only widgets and theme is actually needed then multi-product
plugin may be disabled and upgrades skipped .
> In the
> global context (URL namespace), the code should get the
> ProductEnvironment with the `product == ''`, which would reference
> global resources. This would keep the backward compatibility for 3rd
> party plugins/trac code even when running in global context - with the
> database view limited to 'global product'.
>
I think of this a different way . IMO the global environment should be
the usual instance of `trac.env.Environment` class .
However in either case there's a subtle argument to consider and it is
that the contract of Environment class spans beyond the traditional
situation considering attributes , methods , etc ... It also encloses
the fact that components running on top of it will expect an
underlying DB schema to be installed .
Your suggestion consists in breaking that part of the contract at the
exact point where it's always worked and implement a whole new concept
sticking to it as before . In spite making components functional with
little or no modifications by keeping them agnostic to multi-product
enhancements , that approach IMO does not play well .
If not convinced yet I'll mention one such use case . In previous
threads we discussed the fact that site admin *must only* be performed
in global scope and only a subset of those tasks should also be
allowed for product admin . That excludes e.g. environment upgrade
tasks , and also means that checks for TRAC_ADMIN permission might
only be valid in global scope .
The other side of the same coin is that permissions are asserted
against the global env . Those checks are performed by one of those
components that should be muti-product agnostic (e.g.
`DefaultPermissionStore`) instantiated at global environment scope
(i.e. `trac.env.Environment` ). The SQL queries in component source
code are exactly the same as those used for product environments
themselves i.e. no reference to product prefix or alike . At that time
you'll realize that they will fail (like it happens nowadays in perm
test cases, that's what it is for ;) and the Bloodhound instance will
be unusable .
The same reasoning is valid for components implementing
IEnvironmentSetupParticipant because they'll only be instantiated at
global scope (i.e. `trac.env.Environment` ) ... and so on .
[...]
> The translator code would still be disabled
> when accessing database through env.parent, making it possible for the
> code/plugins to access all product's resources.
>
IMO , we'll have to figure out another to get this done rather than
breaking the global environment .
--
Regards,
Olemis.
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Jure Zitnik <ju...@digiverse.si>.
On 1/25/13 7:05 PM, Apache Bloodhound wrote:
> > One note though, the permission test should include product column value
> when inserting into global environment. But you might wait with that till
> after my patch...
>
> IMO it's quite important to be consistent with the previous approach (i.e.
> no explicit reference to products in SQL queries ). All this should be
> encapsulated e.g. the proxy has to detect whether a product is not set in
> context and still translate the query considering `product = NULL` (
> `product = ''` ? ) . All instances of `product = NULL` will refer to
> resources in the global environment . That would be enough to keep the
> interface exactly the same as before .
>
> I don't know how much complicated this will be for you to add this feature
> , but IMO it's a **MUST** . Is it feasible up to this point ?
>
> Everything **must** be just like before for both product envs and global
> env . That's exactly the point .
> ''';)'''
I don't find 'product detection' code very appealing. Assuming that
detecting the product column within the INSERT statement would be
relatively easy to do, detecting WHERE conditions is not that trivial.
Therefor I would suggest the following solution - the code should always
run within ProductEnvironment (and never global Environment). In the
global context (URL namespace), the code should get the
ProductEnvironment with the `product == ''`, which would reference
global resources. This would keep the backward compatibility for 3rd
party plugins/trac code even when running in global context - with the
database view limited to 'global product'.
The product aware code would, on the other hand, in both cases (global
and/or product context), have access to global view of the resources
through env.parent property. The translator code would still be disabled
when accessing database through env.parent, making it possible for the
code/plugins to access all product's resources.
Any other takes on this?
Cheers,
Jure
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
Replying to [comment:15 jure]:
> Replying to [comment:14 olemis]:
[...]
> >
> > ... but ... there still some issues identified while running test
cases for dafault permissions store . The most critical being
> >
> > [...]
> >
> > @jure : could you please take a look ?
> >
>
> Yes, that's what I'm looking into atm, the problem are INSERTs w/o
specifying column names as we get column mismatches. I plan to have a
patch for this later today.
>
I see it's already there . I'll take a look once the fix mentioned below
will be added .
> > Anyway , I'll continue working on #355 . By the time this will be
fixed I'll come back to this once again .
>
> Great.
:)
> One note though, the permission test should include product column value
when inserting into global environment. But you might wait with that till
after my patch...
IMO it's quite important to be consistent with the previous approach (i.e.
no explicit reference to products in SQL queries ). All this should be
encapsulated e.g. the proxy has to detect whether a product is not set in
context and still translate the query considering `product = NULL` (
`product = ''` ? ) . All instances of `product = NULL` will refer to
resources in the global environment . That would be enough to keep the
interface exactly the same as before .
I don't know how much complicated this will be for you to add this feature
, but IMO it's a **MUST** . Is it feasible up to this point ?
Everything **must** be just like before for both product envs and global
env . That's exactly the point .
''';)'''
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:17>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Replying to [comment:14 olemis]:
> Replying to [comment:12 olemis]:
> > Replying to [comment:11 jure]:
> > > Replying to [comment:8 olemis]:
> > >
> > > > AFAICS the issue is related to this ticket .
> > >
> > > It was, permission table did not get translated, in r1438011 the
table has been added into translated tables list.
> >
> > Thanks very much .
> >
>
> ... but ... there still some issues identified while running test cases
for dafault permissions store . The most critical being
>
> [...]
>
> @jure : could you please take a look ?
>
Yes, that's what I'm looking into atm, the problem are INSERTs w/o
specifying column names as we get column mismatches. I plan to have a
patch for this later today.
> Anyway , I'll continue working on #355 . By the time this will be fixed
I'll come back to this once again .
Great. One note though, the permission test should include product column
value when inserting into global environment. But you might wait with that
till after my patch...
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:15>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: accepted
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Changes (by jure):
* status: assigned => accepted
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:5>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Follow up ... r1440970 implements proper handling of transaction scopes.
Solution uses connection wrappers to keep the context of the transactions
and properly set cursor context when executing queries. Most of the tests
work, I'm currently looking into those that are not. Please test it out
and let me know how it performs. Anyway, the `db_direct_transaction` now
works.
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:20>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by olemis):
Replying to [comment:11 jure]:
> Replying to [comment:8 olemis]:
>
> > AFAICS the issue is related to this ticket .
>
> It was, permission table did not get translated, in r1438011 the table
has been added into translated tables list.
Thanks very much .
I wander ... how will translation be performed when dealing with tables
contributed by plugins ?
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:12>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker
Re: [Apache Bloodhound] #288: Create prototype for legacy database
schema proxy
Posted by Apache Bloodhound <bl...@incubator.apache.org>.
#288: Create prototype for legacy database schema proxy
---------------------------+-----------------------------------
Reporter: jure | Owner: jure
Type: enhancement | Status: new
Priority: major | Milestone:
Component: multiproduct | Version:
Resolution: | Keywords: bep-0003 multiproduct
---------------------------+-----------------------------------
Comment (by jure):
Replying to [comment:12 olemis]:
>
> I wander ... how will translation be performed when dealing with tables
contributed by plugins ?
Tables contributed by plugins will be prefixed with the product prefix
(table name that is).
--
Ticket URL: <https://issues.apache.org/bloodhound/ticket/288#comment:13>
Apache Bloodhound <https://issues.apache.org/bloodhound/>
The Apache Bloodhound (incubating) issue tracker