Provide support for tracking of in-place changes to scalar values, which are propagated into ORM change events on owning parent objects.
New in version 0.7: sqlalchemy.ext.mutable
replaces SQLAlchemy’s
legacy approach to in-place mutations of scalar values; see
Mutation event extension, supersedes “mutable=True”.
A typical example of a “mutable” structure is a Python dictionary. Following the example introduced in Column and Data Types, we begin with a custom type that marshals Python dictionaries into JSON strings before being persisted:
from sqlalchemy.types import TypeDecorator, VARCHAR
import json
class JSONEncodedDict(TypeDecorator):
"Represents an immutable structure as a json-encoded string."
impl = VARCHAR
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
The usage of json
is only for the purposes of example. The
sqlalchemy.ext.mutable
extension can be used
with any type whose target Python type may be mutable, including
PickleType
, postgresql.ARRAY
, etc.
When using the sqlalchemy.ext.mutable
extension, the value itself
tracks all parents which reference it. Below, we illustrate the a simple
version of the MutableDict
dictionary object, which applies
the Mutable
mixin to a plain Python dictionary:
from sqlalchemy.ext.mutable import Mutable
class MutableDict(Mutable, dict):
@classmethod
def coerce(cls, key, value):
"Convert plain dictionaries to MutableDict."
if not isinstance(value, MutableDict):
if isinstance(value, dict):
return MutableDict(value)
# this call will raise ValueError
return Mutable.coerce(key, value)
else:
return value
def __setitem__(self, key, value):
"Detect dictionary set events and emit change events."
dict.__setitem__(self, key, value)
self.changed()
def __delitem__(self, key):
"Detect dictionary del events and emit change events."
dict.__delitem__(self, key)
self.changed()
The above dictionary class takes the approach of subclassing the Python
built-in dict
to produce a dict
subclass which routes all mutation events through __setitem__
. There are
variants on this approach, such as subclassing UserDict.UserDict
or
collections.MutableMapping
; the part that’s important to this example is
that the Mutable.changed()
method is called whenever an in-place
change to the datastructure takes place.
We also redefine the Mutable.coerce()
method which will be used to
convert any values that are not instances of MutableDict
, such
as the plain dictionaries returned by the json
module, into the
appropriate type. Defining this method is optional; we could just as well
created our JSONEncodedDict
such that it always returns an instance
of MutableDict
, and additionally ensured that all calling code
uses MutableDict
explicitly. When Mutable.coerce()
is not
overridden, any values applied to a parent object which are not instances
of the mutable type will raise a ValueError
.
Our new MutableDict
type offers a class method
as_mutable()
which we can use within column metadata
to associate with types. This method grabs the given type object or
class and associates a listener that will detect all future mappings
of this type, applying event listening instrumentation to the mapped
attribute. Such as, with classical table metadata:
from sqlalchemy import Table, Column, Integer
my_data = Table('my_data', metadata,
Column('id', Integer, primary_key=True),
Column('data', MutableDict.as_mutable(JSONEncodedDict))
)
Above, as_mutable()
returns an instance of JSONEncodedDict
(if the type object was not an instance already), which will intercept any
attributes which are mapped against this type. Below we establish a simple
mapping against the my_data
table:
from sqlalchemy import mapper
class MyDataClass(object):
pass
# associates mutation listeners with MyDataClass.data
mapper(MyDataClass, my_data)
The MyDataClass.data
member will now be notified of in place changes
to its value.
There’s no difference in usage when using declarative:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyDataClass(Base):
__tablename__ = 'my_data'
id = Column(Integer, primary_key=True)
data = Column(MutableDict.as_mutable(JSONEncodedDict))
Any in-place changes to the MyDataClass.data
member
will flag the attribute as “dirty” on the parent object:
>>> from sqlalchemy.orm import Session
>>> sess = Session()
>>> m1 = MyDataClass(data={'value1':'foo'})
>>> sess.add(m1)
>>> sess.commit()
>>> m1.data['value1'] = 'bar'
>>> assert m1 in sess.dirty
True
The MutableDict
can be associated with all future instances
of JSONEncodedDict
in one step, using
associate_with()
. This is similar to
as_mutable()
except it will intercept all occurrences
of MutableDict
in all mappings unconditionally, without
the need to declare it individually:
MutableDict.associate_with(JSONEncodedDict)
class MyDataClass(Base):
__tablename__ = 'my_data'
id = Column(Integer, primary_key=True)
data = Column(JSONEncodedDict)
The key to the sqlalchemy.ext.mutable
extension relies upon the
placement of a weakref.WeakKeyDictionary
upon the value object, which
stores a mapping of parent mapped objects keyed to the attribute name under
which they are associated with this value. WeakKeyDictionary
objects are
not picklable, due to the fact that they contain weakrefs and function
callbacks. In our case, this is a good thing, since if this dictionary were
picklable, it could lead to an excessively large pickle size for our value
objects that are pickled by themselves outside of the context of the parent.
The developer responsibility here is only to provide a __getstate__
method
that excludes the _parents()
collection from the pickle
stream:
class MyMutableType(Mutable):
def __getstate__(self):
d = self.__dict__.copy()
d.pop('_parents', None)
return d
With our dictionary example, we need to return the contents of the dict itself (and also restore them on __setstate__):
class MutableDict(Mutable, dict):
# ....
def __getstate__(self):
return dict(self)
def __setstate__(self, state):
self.update(state)
In the case that our mutable value object is pickled as it is attached to one
or more parent objects that are also part of the pickle, the Mutable
mixin will re-establish the Mutable._parents
collection on each value
object as the owning parents themselves are unpickled.
Composites are a special ORM feature which allow a single scalar attribute to be assigned an object value which represents information “composed” from one or more columns from the underlying mapped table. The usual example is that of a geometric “point”, and is introduced in Composite Column Types.
Changed in version 0.7: The internals of orm.composite()
have been
greatly simplified and in-place mutation detection is no longer enabled by
default; instead, the user-defined value must detect changes on its own and
propagate them to all owning parents. The sqlalchemy.ext.mutable
extension provides the helper class MutableComposite
, which is a
slight variant on the Mutable
class.
As is the case with Mutable
, the user-defined composite class
subclasses MutableComposite
as a mixin, and detects and delivers
change events to its parents via the MutableComposite.changed()
method.
In the case of a composite class, the detection is usually via the usage of
Python descriptors (i.e. @property
), or alternatively via the special
Python method __setattr__()
. Below we expand upon the Point
class
introduced in Composite Column Types to subclass MutableComposite
and to also route attribute set events via __setattr__
to the
MutableComposite.changed()
method:
from sqlalchemy.ext.mutable import MutableComposite
class Point(MutableComposite):
def __init__(self, x, y):
self.x = x
self.y = y
def __setattr__(self, key, value):
"Intercept set events"
# set the attribute
object.__setattr__(self, key, value)
# alert all parents to the change
self.changed()
def __composite_values__(self):
return self.x, self.y
def __eq__(self, other):
return isinstance(other, Point) and \
other.x == self.x and \
other.y == self.y
def __ne__(self, other):
return not self.__eq__(other)
The MutableComposite
class uses a Python metaclass to automatically
establish listeners for any usage of orm.composite()
that specifies our
Point
type. Below, when Point
is mapped to the Vertex
class,
listeners are established which will route change events from Point
objects to each of the Vertex.start
and Vertex.end
attributes:
from sqlalchemy.orm import composite, mapper
from sqlalchemy import Table, Column
vertices = Table('vertices', metadata,
Column('id', Integer, primary_key=True),
Column('x1', Integer),
Column('y1', Integer),
Column('x2', Integer),
Column('y2', Integer),
)
class Vertex(object):
pass
mapper(Vertex, vertices, properties={
'start': composite(Point, vertices.c.x1, vertices.c.y1),
'end': composite(Point, vertices.c.x2, vertices.c.y2)
})
Any in-place changes to the Vertex.start
or Vertex.end
members
will flag the attribute as “dirty” on the parent object:
>>> from sqlalchemy.orm import Session
>>> sess = Session()
>>> v1 = Vertex(start=Point(3, 4), end=Point(12, 15))
>>> sess.add(v1)
>>> sess.commit()
>>> v1.end.x = 8
>>> assert v1 in sess.dirty
True
The MutableBase.coerce()
method is also supported on composite types.
In the case of MutableComposite
, the MutableBase.coerce()
method is only called for attribute set operations, not load operations.
Overriding the MutableBase.coerce()
method is essentially equivalent
to using a validates()
validation routine for all attributes which
make use of the custom composite type:
class Point(MutableComposite):
# other Point methods
# ...
def coerce(cls, key, value):
if isinstance(value, tuple):
value = Point(*value)
elif not isinstance(value, Point):
raise ValueError("tuple or Point expected")
return value
New in version 0.7.10,0.8.0b2: Support for the MutableBase.coerce()
method in conjunction with
objects of type MutableComposite
.
As is the case with Mutable
, the MutableComposite
helper
class uses a weakref.WeakKeyDictionary
available via the
MutableBase._parents()
attribute which isn’t picklable. If we need to
pickle instances of Point
or its owning class Vertex
, we at least need
to define a __getstate__
that doesn’t include the _parents
dictionary.
Below we define both a __getstate__
and a __setstate__
that package up
the minimal form of our Point
class:
class Point(MutableComposite):
# ...
def __getstate__(self):
return self.x, self.y
def __setstate__(self, state):
self.x, self.y = state
As with Mutable
, the MutableComposite
augments the
pickling process of the parent’s object-relational state so that the
MutableBase._parents()
collection is restored to all Point
objects.
sqlalchemy.ext.mutable.
MutableBase
¶Common base class to Mutable
and MutableComposite
.
_parents
¶Dictionary of parent object->attribute name on the parent.
This attribute is a so-called “memoized” property. It initializes
itself with a new weakref.WeakKeyDictionary
the first time
it is accessed, returning the same object upon subsequent access.
coerce
(key, value)¶Given a value, coerce it into the target type.
Can be overridden by custom subclasses to coerce incoming data into a particular type.
By default, raises ValueError
.
This method is called in different scenarios depending on if
the parent class is of type Mutable
or of type
MutableComposite
. In the case of the former, it is called
for both attribute-set operations as well as during ORM loading
operations. For the latter, it is only called during attribute-set
operations; the mechanics of the composite()
construct
handle coercion during load operations.
Parameters: | |
---|---|
Returns: | the method should return the coerced value, or raise
|
sqlalchemy.ext.mutable.
Mutable
¶Bases: sqlalchemy.ext.mutable.MutableBase
Mixin that defines transparent propagation of change events to a parent object.
See the example in Establishing Mutability on Scalar Column Values for usage information.
__init__
¶__init__
attribute of object
x.__init__(...) initializes x; see help(type(x)) for signature
_get_listen_keys
(attribute)¶_get_listen_keys()
method of MutableBase
Given a descriptor attribute, return a set()
of the attribute
keys which indicate a change in the state of this attribute.
This is normally just set([attribute.key])
, but can be overridden
to provide for additional keys. E.g. a MutableComposite
augments this set with the attribute keys associated with the columns
that comprise the composite value.
This collection is consulted in the case of intercepting the
InstanceEvents.refresh()
and
InstanceEvents.refresh_flush()
events, which pass along a list
of attribute names that have been refreshed; the list is compared
against this set to determine if action needs to be taken.
New in version 1.0.5.
_listen_on_attribute
(attribute, coerce, parent_cls)¶_listen_on_attribute()
method of MutableBase
Establish this type as a mutation listener for the given mapped descriptor.
_parents
¶_parents
attribute of MutableBase
Dictionary of parent object->attribute name on the parent.
This attribute is a so-called “memoized” property. It initializes
itself with a new weakref.WeakKeyDictionary
the first time
it is accessed, returning the same object upon subsequent access.
as_mutable
(sqltype)¶Associate a SQL type with this mutable Python type.
This establishes listeners that will detect ORM mappings against the given type, adding mutation event trackers to those mappings.
The type is returned, unconditionally as an instance, so that
as_mutable()
can be used inline:
Table('mytable', metadata,
Column('id', Integer, primary_key=True),
Column('data', MyMutableType.as_mutable(PickleType))
)
Note that the returned type is always an instance, even if a class is given, and that only columns which are declared specifically with that type instance receive additional instrumentation.
To associate a particular mutable type with all occurrences of a
particular type, use the Mutable.associate_with()
classmethod
of the particular Mutable
subclass to establish a global
association.
Warning
The listeners established by this method are global
to all mappers, and are not garbage collected. Only use
as_mutable()
for types that are permanent to an application,
not with ad-hoc types else this will cause unbounded growth
in memory usage.
associate_with
(sqltype)¶Associate this wrapper with all future mapped columns of the given type.
This is a convenience method that calls
associate_with_attribute
automatically.
Warning
The listeners established by this method are global
to all mappers, and are not garbage collected. Only use
associate_with()
for types that are permanent to an
application, not with ad-hoc types else this will cause unbounded
growth in memory usage.
associate_with_attribute
(attribute)¶Establish this type as a mutation listener for the given mapped descriptor.
changed
()¶Subclasses should call this method whenever change events occur.
coerce
(key, value)¶coerce()
method of MutableBase
Given a value, coerce it into the target type.
Can be overridden by custom subclasses to coerce incoming data into a particular type.
By default, raises ValueError
.
This method is called in different scenarios depending on if
the parent class is of type Mutable
or of type
MutableComposite
. In the case of the former, it is called
for both attribute-set operations as well as during ORM loading
operations. For the latter, it is only called during attribute-set
operations; the mechanics of the composite()
construct
handle coercion during load operations.
Parameters: | |
---|---|
Returns: | the method should return the coerced value, or raise
|
sqlalchemy.ext.mutable.
MutableComposite
¶Bases: sqlalchemy.ext.mutable.MutableBase
Mixin that defines transparent propagation of change events on a SQLAlchemy “composite” object to its owning parent or parents.
See the example in Establishing Mutability on Composites for usage information.
changed
()¶Subclasses should call this method whenever change events occur.
sqlalchemy.ext.mutable.
MutableDict
¶Bases: sqlalchemy.ext.mutable.Mutable
, __builtin__.dict
A dictionary type that implements Mutable
.
New in version 0.8.
clear
()¶coerce
(key, value)¶Convert plain dictionary to instance of this class.
pop
(*arg)¶popitem
()¶setdefault
(key, value)¶update
(*a, **kw)¶