Catalog

Object changes (Mutation)

__slots__ magic

Object changes (Mutation)

Python Medium variable (mutable) And immutable (immutable) New data types are a headache for novices . To put it simply , variable (mutable) signify " Can be changed ", And immutable (immutable) It means “ Constant (constant)”. Want to turn your head ？ Consider this example ：

foo = ['hi']
print(foo)
# Output: ['hi']
bar = foo
bar += ['bye']
print(foo)
# Output: ['hi', 'bye']

What just happened ？ We didn't expect that ！ We expect to see this ：

foo = ['hi']
print(foo)
# Output: ['hi']
bar = foo
bar += ['bye']
print(foo)
# Output: ['hi']
print(bar)
# Output: ['hi', 'bye']

This is not a bug. This is object variability (mutability) I'm making a mischief . Whenever you assign a variable to another variable of variable type , Any changes to this data will be reflected in these two variables at the same time . The new variable is just an alias for the old variable . This is only for variable data types . The following functions and variable data types let you understand at a glance ：

def add_to(num, target=[]):
    target.append(num)
    return target
add_to(1)
# Output: [1]
add_to(2)
# Output: [1, 2]
add_to(3)
# Output: [1, 2, 3]

You might expect it to behave differently . You may wish , When you call add_to when , A new list is created , Just like this. ：

def add_to(num, target=[]):
    target.append(num)
    return target
add_to(1)
# Output: [1]
add_to(2)
# Output: [2]
add_to(3)
# Output: [3]

Ah ！ This time it didn't meet the expectation , It is the variability of the list that is causing trouble . stay Python When a function is defined in , The default parameter is only evaluated once , Instead of recalculating every time it is called . You should never define default parameters for mutable types , Unless you know what you're doing . You should do it like this ：

def add_to(element, target=None):
    if target is None:
        target = []
    target.append(element)
    return target

Now whenever you call this function, it doesn't pass in target Parameter time , A new list will be created . for instance ：

add_to(42)
# Output: [42]
add_to(42)
# Output: [42]
add_to(42)
# Output: [42]

__slots__ magic

stay Python in , Each class has instance properties . By default Python Use a dictionary to store the instance properties of an object . This is very useful . Because it allows us to set any new properties at run time .

However , For small classes with known properties , It may be a bottleneck . This dictionary wastes a lot of memory .

Python You can't directly allocate a fixed amount of memory to store all properties when creating an object . So if you create many objects （ I mean thousands of ）, It consumes a lot of memory .

But there is still a way to avoid this problem . This method needs to use __slots__ To tell Python Don't use a dictionary , And only allocate space to the attributes of a fixed set .

Here is a use and not use __slots__ Example ：

Don't use __slots__:

class MyClass(object):
    def __init__(self, name, identifier):
        self.name = name
        self.identifier = identifier
        self.set_up()

Use __slots__:

class MyClass(object):
    __slots__ = ['name', 'identifier']
    def __init__(self, name, identifier):
        self.name = name
        self.identifier = identifier
        self.set_up()

The second piece of code will reduce the burden on your memory . Through this technique , Some people have seen that memory usage is almost 40%~50% The reduction of .

* A little note , You may need to try PyPy. It has done all these optimizations by default .

Here you can see an example , It USES IPython To show whether there is __slots__ The exact memory in case

Occupy .

Python 3.4.3 (default, Jun 6 2015, 13:32:34)

Type "copyright", "credits" or "license" for more information.

IPython 4.0.0 -- An enhanced Interactive Python.

?           -> Introduction and overview of IPython's features.

%quickref   -> Quick reference.

help        -> Python's own help system.

object?     -> Details about 'object', use 'object??' for extra details.

In [1]: import ipython_memory_usage.ipython_memory_usage as imu

In [2]: imu.start_watching_memory()

In [2] used 0.0000 MiB RAM in 5.31s, peaked 0.00 MiB above current, total RAM

In [3]: %cat slots.py

class MyClass(object):
    __slots__ = ['name', 'identifier']
    def __init__(self, name, identifier):
        self.name = name
        self.identifier = identifier

num = 1024*256

x = [MyClass(1,1) for i in range(num)]

In [3] used 0.2305 MiB RAM in 0.12s, peaked 0.00 MiB above current, total RAM

In [4]: from slots import *

In [4] used 9.3008 MiB RAM in 0.72s, peaked 0.00 MiB above current, total RAM

In [5]: %cat noslots.py

class MyClass(object):
    def __init__(self, name, identifier):
        self.name = name
        self.identifier = identifier

num = 1024*256

x = [MyClass(1,1) for i in range(num)]

In [5] used 0.1758 MiB RAM in 0.12s, peaked 0.00 MiB above current, total RAM

In [6]: from noslots import *

In [6] used 22.6680 MiB RAM in 0.80s, peaked 0.00 MiB above current