Teach you how to implement a python timer

author | Cloud King

source |  data STUDIO

Although many data workers believe that Python Is an effective programming language , But pure Python Procedure ratio C、Rust and Java The corresponding program in the compiled language runs slower , To better monitor and optimize Python Program , I will learn how to use it with you  Python Timer to monitor the speed of the program , In order to improve code performance .

In order to better grasp Python Application of timer , We will also add the following information about Python class 、 Context manager and decorator Background knowledge of . Due to space limitation , The context manager and decorator are used to optimize Python timer , I will learn about it in the following articles , Not within the scope of this article .

Python timer

First , We add a to a piece of code Python timer To monitor its performance .

Python Timer Functions

Python Built in time^[1] There are several functions in the module that can measure time ：

• monotonic()

• perf_counter()

• process_time()

• time()

Python 3.7 Several new functions are introduced , Such as thread_time()^[2], And all the above functions nanosecond edition , With _ns Suffix naming . for example ,perf_counter_ns() yes perf_counter() Nanosecond version of .

perf_counter()
Returns the value of the performance counter （ In seconds ）, That is, a clock with the highest available resolution to measure short duration .

First , Use perf_counter() Create a Python timer . Will put it with other Python Timer function for comparison , have a look perf_counter() The advantages of .

Example

Create a script , Define a short function ： Download a set of data from Tsinghua cloud .

import requests
def main():
    source_url = 'https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/files/?p=%2Fall_six_datasets.zip&dl=1'
    headers = {'User-Agent': 'Mozilla/5.0'}
    res = requests.get(source_url, headers=headers)
    with open('dataset/datasets.zip', 'wb') as f:
        f.write(res.content)
       
if __name__=="__main__":
    main()

We can use Python Timer to monitor the performance of the script .

first Python timer

Now use the function time.perf_counter() Function to create a timer , This is a very suitable counter for timing the performance of some code .

perf_counter() Measure time from an unspecified time （ In seconds ）, This means that the return value of a single call to the function is useless . But when looking at perf_counter() The difference between two calls , You can calculate how many seconds have passed between calls .

>>> import time
>>> time.perf_counter()
394.540232282
>>> time.perf_counter()  #  After a few seconds
413.31714087

In this example , Two calls perf_counter() Close to each other 19 second . This can be confirmed by calculating the difference between the two outputs ：413.31714087 - 394.540232282 = 18.78.

Now we can put Python Timers are added to the sample code ：

# download_data.py
import requests
import time
def main():
    tic = time.perf_counter()
    source_url = 'https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/files/?p=%2Fall_six_datasets.zip&dl=1'
    headers = {'User-Agent': 'Mozilla/5.0'}
    res = requests.get(source_url, headers=headers)
    with open('dataset/datasets.zip', 'wb') as f:
        f.write(res.content)
    toc = time.perf_counter()
    print(f" This procedure takes time : {toc - tic:0.4f} seconds")
if __name__=="__main__":
    main()

Be careful perf_counter() Calculate the difference between the two calls to print the time spent running the whole program .

print() Function f The preceding string indicates that this is a f-string , This is a convenient way to format text strings .:0.4f Is a format specifier , Representation number ,toc - tic Should be printed as a decimal number with four decimal places .

Run the program to see the elapsed time ：

 This procedure takes time : 0.026 seconds

It's that simple . Next, let's learn how to make Python The timer is wrapped in a class 、 A context manager and a decorator （ The next two articles in this series , To be updated ）, This makes it more consistent and convenient to use timers .

One Python Timer class

Here we need at least one variable to store Python The state of the timer . Next we create a manual call with perf_counter() The same class , But more readable and consistent .

Create and update Timer class , Use this class to time your code in many different ways .

$ python -m pip install codetiming

understand Python Class in

Class class Is the main building block of object-oriented programming . Class is essentially a template , You can use it to create object .

stay Python in , When you need to model things that need to track a specific state , Class is very useful . Generally speaking , A class is a collection of attributes , be called attribute , And behavior , be called Method .

establish Python Timer class

Class is useful for tracking state . stay Timer Class , Want to track when the timer starts and how much time has elapsed . about Timer The first implementation of the class , Will add a ._start_time Properties and .start() and .stop() Method . Add the following code to the file named timer.py In the file of ：

# timer.py
import time
class TimerError(Exception):
    """ A custom exception , Used for reporting Timer Class error """
class Timer:
    def __init__(self):
        self._start_time = None
    def start(self):
        """Start a new timer"""
        if self._start_time is not None:
            raise TimerError(f"Timer is running. Use .stop() to stop it")
        self._start_time = time.perf_counter()
    def stop(self):
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")
        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None
        print(f"Elapsed time: {elapsed_time:0.4f} seconds")

Here we need to spend some time looking at the code carefully , You will find something different .

So we define one TimerError Python class . The (Exception) Symbolic representation TimerError Inherit From another name Exception Parent class of . Use this built-in class for error handling . There is no need to TimerError Add any properties or methods , But custom errors can be handled more flexibly Timer Internal problems .

Next customize Timer class . When created from a class or Instantiation When an object , The code calls special methods .__init__() Initialize instance . The first one defined here Timer In the version , Just initialize ._start_time attribute , Will use it to track Python The state of the timer , When the timer is not running, its value is None. After the timer runs , Use it to track the start time of the timer .

Be careful ：._start_time The first underline of (_) The prefix is Python Appointment . It said ._start_time yes Timer The user of a class should not manipulate the internal properties of .

When calling .start() Start a new Python Timer time , First, check whether the timer is running . And then perf_counter() The current value of is stored in ._start_time in .

On the other hand , When calling .stop() when , First check Python Whether the timer is running . If it is , Then the running time is calculated as perf_counter() The current value of is stored in ._start_time The difference between the values in . Last , Reset ._start_time, To restart the timer , And print the running time .

Here are the USES Timer Method ：

from timer import Timer
t = Timer()
t.start()
#  After a few seconds
t.stop()

Elapsed time: 3.8191 seconds

Use this example directly with the previous perf_counter() Compare with the example of . The structure of the code is similar , But now the code is clearer , This is one of the benefits of using classes . By carefully selecting classes 、 Method and property names , Can make your code very descriptive !

Use Python Timer class

Now? Timer Class download_data.py. Just make some changes to the previous code ：

# download_data.py
import requests
from timer import Timer
def main():
    t = Timer()
    t.start()
    source_url = 'https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/files/?p=%2Fall_six_datasets.zip&dl=1'
    headers = {'User-Agent': 'Mozilla/5.0'}
    res = requests.get(source_url, headers=headers)
    with open('dataset/datasets.zip', 'wb') as f:
        f.write(res.content)
    t.stop()
if __name__=="__main__":
    main()

Be careful , This code is very similar to the code used before . In addition to making the code more readable ,Timer It is also responsible for printing the elapsed time to the console , Make the time record more consistent . When running code , The output is almost the same ：

Elapsed time: 0.502 seconds
...

Print elapsed time Timer May be consistent , But this method seems not very flexible . Let's add something more flexible to the code .

Add more convenience and flexibility

up to now , We have learned that classes are suitable for situations where we want to encapsulate state and ensure code consistency . In this section , We are going to give Python Timers add more convenience and flexibility , How to do it ？

• When reporting elapsed time , Use Adjustable text and formatting

• take logging Print to console 、 Write to a log file or other part of the program

• Create one that can be called multiple times Can accumulate Of Python timer

• structure Python The timer Information representation

First , Customize the text used to report time . In the previous code , Text f"Elapsed time: {elapsed_time:0.4f} seconds" Hard coded into .stop() in . If you want to make class code more flexible , You can use instance variables , Its value is usually passed as a parameter to .__init__() And store it to self attribute . For convenience , We can also provide reasonable default values .

You want to add .text by Timer Instance variables , You can do the following timer.py：

# timer.py
def __init__(self, text="Elapsed time: {:0.4f} seconds"):
    self._start_time = None
    self.text = text

Be careful , Default text "Elapsed time: {:0.4f} seconds" Is given as a regular string , instead of f-string. Not available here f-string, because f-string Will calculate immediately , When you instantiate Timer when , Your code hasn't figured out how much time it will take .

Be careful ： If you want to use f-string To specify the .text, You need to use double curly braces to escape the curly braces that will be replaced by the actual elapsed time .

Such as ：f"Finished {task} in { {:0.4f}} seconds". If task The value of is "reading", So this f-string Will be counted as "Finished reading in {:0.4f} seconds".

stay .stop() in ,.text Use as a template and use .format() Method to populate the template ：

# timer.py
def stop(self):
    """Stop the timer, and report the elapsed time"""
    if self._start_time is None:
        raise TimerError(f"Timer is not running. Use .start() to start it")
    elapsed_time = time.perf_counter() - self._start_time
    self._start_time = None
    print(self.text.format(elapsed_time))

Update here to timer.py after , You can change the text as follows ：

from timer import Timer
t = Timer(text="You waited {:.1f} seconds")
t.start()
#  After a few seconds
t.stop()

You waited 4.1 seconds

Next , We don't just want to print messages to the console , You also want to save the time measurement results , This makes it easy to store them in a database . You can do this by .stop() return elapsed_time To achieve this . then , The calling code can choose to ignore the return value or save it for later processing .

If you want to Timer Integration into logs logging in . To support logging or other output of timers , Need to change to print() Call to , So that users can provide their own logging functions . This can be done with text similar to what you customized before :

# timer.py
# ...
class Timer:
    def __init__(
        self,
        text="Elapsed time: {:0.4f} seconds",
        logger=print
    ):
        self._start_time = None
        self.text = text
        self.logger = logger
    #  Other methods remain unchanged
    def stop(self):
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")
        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None
        if self.logger:
            self.logger(self.text.format(elapsed_time))
        return elapsed_time

Not directly print(), Instead, create another instance variable self.logger, Reference a function that accepts a string as a parameter . besides , You can also use... For file objects logging.info() or .write() Such as function . Also pay attention to if in , It allows you to pass logger=None To turn off printing completely .

Here are two examples , It shows the practical application of the new function ：

from timer import Timer
import logging
t = Timer(logger=logging.warning)
t.start()
#  After a few seconds
t.stop()  # A few seconds later

WARNING:root:Elapsed time: 3.1610 seconds
3.1609658249999484

t = Timer(logger=None)
t.start()
#  After a few seconds
value = t.stop()
value

4.710851433001153

The next third improvement is The ability to accumulate time measurements . for example , When a slow function is called in a loop , I hope to add more functions in the form of named timers , And use a dictionary to track every... In the code Python timer .

We expand download_data.py Script .

# download_data.py
import requests
from timer import Timer
def main():
    t = Timer()
    t.start()
    source_url = 'https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/files/?p=%2Fall_six_datasets.zip&dl=1'
    headers = {'User-Agent': 'Mozilla/5.0'}
    for i in range(10):
        res = requests.get(source_url, headers=headers)
        with open('dataset/datasets.zip', 'wb') as f:
            f.write(res.content)
    t.stop()
if __name__=="__main__":
    main()

A subtle problem with this code is , Not only does it take time to download data , Also measure Python Time spent storing data to disk . This may not be important , Sometimes the time taken for both can be neglected . But I still hope there is a way to accurately time every step , It will be better .

There are several ways to do this without changing Timer Solve this problem with the current implementation , And it only needs a few lines of code to realize .

First , Will introduce a named .timers As a dictionary of Timer Class variables , here Timer All instances of will share it . Implement it by defining it outside of any method :

class Timer:
    timers = {}

Class variables can be accessed directly on the class , You can also access... Through an instance of a class ：

>>> from timer import Timer
>>> Timer.timers
{}
>>> t = Timer()
>>> t.timers
{}
>>> Timer.timers is t.timers
True

In both cases , The code returns the same empty class Dictionary .

Next to Python Add an optional name to the timer . This name can be used for two different purposes ：

1. In the code lookup Time passed

2. Add up Timer with the same name

Want to Python Timer add name , Need to be right timer.py Make changes . First ,Timer Accept name Parameters . second , When the timer stops , The runtime should be added to .timers in :

# timer.py
# ...
class Timer:
    timers = {}
    def __init__(
        self,
        name=None,
        text="Elapsed time: {:0.4f} seconds",
        logger=print,
    ):
        self._start_time = None
        self.name = name
        self.text = text
        self.logger = logger
        #  Add a new named timer to the timer Dictionary
        if name:
            self.timers.setdefault(name, 0)
    #  Other methods remain unchanged
    
    def stop(self):
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")
        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None
        if self.logger:
            self.logger(self.text.format(elapsed_time))
        if self.name:
            self.timers[self.name] += elapsed_time
        return elapsed_time

Be careful , In the .timers Add a new Python Timer time , Used .setdefault() Method . It is not defined in the dictionary name Set the value in the case of , If name Already in .timers Use in , Then the value will remain unchanged , At this time, several timers can be accumulated :

>>> from timer import Timer
>>> t = Timer("accumulate")
>>> t.start()
>>> t.stop()  # A few seconds later
Elapsed time: 3.7036 seconds
3.703554293999332
>>> t.start()
>>> t.stop()  # A few seconds later
Elapsed time: 2.3449 seconds
2.3448921170001995
>>> Timer.timers
{'accumulate': 6.0484464109995315}

You can now revisit download_data.py And ensure that only the time spent downloading data is measured ：

# download_data.py
import requests
from timer import Timer
def main():
    t = Timer("download", logger=None)
    source_url = 'https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/files/?p=%2Fall_six_datasets.zip&dl=1'
    headers = {'User-Agent': 'Mozilla/5.0'}
    for i in range(10):
        t.start()
        res = requests.get(source_url, headers=headers)
        t.stop()
        with open('dataset/datasets.zip', 'wb') as f:
            f.write(res.content)
    download_time = Timer.timers["download"]
    print(f"Downloaded 10 dataset in {download_time:0.2f} seconds")
    
if __name__=="__main__":
    main()

Now you have a very concise version ,Timer It is consistent 、 flexible 、 Convenient and informative ！ You can also apply many of the improvements made in this section to other types of classes in your project .

Timer improvement

The last improvement Timer, Make it more informative when used interactively . The following operation is to instantiate a timer class , And view its information ：

>>> from timer import Timer
>>> t = Timer()
>>> t
<timer.Timer object at 0x7f0578804320>

The last line is Python The default way to represent an object . The information we see from this result , Not very clear , We will improve it next .

Here is an introduction dataclasses class , This class is only contained in Python 3.7 And later .

pip install dataclasses

have access to @dataclass The decorator will Python The timer is converted to a data class

# timer.py
import time
from dataclasses import dataclass, field
from typing import Any, ClassVar
# ...
@dataclass
class Timer:
    timers: ClassVar = {}
    name: Any = None
    text: Any = "Elapsed time: {:0.4f} seconds"
    logger: Any = print
    _start_time: Any = field(default=None, init=False, repr=False)
    def __post_init__(self):
        """Initialization: add timer to dict of timers"""
        if self.name:
            self.timers.setdefault(self.name, 0)
    #  The rest of the code remains the same 

This code replaces the previous .__init__() Method . Notice how the data class uses a syntax similar to the class variable syntax you saw earlier for defining all variables . in fact ,.__init__() It is automatically created for the data class according to the annotation variables in the class definition .

If you need to annotate variables to use data classes . You can use this annotation to add type hints to your code . If you don't want to use type hints , Then you can use Any To annotate all variables . We will soon learn how to add actual type hints to our data classes .

The following is about Timer Some considerations for data classes ：

• The first 6 That's ok ：@dataclass The decorator will Timer Defined as a data class .

• The first 8 That's ok ： Data classes require special ClassVar Comment to specify .timers It's a class variable .

• The first 9 To 11 That's ok ：.name、.text and .logger Will be defined as Timer Properties on , You can create Timer Instance to specify its value . They all have given default values .

• The first 12 That's ok ： Think about it ._start_time It's a special property , For tracking Python The state of the timer , But it should be hidden from users . Use dataclasses.field(), ._start_time It should be from .__init__() and Timer Deleted from the representation of .

• In addition to setting instance properties , You can use special .__post_init__() Method to initialize . Use it here to add a named timer to .timers.

new Timer The data class uses the same functions as the previous regular class , But it now has a very good Information representation ：

from timer import Timer
t = Timer()
t

Timer(name=None,
text='Elapsed time: {:0.4f} seconds',
logger=<built-in function print>)

t.start()
#  After a few seconds
t.stop()

Elapsed time: 6.7197 seconds
6.719705373998295

summary

Now we have a very simple Timer edition , It is consistent 、 flexible 、 Convenient and informative ！ We can also apply many of the improvements made in this article to other types of classes in the project .

Now let's access the current full source code Timer. You'll notice that type hints have been added to the code to get additional documentation ：

# timer.py
from dataclasses import dataclass, field
import time
from typing import Callable, ClassVar, Dict, Optional
class TimerError(Exception):
    """A custom exception used to report errors in use of Timer class"""
@dataclass
class Timer:
    timers: ClassVar[Dict[str, float]] = {}
    name: Optional[str] = None
    text: str = "Elapsed time: {:0.4f} seconds"
    logger: Optional[Callable[[str], None]] = print
    _start_time: Optional[float] = field(default=None, init=False, repr=False)
    def __post_init__(self) -> None:
        """Add timer to dict of timers after initialization"""
        if self.name is not None:
            self.timers.setdefault(self.name, 0)
    def start(self) -> None:
        """Start a new timer"""
        if self._start_time is not None:
            raise TimerError(f"Timer is running. Use .stop() to stop it")
        self._start_time = time.perf_counter()
    def stop(self) -> float:
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")
        # Calculate elapsed time
        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None
        # Report elapsed time
        if self.logger:
            self.logger(self.text.format(elapsed_time))
        if self.name:
            self.timers[self.name] += elapsed_time
        return elapsed_time

Sum up ： Use class to create Python Timers have several advantages ：

• Readability ： Carefully select class and method names , Your code will read more naturally .

• Uniformity ： Encapsulate properties and behaviors into properties and methods , Your code will be easier to use .

• flexibility ： Use attributes that have default values instead of hard coded values , Your code will be reusable .

This class is very flexible , It can be used in almost any situation where you need to monitor code runtime . however , In the next section , We will learn how to use the context manager and decorator , This makes it easier to time code blocks and functions .

Reference material

[1]

time: https://docs.python.org/3/library/time.html

thread_time(): https://docs.python.org/3/library/time.html#time.thread_time

Looking back

Matplotlib Two methods of drawing torus ！

Python Common encryption algorithms in crawlers ！

2D Transformation 3D, Look at NVIDIA's AI“ new ” magic ！

Get it done Python Several common data structures ！

 Share
Point collection
A little bit of praise
Click to see