pyinstrument
pip install pyinstrument
使用参数
Usage: pyinstrument [options] scriptfile [arg] ...
Options:
--version show program's version number and exit
-h, --help show this help message and exit
--load-prev=ID Instead of running a script, load a previous report
-m MODULE_NAME run library module as a script, like 'python -m
module'
-o OUTFILE, --outfile=OUTFILE
save to <outfile>
-r RENDERER, --renderer=RENDERER
how the report should be rendered. One of: 'text',
'html', 'json', or python import path to a renderer
class
-t, --timeline render as a timeline - preserve ordering and don't
condense repeated calls
--hide=EXPR glob-style pattern matching the file paths whose
frames to hide. Defaults to '*/lib/*'.
--hide-regex=REGEX regex matching the file paths whose frames to hide.
Useful if --hide doesn't give enough control.
--show-all (text renderer only) show external library code
--unicode (text renderer only) force unicode text output
--no-unicode (text renderer only) force ascii text output
--color (text renderer only) force ansi color text output
--no-color (text renderer only) force no color text output
在代码中分析部分
from pyinstrument import Profiler
profiler = Profiler()
profiler.start()
# code you want to profile
profiler.stop()
print(profiler.output_text(unicode=True, color=True))
在Django中配置Web请求
settings.py 中 MIDDLEWARE_CLASSES 加入 pyinstrument.middleware.ProfilerMiddleware
在原来的请求api上加上 ?profile后缀
pyinstrument RECORDED: 2020/11/11 下午5:01:38 DURATION: 0.142 SECONDS
SAMPLES: 110 CPU TIME: 0.0781 SECONDS
0.142 __call__ django\utils\deprecation.py:87
0.142 inner django\core\handlers\exception.py:31
0.142 get test_pro1111\views.py:6
0.108 <module> requests\__init__.py:41
0.025 get requests\api.py:63
0.008 stat <built-in>:0
保存请求的分析html文件
在settings中 添加 PYINSTRUMENT_PROFILE_DIR = "profiles"
(每次请求会生成一个html分析文件在profiles文件夹中, 不需要?profile后缀)
在Flask中配置Web请求
from flask import Flask, g, make_response, request
app = Flask(__name__)
@app.before_request
def before_request():
if "profile" in request.args:
g.profiler = Profiler()
g.profiler.start()
@app.after_request
def after_request(response):
if not hasattr(g, "profiler"):
return response
g.profiler.stop()
output_html = g.profiler.output_html()
return make_response(output_html)
py-spy
pip install py-spy
Py-Spy的开销非常低：它是用Rust编写的，速度与编译的Python程序不在同一个进程中运行。
意味着Py-Spy可以安全地用于生成生产环境中的Python应用调优分析。
基本使用
py-spy 0.3.3
Sampling profiler for Python programs
USAGE:
py-spy <SUBCOMMAND>
OPTIONS:
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
record Records stack trace information to a flamegraph, speedscope or raw file
top Displays a top like view of functions consuming CPU
dump Dumps stack traces for a target program to stdout
help Prints this message or the help of the given subcommand(s)
py-spy top -r 500 -- python3 test.py
Collecting samples from '"python3" test.py' (python v3.6.6)
Total Samples 2500
GIL: 0.00%, Active: 109.20%, Threads: 1
%Own %Total OwnTime TotalTime Function (filename:line)
0.00% 0.00% 2.27s 2.27s __new__ (ssl.py:391)
0.00% 0.00% 0.468s 0.468s ssl_wrap_socket (urllib3\util\ssl_.py:321)
22.60% 23.00% 0.286s 0.292s close (urllib3\connectionpool.py:418)
9.80% 9.80% 0.212s 0.212s read (ssl.py:631)
0.00% 0.00% 0.170s 0.170s do_handshake (ssl.py:689)
14.20% 18.40% 0.162s 0.210s g (test.py:6)
3.40% 3.40% 0.098s 0.098s decompress (urllib3\response.py:77)
4.00% 4.00% 0.046s 0.046s text (requests\models.py:861)
2.40% 2.40% 0.032s 0.032s _worker (concurrent\futures\thread.py:71)
0.40% 15.60% 0.012s 0.364s content (requests\models.py:828)
0.60% 0.60% 0.012s 0.012s _real_close (ssl.py:1058)
0.60% 0.60% 0.010s 0.010s _real_close (socket.py:411)
0.00% 0.00% 0.010s 0.014s get (_collections_abc.py:660)
0.00% 0.00% 0.010s 0.010s _parsegen (email\feedparser.py:228)
0.00% 0.00% 0.010s 0.010s _coerce_args (urllib\parse.py:115)
0.00% 0.00% 0.008s 0.008s request_host (http\cookiejar.py:622)
0.00% 0.00% 0.008s 0.008s __new__ (ssl.py:393)
0.00% 0.00% 0.008s 0.046s _update_chunk_length (urllib3\response.py:598)
0.00% 0.00% 0.006s 0.008s quote (urllib\parse.py:789)
0.00% 0.00% 0.006s 0.006s set_ok_domain (http\cookiejar.py:1003)
py-spy dump --p 1500
Process 26068: D:\python37\python.exe C:/Users/PycharmProjects/oneway_test/test.py
Python v3.7.7 (D:\python37\python.exe)
Thread 10356 (idle): "MainThread"
_wait_for_tstate_lock (threading.py:1060)
join (threading.py:1044)
shutdown (concurrent\futures\thread.py:216)
__exit__ (concurrent\futures\_base.py:623)
<module> (test.py:13)
Thread 31092 (active): "ThreadPoolExecutor-0_0"
g (test.py:8)
run (concurrent\futures\thread.py:57)
_worker (concurrent\futures\thread.py:80)
run (threading.py:870)
_bootstrap_inner (threading.py:926)
_bootstrap (threading.py:890)
Thread 25524 (active): "ThreadPoolExecutor-0_1"
g (test.py:8)
run (concurrent\futures\thread.py:57)
_worker (concurrent\futures\thread.py:80)
run (threading.py:870)
_bootstrap_inner (threading.py:926)
_bootstrap (threading.py:890)
Thread 20668 (active): "ThreadPoolExecutor-0_2"
g (test.py:8)
run (concurrent\futures\thread.py:57)
_worker (concurrent\futures\thread.py:80)
run (threading.py:870)
_bootstrap_inner (threading.py:926)
_bootstrap (threading.py:890)
Thread 18092 (active): "ThreadPoolExecutor-0_3"
g (test.py:8)
run (concurrent\futures\thread.py:57)
_worker (concurrent\futures\thread.py:80)
run (threading.py:870)
_bootstrap_inner (threading.py:926)
_bootstrap (threading.py:890)
Thread 30036 (active): "ThreadPoolExecutor-0_4"
g (test.py:8)
run (concurrent\futures\thread.py:57)
_worker (concurrent\futures\thread.py:80)
run (threading.py:870)
_bootstrap_inner (threading.py:926)
_bootstrap (threading.py:890)
viztracer
usage: python -m viztracer [-h] [--tracer_entries [TRACER_ENTRIES]]
[--output_file [OUTPUT_FILE]]
[--output_dir [OUTPUT_DIR]] [--quiet]
[--max_stack_depth [MAX_STACK_DEPTH]]
[--exclude_files [EXCLUDE_FILES [EXCLUDE_FILES ...]]]
[--include_files [INCLUDE_FILES [INCLUDE_FILES ...]]]
[--ignore_c_function] [--ignore_non_file]
[--log_return_value] [--log_print] [--log_sparse]
[--log_function_args] [--log_gc]
[--log_var [LOG_VAR [LOG_VAR ...]]]
[--log_number [LOG_NUMBER [LOG_NUMBER ...]]]
[--log_attr [LOG_ATTR [LOG_ATTR ...]]]
[--log_func_exec [LOG_FUNC_EXEC [LOG_FUNC_EXEC ...]]]
[--log_func_entry [LOG_FUNC_ENTRY [LOG_FUNC_ENTRY ...]]]
[--log_exception] [--log_subprocess]
[--log_multiprocess] [--novdb] [--pid_suffix]
[--save_flamegraph]
[--generate_flamegraph [GENERATE_FLAMEGRAPH]]
[--module [MODULE]]
[--combine [COMBINE [COMBINE ...]]] [--open]
optional arguments:
-h, --help show this help message and exit
--tracer_entries [TRACER_ENTRIES]
size of circular buffer. How many entries can it store
--output_file [OUTPUT_FILE], -o [OUTPUT_FILE]
output file path. End with .json or .html or .gz
--output_dir [OUTPUT_DIR]
output directory. Should only be used when
--pid_suffix is used
--quiet stop VizTracer from printing anything
--max_stack_depth [MAX_STACK_DEPTH]
maximum stack depth you want to trace.
--exclude_files [EXCLUDE_FILES [EXCLUDE_FILES ...]]
specify the files(directories) you want to exclude
from tracing. Can't be used with --include_files
--include_files [INCLUDE_FILES [INCLUDE_FILES ...]]
specify the only files(directories) you want to
include from tracing. Can't be used with
--exclude_files
--ignore_c_function ignore all c functions including most builtin
functions and libraries
--ignore_non_file ignore all functions that are not in a vaild file(like
import)
--log_return_value log return value of the function in the report
--log_print replace all print() function to adding an event to the
result
--log_sparse log only selected functions with @log_sparse
--log_function_args log all function arguments, this will introduce large
overhead
--log_gc log ref cycle garbage collection operations
--log_var [LOG_VAR [LOG_VAR ...]]
log variable with specified names
--log_number [LOG_NUMBER [LOG_NUMBER ...]]
log variable with specified names as a number(using
VizCounter)
--log_attr [LOG_ATTR [LOG_ATTR ...]]
log attribute with specified names
--log_func_exec [LOG_FUNC_EXEC [LOG_FUNC_EXEC ...]]
log execution of function with specified names
--log_func_entry [LOG_FUNC_ENTRY [LOG_FUNC_ENTRY ...]]
log entry of the function with specified names
--log_exception log all exception when it's raised
--log_subprocess log subprocesses
--log_multiprocess log multiprocesses
--novdb Do not instrument for vdb, will reduce the overhead
--pid_suffix append pid to file name. This should be used when you
try to trace multi process programs. Will by default
generate json files
--save_flamegraph save flamegraph after generating the VizTracer report
--generate_flamegraph [GENERATE_FLAMEGRAPH]
generate a flamegraph from json VizTracer report.
Specify the json file to use
--module [MODULE], -m [MODULE]
run module with VizTracer
--combine [COMBINE [COMBINE ...]]
combine all json reports to a single report. Specify
all the json reports you want to combine
--open open the report in browser after saving
基本使用
viztracer my_script.py
# OR
python3 -m viztracer my_script.py
viztracer -o other_name.html my_script.py
viztracer -o other_name.json my_script.py
viztracer -o other_name.json.gz my_script.py
# 代码内
from viztracer import VizTracer
tracer = VizTracer()
tracer.run("import random;random.randrange(10)")
tracer.run("import random; random.randrange(10)", output_file="better_name.html")
tracer.start()
# Something happens here
tracer.stop()
tracer.save() # also takes output_file as an optional argument
with VizTracer(output_file="optional.html") as tracer:
# Something happens here
demo
import requests
from concurrent.futures import ThreadPoolExecutor
from viztracer import VizTracer
import time
tracer = VizTracer(max_stack_depth=2)
tracer.start()
def f():
n = 0
for i in range(1, 100):
for o in range(1, 20):
n += (i+0)
foo()
print(n)
def foo():
print("in")
f()
tracer.stop()
tracer.save()
默认情况 VizTracer 会trace python 和 C 函数.
# 关闭 c 函数的记录
viztracer --ignore_c_function my_script.py
tracer = VizTracer(ignore_c_function=True)
Django 使用
1 在需要追踪的函数加
@log_sparse装饰器
@log_sparse
def foo():
return 'foo'
2 需要给django一个save report 的entry入口
配置入口函数
添加一个url到这个save函数，通过浏览/save来保存report
from viztracer import get_tracer
def save(request):
tracer = get_tracer()
# 注意这里如果你不给output file，它会默认保存成json格式（因为是subprocess）
tracer.save(output_file="result.html")
return HttpResponse("done")
3 启动
viztracer --log_subprocess --log_sparse manage.py runserver
PySnooper
知道哪些行正在运行，以及局部变量的值是什么
# 重定向输出信息
pysnooper.snoop(output='xxx/debug.log')
# prefix 给观察的函数增加识别前缀
@pysnooper.snoop(prefix="funcTwo ")
def two(x, y):
z = x + y
return z
@pysnooper.snoop(prefix="funcOne ")
def one(number):
k = 0
while number:
k = two(k, number)
number -= 1
return number
# 参数watch可以用于查看一些非局部变量 d, class A
d = {
"one": 1,
"two": 1
}
class A:
t = 1
def foo(a):
return a
import pysnooper
with pysnooper.snoop(watch=("A.t", "d", "foo.a")): # watch_explode 可以展开字典
foo(d)
Source path:... C:/Users/PycharmProjects/oneway_test/demo.py
New var:....... __name__ = '__main__'
New var:....... __doc__ = None
New var:....... __package__ = None
New var:....... __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x000001B9FD94EB88>
New var:....... __spec__ = None
New var:....... __annotations__ = {}
New var:....... __builtins__ = <module 'builtins' (built-in)>
New var:....... __file__ = 'C:/Users/PycharmProjects/oneway_test/demo.py'
New var:....... __cached__ = None
New var:....... d = {'one': 1, 'two': 1}
New var:....... A = <class '__main__.A'>
New var:....... foo = <function foo at 0x000001B9FF4CD798>
New var:....... pysnooper = <module 'pysnooper' from 'D:\\virtual\\Envs\\smb...v37\\lib\\site-packages\\pysnooper\\__init__.py'>
New var:....... A.t = 1
18:03:03.277484 line 18 foo(d)
Elapsed time: 00:00:00.001001
Process finished with exit code 0
# depth显示函数中调用函数的snoop行，默认值为1，参数值需要大于或等于1。
pysnooper.snoop(depth=2)
cyberbrain
