1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
|
<span id="instrumentation"></span><h1>Instrumenting CPython with DTrace and SystemTap</h1> <dl class="field-list simple"> <dt class="field-odd">author</dt> <dd class="field-odd">
<p>David Malcolm</p> </dd> <dt class="field-even">author</dt> <dd class="field-even">
<p>Łukasz Langa</p> </dd> </dl> <p>DTrace and SystemTap are monitoring tools, each providing a way to inspect what the processes on a computer system are doing. They both use domain-specific languages allowing a user to write scripts which:</p> <ul class="simple"> <li>filter which processes are to be observed</li> <li>gather data from the processes of interest</li> <li>generate reports on the data</li> </ul> <p>As of Python 3.6, CPython can be built with embedded “markers”, also known as “probes”, that can be observed by a DTrace or SystemTap script, making it easier to monitor what the CPython processes on a system are doing.</p> <div class="impl-detail compound"> <p><strong>CPython implementation detail:</strong> DTrace markers are implementation details of the CPython interpreter. No guarantees are made about probe compatibility between versions of CPython. DTrace scripts can stop working or work incorrectly without warning when changing CPython versions.</p> </div> <section id="enabling-the-static-markers"> <h2>Enabling the static markers</h2> <p>macOS comes with built-in support for DTrace. On Linux, in order to build CPython with the embedded markers for SystemTap, the SystemTap development tools must be installed.</p> <p>On a Linux machine, this can be done via:</p> <pre data-language="shell">$ yum install systemtap-sdt-devel
</pre> <p>or:</p> <pre data-language="shell">$ sudo apt-get install systemtap-sdt-dev
</pre> <p>CPython must then be <a class="reference internal" href="../using/configure#cmdoption-with-dtrace"><code>configured with the --with-dtrace option</code></a>:</p> <pre data-language="none">checking for --with-dtrace... yes
</pre> <p>On macOS, you can list available DTrace probes by running a Python process in the background and listing all probes made available by the Python provider:</p> <pre data-language="shell">$ python3.6 -q &
$ sudo dtrace -l -P python$! # or: dtrace -l -m python3.6
ID PROVIDER MODULE FUNCTION NAME
29564 python18035 python3.6 _PyEval_EvalFrameDefault function-entry
29565 python18035 python3.6 dtrace_function_entry function-entry
29566 python18035 python3.6 _PyEval_EvalFrameDefault function-return
29567 python18035 python3.6 dtrace_function_return function-return
29568 python18035 python3.6 collect gc-done
29569 python18035 python3.6 collect gc-start
29570 python18035 python3.6 _PyEval_EvalFrameDefault line
29571 python18035 python3.6 maybe_dtrace_line line
</pre> <p>On Linux, you can verify if the SystemTap static markers are present in the built binary by seeing if it contains a “.note.stapsdt” section.</p> <pre data-language="shell">$ readelf -S ./python | grep .note.stapsdt
[30] .note.stapsdt NOTE 0000000000000000 00308d78
</pre> <p>If you’ve built Python as a shared library (with the <a class="reference internal" href="../using/configure#cmdoption-enable-shared"><code>--enable-shared</code></a> configure option), you need to look instead within the shared library. For example:</p> <pre data-language="shell">$ readelf -S libpython3.3dm.so.1.0 | grep .note.stapsdt
[29] .note.stapsdt NOTE 0000000000000000 00365b68
</pre> <p>Sufficiently modern readelf can print the metadata:</p> <pre data-language="shell">$ readelf -n ./python
Displaying notes found at file offset 0x00000254 with length 0x00000020:
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found at file offset 0x00000274 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: df924a2b08a7e89f6e11251d4602022977af2670
Displaying notes found at file offset 0x002d6c30 with length 0x00000144:
Owner Data size Description
stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__start
Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6
Arguments: -4@%ebx
stapsdt 0x00000030 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__done
Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8
Arguments: -8@%rax
stapsdt 0x00000045 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__entry
Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8
Arguments: 8@%rbp 8@%r12 -4@%eax
stapsdt 0x00000046 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__return
Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea
Arguments: 8@%rbp 8@%r12 -4@%eax
</pre> <p>The above metadata contains information for SystemTap describing how it can patch strategically placed machine code instructions to enable the tracing hooks used by a SystemTap script.</p> </section> <section id="static-dtrace-probes"> <h2>Static DTrace probes</h2> <p>The following example DTrace script can be used to show the call/return hierarchy of a Python script, only tracing within the invocation of a function called “start”. In other words, import-time function invocations are not going to be listed:</p> <pre data-language="none">self int indent;
python$target:::function-entry
/copyinstr(arg1) == "start"/
{
self->trace = 1;
}
python$target:::function-entry
/self->trace/
{
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
self->indent++;
}
python$target:::function-return
/self->trace/
{
self->indent--;
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
}
python$target:::function-return
/copyinstr(arg1) == "start"/
{
self->trace = 0;
}
</pre> <p>It can be invoked like this:</p> <pre data-language="shell">$ sudo dtrace -q -s call_stack.d -c "python3.6 script.py"
</pre> <p>The output looks like this:</p> <pre data-language="none">156641360502280 function-entry:call_stack.py:start:23
156641360518804 function-entry: call_stack.py:function_1:1
156641360532797 function-entry: call_stack.py:function_3:9
156641360546807 function-return: call_stack.py:function_3:10
156641360563367 function-return: call_stack.py:function_1:2
156641360578365 function-entry: call_stack.py:function_2:5
156641360591757 function-entry: call_stack.py:function_1:1
156641360605556 function-entry: call_stack.py:function_3:9
156641360617482 function-return: call_stack.py:function_3:10
156641360629814 function-return: call_stack.py:function_1:2
156641360642285 function-return: call_stack.py:function_2:6
156641360656770 function-entry: call_stack.py:function_3:9
156641360669707 function-return: call_stack.py:function_3:10
156641360687853 function-entry: call_stack.py:function_4:13
156641360700719 function-return: call_stack.py:function_4:14
156641360719640 function-entry: call_stack.py:function_5:18
156641360732567 function-return: call_stack.py:function_5:21
156641360747370 function-return:call_stack.py:start:28
</pre> </section> <section id="static-systemtap-markers"> <h2>Static SystemTap markers</h2> <p>The low-level way to use the SystemTap integration is to use the static markers directly. This requires you to explicitly state the binary file containing them.</p> <p>For example, this SystemTap script can be used to show the call/return hierarchy of a Python script:</p> <pre data-language="none">probe process("python").mark("function__entry") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
printf("%s => %s in %s:%d\\n",
thread_indent(1), funcname, filename, lineno);
}
probe process("python").mark("function__return") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
printf("%s <= %s in %s:%d\\n",
thread_indent(-1), funcname, filename, lineno);
}
</pre> <p>It can be invoked like this:</p> <pre data-language="shell">$ stap \
show-call-hierarchy.stp \
-c "./python test.py"
</pre> <p>The output looks like this:</p> <pre data-language="none">11408 python(8274): => __contains__ in Lib/_abcoll.py:362
11414 python(8274): => __getitem__ in Lib/os.py:425
11418 python(8274): => encode in Lib/os.py:490
11424 python(8274): <= encode in Lib/os.py:493
11428 python(8274): <= __getitem__ in Lib/os.py:426
11433 python(8274): <= __contains__ in Lib/_abcoll.py:366
</pre> <p>where the columns are:</p> <ul class="simple"> <li>time in microseconds since start of script</li> <li>name of executable</li> <li>PID of process</li> </ul> <p>and the remainder indicates the call/return hierarchy as the script executes.</p> <p>For a <a class="reference internal" href="../using/configure#cmdoption-enable-shared"><code>--enable-shared</code></a> build of CPython, the markers are contained within the libpython shared library, and the probe’s dotted path needs to reflect this. For example, this line from the above example:</p> <pre data-language="none">probe process("python").mark("function__entry") {
</pre> <p>should instead read:</p> <pre data-language="none">probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") {
</pre> <p>(assuming a <a class="reference internal" href="../using/configure#debug-build"><span class="std std-ref">debug build</span></a> of CPython 3.6)</p> </section> <section id="available-static-markers"> <h2>Available static markers</h2> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">function__entry(str filename, str funcname, int lineno)</span>
</dt> <dd>
<p>This marker indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions.</p> <p>The filename, function name, and line number are provided back to the tracing script as positional arguments, which must be accessed using <code>$arg1</code>, <code>$arg2</code>, <code>$arg3</code>:</p> <ul class="simple"> <li>
<code>$arg1</code> : <code>(const char *)</code> filename, accessible using <code>user_string($arg1)</code>
</li> <li>
<code>$arg2</code> : <code>(const char *)</code> function name, accessible using <code>user_string($arg2)</code>
</li> <li>
<code>$arg3</code> : <code>int</code> line number</li> </ul> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">function__return(str filename, str funcname, int lineno)</span>
</dt> <dd>
<p>This marker is the converse of <code>function__entry()</code>, and indicates that execution of a Python function has ended (either via <code>return</code>, or via an exception). It is only triggered for pure-Python (bytecode) functions.</p> <p>The arguments are the same as for <code>function__entry()</code></p> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">line(str filename, str funcname, int lineno)</span>
</dt> <dd>
<p>This marker indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. It is not triggered within C functions.</p> <p>The arguments are the same as for <code>function__entry()</code>.</p> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">gc__start(int generation)</span>
</dt> <dd>
<p>Fires when the Python interpreter starts a garbage collection cycle. <code>arg0</code> is the generation to scan, like <a class="reference internal" href="../library/gc#gc.collect" title="gc.collect"><code>gc.collect()</code></a>.</p> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">gc__done(long collected)</span>
</dt> <dd>
<p>Fires when the Python interpreter finishes a garbage collection cycle. <code>arg0</code> is the number of collected objects.</p> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">import__find__load__start(str modulename)</span>
</dt> <dd>
<p>Fires before <a class="reference internal" href="../library/importlib#module-importlib" title="importlib: The implementation of the import machinery."><code>importlib</code></a> attempts to find and load the module. <code>arg0</code> is the module name.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.7.</span></p> </div> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">import__find__load__done(str modulename, int found)</span>
</dt> <dd>
<p>Fires after <a class="reference internal" href="../library/importlib#module-importlib" title="importlib: The implementation of the import machinery."><code>importlib</code></a>’s find_and_load function is called. <code>arg0</code> is the module name, <code>arg1</code> indicates if module was successfully loaded.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.7.</span></p> </div> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">audit(str event, void *tuple)</span>
</dt> <dd>
<p>Fires when <a class="reference internal" href="../library/sys#sys.audit" title="sys.audit"><code>sys.audit()</code></a> or <a class="reference internal" href="../c-api/sys#c.PySys_Audit" title="PySys_Audit"><code>PySys_Audit()</code></a> is called. <code>arg0</code> is the event name as C string, <code>arg1</code> is a <a class="reference internal" href="../c-api/structures#c.PyObject" title="PyObject"><code>PyObject</code></a> pointer to a tuple object.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> </dd>
</dl> </section> <section id="systemtap-tapsets"> <h2>SystemTap Tapsets</h2> <p>The higher-level way to use the SystemTap integration is to use a “tapset”: SystemTap’s equivalent of a library, which hides some of the lower-level details of the static markers.</p> <p>Here is a tapset file, based on a non-shared build of CPython:</p> <pre data-language="none">/*
Provide a higher-level wrapping around the function__entry and
function__return markers:
\*/
probe python.function.entry = process("python").mark("function__entry")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
probe python.function.return = process("python").mark("function__return")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
</pre> <p>If this file is installed in SystemTap’s tapset directory (e.g. <code>/usr/share/systemtap/tapset</code>), then these additional probepoints become available:</p> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">python.function.entry(str filename, str funcname, int lineno, frameptr)</span>
</dt> <dd>
<p>This probe point indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions.</p> </dd>
</dl> <dl class="object"> <dt class="sig sig-object"> <span class="sig-name descname">python.function.return(str filename, str funcname, int lineno, frameptr)</span>
</dt> <dd>
<p>This probe point is the converse of <code>python.function.return</code>, and indicates that execution of a Python function has ended (either via <code>return</code>, or via an exception). It is only triggered for pure-Python (bytecode) functions.</p> </dd>
</dl> </section> <section id="examples"> <h2>Examples</h2> <p>This SystemTap script uses the tapset above to more cleanly implement the example given above of tracing the Python function-call hierarchy, without needing to directly name the static markers:</p> <pre data-language="none">probe python.function.entry
{
printf("%s => %s in %s:%d\n",
thread_indent(1), funcname, filename, lineno);
}
probe python.function.return
{
printf("%s <= %s in %s:%d\n",
thread_indent(-1), funcname, filename, lineno);
}
</pre> <p>The following script uses the tapset above to provide a top-like view of all running CPython code, showing the top 20 most frequently entered bytecode frames, each second, across the whole system:</p> <pre data-language="none">global fn_calls;
probe python.function.entry
{
fn_calls[pid(), filename, funcname, lineno] += 1;
}
probe timer.ms(1000) {
printf("\033[2J\033[1;1H") /* clear screen \*/
printf("%6s %80s %6s %30s %6s\n",
"PID", "FILENAME", "LINE", "FUNCTION", "CALLS")
foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) {
printf("%6d %80s %6d %30s %6d\n",
pid, filename, lineno, funcname,
fn_calls[pid, filename, funcname, lineno]);
}
delete fn_calls;
}
</pre> </section> <div class="_attribution">
<p class="_attribution-p">
© 2001–2023 Python Software Foundation<br>Licensed under the PSF License.<br>
<a href="https://docs.python.org/3.12/howto/instrumentation.html" class="_attribution-link">https://docs.python.org/3.12/howto/instrumentation.html</a>
</p>
</div>
|
