diff options
Diffstat (limited to 'devdocs/python~3.12/library%2Fstatistics.html')
| -rw-r--r-- | devdocs/python~3.12/library%2Fstatistics.html | 360 |
1 files changed, 360 insertions, 0 deletions
diff --git a/devdocs/python~3.12/library%2Fstatistics.html b/devdocs/python~3.12/library%2Fstatistics.html new file mode 100644 index 00000000..53e9c9a5 --- /dev/null +++ b/devdocs/python~3.12/library%2Fstatistics.html @@ -0,0 +1,360 @@ + <span id="statistics-mathematical-statistics-functions"></span><h1>statistics — Mathematical statistics functions</h1> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.4.</span></p> </div> <p><strong>Source code:</strong> <a class="reference external" href="https://github.com/python/cpython/tree/3.12/Lib/statistics.py">Lib/statistics.py</a></p> <p>This module provides functions for calculating mathematical statistics of numeric (<a class="reference internal" href="numbers#numbers.Real" title="numbers.Real"><code>Real</code></a>-valued) data.</p> <p>The module is not intended to be a competitor to third-party libraries such as <a class="reference external" href="https://numpy.org">NumPy</a>, <a class="reference external" href="https://scipy.org/">SciPy</a>, or proprietary full-featured statistics packages aimed at professional statisticians such as Minitab, SAS and Matlab. It is aimed at the level of graphing and scientific calculators.</p> <p>Unless explicitly noted, these functions support <a class="reference internal" href="functions#int" title="int"><code>int</code></a>, <a class="reference internal" href="functions#float" title="float"><code>float</code></a>, <a class="reference internal" href="decimal#decimal.Decimal" title="decimal.Decimal"><code>Decimal</code></a> and <a class="reference internal" href="fractions#fractions.Fraction" title="fractions.Fraction"><code>Fraction</code></a>. Behaviour with other types (whether in the numeric tower or not) is currently unsupported. Collections with a mix of types are also undefined and implementation-dependent. If your input data consists of mixed types, you may be able to use <a class="reference internal" href="functions#map" title="map"><code>map()</code></a> to ensure a consistent result, for example: <code>map(float, input_data)</code>.</p> <p>Some datasets use <code>NaN</code> (not a number) values to represent missing data. Since NaNs have unusual comparison semantics, they cause surprising or undefined behaviors in the statistics functions that sort data or that count occurrences. The functions affected are <code>median()</code>, <code>median_low()</code>, <code>median_high()</code>, <code>median_grouped()</code>, <code>mode()</code>, <code>multimode()</code>, and <code>quantiles()</code>. The <code>NaN</code> values should be stripped before calling these functions:</p> <pre data-language="python">>>> from statistics import median +>>> from math import isnan +>>> from itertools import filterfalse + +>>> data = [20.7, float('NaN'),19.2, 18.3, float('NaN'), 14.4] +>>> sorted(data) # This has surprising behavior +[20.7, nan, 14.4, 18.3, 19.2, nan] +>>> median(data) # This result is unexpected +16.35 + +>>> sum(map(isnan, data)) # Number of missing values +2 +>>> clean = list(filterfalse(isnan, data)) # Strip NaN values +>>> clean +[20.7, 19.2, 18.3, 14.4] +>>> sorted(clean) # Sorting now works as expected +[14.4, 18.3, 19.2, 20.7] +>>> median(clean) # This result is now well defined +18.75 +</pre> <section id="averages-and-measures-of-central-location"> <h2>Averages and measures of central location</h2> <p>These functions calculate an average or typical value from a population or sample.</p> <table class="docutils align-default"> <tr> +<td><p><a class="reference internal" href="#statistics.mean" title="statistics.mean"><code>mean()</code></a></p></td> <td><p>Arithmetic mean (“average”) of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.fmean" title="statistics.fmean"><code>fmean()</code></a></p></td> <td><p>Fast, floating point arithmetic mean, with optional weighting.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.geometric_mean" title="statistics.geometric_mean"><code>geometric_mean()</code></a></p></td> <td><p>Geometric mean of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.harmonic_mean" title="statistics.harmonic_mean"><code>harmonic_mean()</code></a></p></td> <td><p>Harmonic mean of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.median" title="statistics.median"><code>median()</code></a></p></td> <td><p>Median (middle value) of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.median_low" title="statistics.median_low"><code>median_low()</code></a></p></td> <td><p>Low median of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.median_high" title="statistics.median_high"><code>median_high()</code></a></p></td> <td><p>High median of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.median_grouped" title="statistics.median_grouped"><code>median_grouped()</code></a></p></td> <td><p>Median, or 50th percentile, of grouped data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.mode" title="statistics.mode"><code>mode()</code></a></p></td> <td><p>Single mode (most common value) of discrete or nominal data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.multimode" title="statistics.multimode"><code>multimode()</code></a></p></td> <td><p>List of modes (most common values) of discrete or nominal data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.quantiles" title="statistics.quantiles"><code>quantiles()</code></a></p></td> <td><p>Divide data into intervals with equal probability.</p></td> </tr> </table> </section> <section id="measures-of-spread"> <h2>Measures of spread</h2> <p>These functions calculate a measure of how much the population or sample tends to deviate from the typical or average values.</p> <table class="docutils align-default"> <tr> +<td><p><a class="reference internal" href="#statistics.pstdev" title="statistics.pstdev"><code>pstdev()</code></a></p></td> <td><p>Population standard deviation of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.pvariance" title="statistics.pvariance"><code>pvariance()</code></a></p></td> <td><p>Population variance of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.stdev" title="statistics.stdev"><code>stdev()</code></a></p></td> <td><p>Sample standard deviation of data.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.variance" title="statistics.variance"><code>variance()</code></a></p></td> <td><p>Sample variance of data.</p></td> </tr> </table> </section> <section id="statistics-for-relations-between-two-inputs"> <h2>Statistics for relations between two inputs</h2> <p>These functions calculate statistics regarding relations between two inputs.</p> <table class="docutils align-default"> <tr> +<td><p><a class="reference internal" href="#statistics.covariance" title="statistics.covariance"><code>covariance()</code></a></p></td> <td><p>Sample covariance for two variables.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.correlation" title="statistics.correlation"><code>correlation()</code></a></p></td> <td><p>Pearson and Spearman’s correlation coefficients.</p></td> </tr> <tr> +<td><p><a class="reference internal" href="#statistics.linear_regression" title="statistics.linear_regression"><code>linear_regression()</code></a></p></td> <td><p>Slope and intercept for simple linear regression.</p></td> </tr> </table> </section> <section id="function-details"> <h2>Function details</h2> <p>Note: The functions do not require the data given to them to be sorted. However, for reading convenience, most of the examples show sorted sequences.</p> <dl class="py function"> <dt class="sig sig-object py" id="statistics.mean"> +<code>statistics.mean(data)</code> </dt> <dd> +<p>Return the sample arithmetic mean of <em>data</em> which can be a sequence or iterable.</p> <p>The arithmetic mean is the sum of the data divided by the number of data points. It is commonly called “the average”, although it is only one of many different mathematical averages. It is a measure of the central location of the data.</p> <p>If <em>data</em> is empty, <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> will be raised.</p> <p>Some examples of use:</p> <pre data-language="pycon3">>>> mean([1, 2, 3, 4, 4]) +2.8 +>>> mean([-1.0, 2.5, 3.25, 5.75]) +2.625 + +>>> from fractions import Fraction as F +>>> mean([F(3, 7), F(1, 21), F(5, 3), F(1, 3)]) +Fraction(13, 21) + +>>> from decimal import Decimal as D +>>> mean([D("0.5"), D("0.75"), D("0.625"), D("0.375")]) +Decimal('0.5625') +</pre> <div class="admonition note"> <p class="admonition-title">Note</p> <p>The mean is strongly affected by <a class="reference external" href="https://en.wikipedia.org/wiki/Outlier">outliers</a> and is not necessarily a typical example of the data points. For a more robust, although less efficient, measure of <a class="reference external" href="https://en.wikipedia.org/wiki/Central_tendency">central tendency</a>, see <a class="reference internal" href="#statistics.median" title="statistics.median"><code>median()</code></a>.</p> <p>The sample mean gives an unbiased estimate of the true population mean, so that when taken on average over all the possible samples, <code>mean(sample)</code> converges on the true mean of the entire population. If <em>data</em> represents the entire population rather than a sample, then <code>mean(data)</code> is equivalent to calculating the true population mean μ.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.fmean"> +<code>statistics.fmean(data, weights=None)</code> </dt> <dd> +<p>Convert <em>data</em> to floats and compute the arithmetic mean.</p> <p>This runs faster than the <a class="reference internal" href="#statistics.mean" title="statistics.mean"><code>mean()</code></a> function and it always returns a <a class="reference internal" href="functions#float" title="float"><code>float</code></a>. The <em>data</em> may be a sequence or iterable. If the input dataset is empty, raises a <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a>.</p> <pre data-language="pycon3">>>> fmean([3.5, 4.0, 5.25]) +4.25 +</pre> <p>Optional weighting is supported. For example, a professor assigns a grade for a course by weighting quizzes at 20%, homework at 20%, a midterm exam at 30%, and a final exam at 30%:</p> <pre data-language="pycon3">>>> grades = [85, 92, 83, 91] +>>> weights = [0.20, 0.20, 0.30, 0.30] +>>> fmean(grades, weights) +87.6 +</pre> <p>If <em>weights</em> is supplied, it must be the same length as the <em>data</em> or a <a class="reference internal" href="exceptions#ValueError" title="ValueError"><code>ValueError</code></a> will be raised.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.11: </span>Added support for <em>weights</em>.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.geometric_mean"> +<code>statistics.geometric_mean(data)</code> </dt> <dd> +<p>Convert <em>data</em> to floats and compute the geometric mean.</p> <p>The geometric mean indicates the central tendency or typical value of the <em>data</em> using the product of the values (as opposed to the arithmetic mean which uses their sum).</p> <p>Raises a <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> if the input dataset is empty, if it contains a zero, or if it contains a negative value. The <em>data</em> may be a sequence or iterable.</p> <p>No special efforts are made to achieve exact results. (However, this may change in the future.)</p> <pre data-language="pycon3">>>> round(geometric_mean([54, 24, 36]), 1) +36.0 +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.harmonic_mean"> +<code>statistics.harmonic_mean(data, weights=None)</code> </dt> <dd> +<p>Return the harmonic mean of <em>data</em>, a sequence or iterable of real-valued numbers. If <em>weights</em> is omitted or <em>None</em>, then equal weighting is assumed.</p> <p>The harmonic mean is the reciprocal of the arithmetic <a class="reference internal" href="#statistics.mean" title="statistics.mean"><code>mean()</code></a> of the reciprocals of the data. For example, the harmonic mean of three values <em>a</em>, <em>b</em> and <em>c</em> will be equivalent to <code>3/(1/a + 1/b + 1/c)</code>. If one of the values is zero, the result will be zero.</p> <p>The harmonic mean is a type of average, a measure of the central location of the data. It is often appropriate when averaging ratios or rates, for example speeds.</p> <p>Suppose a car travels 10 km at 40 km/hr, then another 10 km at 60 km/hr. What is the average speed?</p> <pre data-language="pycon3">>>> harmonic_mean([40, 60]) +48.0 +</pre> <p>Suppose a car travels 40 km/hr for 5 km, and when traffic clears, speeds-up to 60 km/hr for the remaining 30 km of the journey. What is the average speed?</p> <pre data-language="pycon3">>>> harmonic_mean([40, 60], weights=[5, 30]) +56.0 +</pre> <p><a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised if <em>data</em> is empty, any element is less than zero, or if the weighted sum isn’t positive.</p> <p>The current algorithm has an early-out when it encounters a zero in the input. This means that the subsequent inputs are not tested for validity. (This behavior may change in the future.)</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.6.</span></p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.10: </span>Added support for <em>weights</em>.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.median"> +<code>statistics.median(data)</code> </dt> <dd> +<p>Return the median (middle value) of numeric data, using the common “mean of middle two” method. If <em>data</em> is empty, <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised. <em>data</em> can be a sequence or iterable.</p> <p>The median is a robust measure of central location and is less affected by the presence of outliers. When the number of data points is odd, the middle data point is returned:</p> <pre data-language="pycon3">>>> median([1, 3, 5]) +3 +</pre> <p>When the number of data points is even, the median is interpolated by taking the average of the two middle values:</p> <pre data-language="pycon3">>>> median([1, 3, 5, 7]) +4.0 +</pre> <p>This is suited for when your data is discrete, and you don’t mind that the median may not be an actual data point.</p> <p>If the data is ordinal (supports order operations) but not numeric (doesn’t support addition), consider using <a class="reference internal" href="#statistics.median_low" title="statistics.median_low"><code>median_low()</code></a> or <a class="reference internal" href="#statistics.median_high" title="statistics.median_high"><code>median_high()</code></a> instead.</p> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.median_low"> +<code>statistics.median_low(data)</code> </dt> <dd> +<p>Return the low median of numeric data. If <em>data</em> is empty, <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised. <em>data</em> can be a sequence or iterable.</p> <p>The low median is always a member of the data set. When the number of data points is odd, the middle value is returned. When it is even, the smaller of the two middle values is returned.</p> <pre data-language="pycon3">>>> median_low([1, 3, 5]) +3 +>>> median_low([1, 3, 5, 7]) +3 +</pre> <p>Use the low median when your data are discrete and you prefer the median to be an actual data point rather than interpolated.</p> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.median_high"> +<code>statistics.median_high(data)</code> </dt> <dd> +<p>Return the high median of data. If <em>data</em> is empty, <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised. <em>data</em> can be a sequence or iterable.</p> <p>The high median is always a member of the data set. When the number of data points is odd, the middle value is returned. When it is even, the larger of the two middle values is returned.</p> <pre data-language="pycon3">>>> median_high([1, 3, 5]) +3 +>>> median_high([1, 3, 5, 7]) +5 +</pre> <p>Use the high median when your data are discrete and you prefer the median to be an actual data point rather than interpolated.</p> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.median_grouped"> +<code>statistics.median_grouped(data, interval=1)</code> </dt> <dd> +<p>Return the median of grouped continuous data, calculated as the 50th percentile, using interpolation. If <em>data</em> is empty, <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised. <em>data</em> can be a sequence or iterable.</p> <pre data-language="pycon3">>>> median_grouped([52, 52, 53, 54]) +52.5 +</pre> <p>In the following example, the data are rounded, so that each value represents the midpoint of data classes, e.g. 1 is the midpoint of the class 0.5–1.5, 2 is the midpoint of 1.5–2.5, 3 is the midpoint of 2.5–3.5, etc. With the data given, the middle value falls somewhere in the class 3.5–4.5, and interpolation is used to estimate it:</p> <pre data-language="pycon3">>>> median_grouped([1, 2, 2, 3, 4, 4, 4, 4, 4, 5]) +3.7 +</pre> <p>Optional argument <em>interval</em> represents the class interval, and defaults to 1. Changing the class interval naturally will change the interpolation:</p> <pre data-language="pycon3">>>> median_grouped([1, 3, 3, 5, 7], interval=1) +3.25 +>>> median_grouped([1, 3, 3, 5, 7], interval=2) +3.5 +</pre> <p>This function does not check whether the data points are at least <em>interval</em> apart.</p> <div class="impl-detail compound"> <p><strong>CPython implementation detail:</strong> Under some circumstances, <a class="reference internal" href="#statistics.median_grouped" title="statistics.median_grouped"><code>median_grouped()</code></a> may coerce data points to floats. This behaviour is likely to change in the future.</p> </div> <div class="admonition seealso"> <p class="admonition-title">See also</p> <ul class="simple"> <li>“Statistics for the Behavioral Sciences”, Frederick J Gravetter and Larry B Wallnau (8th Edition).</li> <li>The <a class="reference external" href="https://help.gnome.org/users/gnumeric/stable/gnumeric.html#gnumeric-function-SSMEDIAN">SSMEDIAN</a> function in the Gnome Gnumeric spreadsheet, including <a class="reference external" href="https://mail.gnome.org/archives/gnumeric-list/2011-April/msg00018.html">this discussion</a>.</li> </ul> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.mode"> +<code>statistics.mode(data)</code> </dt> <dd> +<p>Return the single most common data point from discrete or nominal <em>data</em>. The mode (when it exists) is the most typical value and serves as a measure of central location.</p> <p>If there are multiple modes with the same frequency, returns the first one encountered in the <em>data</em>. If the smallest or largest of those is desired instead, use <code>min(multimode(data))</code> or <code>max(multimode(data))</code>. If the input <em>data</em> is empty, <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised.</p> <p><code>mode</code> assumes discrete data and returns a single value. This is the standard treatment of the mode as commonly taught in schools:</p> <pre data-language="pycon3">>>> mode([1, 1, 2, 3, 3, 3, 3, 4]) +3 +</pre> <p>The mode is unique in that it is the only statistic in this package that also applies to nominal (non-numeric) data:</p> <pre data-language="pycon3">>>> mode(["red", "blue", "blue", "red", "green", "red", "red"]) +'red' +</pre> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>Now handles multimodal datasets by returning the first mode encountered. Formerly, it raised <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> when more than one mode was found.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.multimode"> +<code>statistics.multimode(data)</code> </dt> <dd> +<p>Return a list of the most frequently occurring values in the order they were first encountered in the <em>data</em>. Will return more than one result if there are multiple modes or an empty list if the <em>data</em> is empty:</p> <pre data-language="pycon3">>>> multimode('aabbbbccddddeeffffgg') +['b', 'd', 'f'] +>>> multimode('') +[] +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.pstdev"> +<code>statistics.pstdev(data, mu=None)</code> </dt> <dd> +<p>Return the population standard deviation (the square root of the population variance). See <a class="reference internal" href="#statistics.pvariance" title="statistics.pvariance"><code>pvariance()</code></a> for arguments and other details.</p> <pre data-language="pycon3">>>> pstdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75]) +0.986893273527251 +</pre> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.pvariance"> +<code>statistics.pvariance(data, mu=None)</code> </dt> <dd> +<p>Return the population variance of <em>data</em>, a non-empty sequence or iterable of real-valued numbers. Variance, or second moment about the mean, is a measure of the variability (spread or dispersion) of data. A large variance indicates that the data is spread out; a small variance indicates it is clustered closely around the mean.</p> <p>If the optional second argument <em>mu</em> is given, it is typically the mean of the <em>data</em>. It can also be used to compute the second moment around a point that is not the mean. If it is missing or <code>None</code> (the default), the arithmetic mean is automatically calculated.</p> <p>Use this function to calculate the variance from the entire population. To estimate the variance from a sample, the <a class="reference internal" href="#statistics.variance" title="statistics.variance"><code>variance()</code></a> function is usually a better choice.</p> <p>Raises <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> if <em>data</em> is empty.</p> <p>Examples:</p> <pre data-language="pycon3">>>> data = [0.0, 0.25, 0.25, 1.25, 1.5, 1.75, 2.75, 3.25] +>>> pvariance(data) +1.25 +</pre> <p>If you have already calculated the mean of your data, you can pass it as the optional second argument <em>mu</em> to avoid recalculation:</p> <pre data-language="pycon3">>>> mu = mean(data) +>>> pvariance(data, mu) +1.25 +</pre> <p>Decimals and Fractions are supported:</p> <pre data-language="pycon3">>>> from decimal import Decimal as D +>>> pvariance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")]) +Decimal('24.815') + +>>> from fractions import Fraction as F +>>> pvariance([F(1, 4), F(5, 4), F(1, 2)]) +Fraction(13, 72) +</pre> <div class="admonition note"> <p class="admonition-title">Note</p> <p>When called with the entire population, this gives the population variance σ². When called on a sample instead, this is the biased sample variance s², also known as variance with N degrees of freedom.</p> <p>If you somehow know the true population mean μ, you may use this function to calculate the variance of a sample, giving the known population mean as the second argument. Provided the data points are a random sample of the population, the result will be an unbiased estimate of the population variance.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.stdev"> +<code>statistics.stdev(data, xbar=None)</code> </dt> <dd> +<p>Return the sample standard deviation (the square root of the sample variance). See <a class="reference internal" href="#statistics.variance" title="statistics.variance"><code>variance()</code></a> for arguments and other details.</p> <pre data-language="pycon3">>>> stdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75]) +1.0810874155219827 +</pre> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.variance"> +<code>statistics.variance(data, xbar=None)</code> </dt> <dd> +<p>Return the sample variance of <em>data</em>, an iterable of at least two real-valued numbers. Variance, or second moment about the mean, is a measure of the variability (spread or dispersion) of data. A large variance indicates that the data is spread out; a small variance indicates it is clustered closely around the mean.</p> <p>If the optional second argument <em>xbar</em> is given, it should be the mean of <em>data</em>. If it is missing or <code>None</code> (the default), the mean is automatically calculated.</p> <p>Use this function when your data is a sample from a population. To calculate the variance from the entire population, see <a class="reference internal" href="#statistics.pvariance" title="statistics.pvariance"><code>pvariance()</code></a>.</p> <p>Raises <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> if <em>data</em> has fewer than two values.</p> <p>Examples:</p> <pre data-language="pycon3">>>> data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5] +>>> variance(data) +1.3720238095238095 +</pre> <p>If you have already calculated the mean of your data, you can pass it as the optional second argument <em>xbar</em> to avoid recalculation:</p> <pre data-language="pycon3">>>> m = mean(data) +>>> variance(data, m) +1.3720238095238095 +</pre> <p>This function does not attempt to verify that you have passed the actual mean as <em>xbar</em>. Using arbitrary values for <em>xbar</em> can lead to invalid or impossible results.</p> <p>Decimal and Fraction values are supported:</p> <pre data-language="pycon3">>>> from decimal import Decimal as D +>>> variance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")]) +Decimal('31.01875') + +>>> from fractions import Fraction as F +>>> variance([F(1, 6), F(1, 2), F(5, 3)]) +Fraction(67, 108) +</pre> <div class="admonition note"> <p class="admonition-title">Note</p> <p>This is the sample variance s² with Bessel’s correction, also known as variance with N-1 degrees of freedom. Provided that the data points are representative (e.g. independent and identically distributed), the result should be an unbiased estimate of the true population variance.</p> <p>If you somehow know the actual population mean μ you should pass it to the <a class="reference internal" href="#statistics.pvariance" title="statistics.pvariance"><code>pvariance()</code></a> function as the <em>mu</em> parameter to get the variance of a sample.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.quantiles"> +<code>statistics.quantiles(data, *, n=4, method='exclusive')</code> </dt> <dd> +<p>Divide <em>data</em> into <em>n</em> continuous intervals with equal probability. Returns a list of <code>n - 1</code> cut points separating the intervals.</p> <p>Set <em>n</em> to 4 for quartiles (the default). Set <em>n</em> to 10 for deciles. Set <em>n</em> to 100 for percentiles which gives the 99 cuts points that separate <em>data</em> into 100 equal sized groups. Raises <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> if <em>n</em> is not least 1.</p> <p>The <em>data</em> can be any iterable containing sample data. For meaningful results, the number of data points in <em>data</em> should be larger than <em>n</em>. Raises <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> if there are not at least two data points.</p> <p>The cut points are linearly interpolated from the two nearest data points. For example, if a cut point falls one-third of the distance between two sample values, <code>100</code> and <code>112</code>, the cut-point will evaluate to <code>104</code>.</p> <p>The <em>method</em> for computing quantiles can be varied depending on whether the <em>data</em> includes or excludes the lowest and highest possible values from the population.</p> <p>The default <em>method</em> is “exclusive” and is used for data sampled from a population that can have more extreme values than found in the samples. The portion of the population falling below the <em>i-th</em> of <em>m</em> sorted data points is computed as <code>i / (m + 1)</code>. Given nine sample values, the method sorts them and assigns the following percentiles: 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%.</p> <p>Setting the <em>method</em> to “inclusive” is used for describing population data or for samples that are known to include the most extreme values from the population. The minimum value in <em>data</em> is treated as the 0th percentile and the maximum value is treated as the 100th percentile. The portion of the population falling below the <em>i-th</em> of <em>m</em> sorted data points is computed as <code>(i - 1) / (m - 1)</code>. Given 11 sample values, the method sorts them and assigns the following percentiles: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%.</p> <pre data-language="pycon3"># Decile cut points for empirically sampled data +>>> data = [105, 129, 87, 86, 111, 111, 89, 81, 108, 92, 110, +... 100, 75, 105, 103, 109, 76, 119, 99, 91, 103, 129, +... 106, 101, 84, 111, 74, 87, 86, 103, 103, 106, 86, +... 111, 75, 87, 102, 121, 111, 88, 89, 101, 106, 95, +... 103, 107, 101, 81, 109, 104] +>>> [round(q, 1) for q in quantiles(data, n=10)] +[81.0, 86.2, 89.0, 99.4, 102.5, 103.6, 106.0, 109.8, 111.0] +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.covariance"> +<code>statistics.covariance(x, y, /)</code> </dt> <dd> +<p>Return the sample covariance of two inputs <em>x</em> and <em>y</em>. Covariance is a measure of the joint variability of two inputs.</p> <p>Both inputs must be of the same length (no less than two), otherwise <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised.</p> <p>Examples:</p> <pre data-language="pycon3">>>> x = [1, 2, 3, 4, 5, 6, 7, 8, 9] +>>> y = [1, 2, 3, 1, 2, 3, 1, 2, 3] +>>> covariance(x, y) +0.75 +>>> z = [9, 8, 7, 6, 5, 4, 3, 2, 1] +>>> covariance(x, z) +-7.5 +>>> covariance(z, x) +-7.5 +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.10.</span></p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.correlation"> +<code>statistics.correlation(x, y, /, *, method='linear')</code> </dt> <dd> +<p>Return the <a class="reference external" href="https://en.wikipedia.org/wiki/Pearson_correlation_coefficient">Pearson’s correlation coefficient</a> for two inputs. Pearson’s correlation coefficient <em>r</em> takes values between -1 and +1. It measures the strength and direction of a linear relationship.</p> <p>If <em>method</em> is “ranked”, computes <a class="reference external" href="https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient">Spearman’s rank correlation coefficient</a> for two inputs. The data is replaced by ranks. Ties are averaged so that equal values receive the same rank. The resulting coefficient measures the strength of a monotonic relationship.</p> <p>Spearman’s correlation coefficient is appropriate for ordinal data or for continuous data that doesn’t meet the linear proportion requirement for Pearson’s correlation coefficient.</p> <p>Both inputs must be of the same length (no less than two), and need not to be constant, otherwise <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised.</p> <p>Example with <a class="reference external" href="https://en.wikipedia.org/wiki/Kepler's_laws_of_planetary_motion">Kepler’s laws of planetary motion</a>:</p> <pre data-language="pycon3">>>> # Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune +>>> orbital_period = [88, 225, 365, 687, 4331, 10_756, 30_687, 60_190] # days +>>> dist_from_sun = [58, 108, 150, 228, 778, 1_400, 2_900, 4_500] # million km + +>>> # Show that a perfect monotonic relationship exists +>>> correlation(orbital_period, dist_from_sun, method='ranked') +1.0 + +>>> # Observe that a linear relationship is imperfect +>>> round(correlation(orbital_period, dist_from_sun), 4) +0.9882 + +>>> # Demonstrate Kepler's third law: There is a linear correlation +>>> # between the square of the orbital period and the cube of the +>>> # distance from the sun. +>>> period_squared = [p * p for p in orbital_period] +>>> dist_cubed = [d * d * d for d in dist_from_sun] +>>> round(correlation(period_squared, dist_cubed), 4) +1.0 +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.10.</span></p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.12: </span>Added support for Spearman’s rank correlation coefficient.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="statistics.linear_regression"> +<code>statistics.linear_regression(x, y, /, *, proportional=False)</code> </dt> <dd> +<p>Return the slope and intercept of <a class="reference external" href="https://en.wikipedia.org/wiki/Simple_linear_regression">simple linear regression</a> parameters estimated using ordinary least squares. Simple linear regression describes the relationship between an independent variable <em>x</em> and a dependent variable <em>y</em> in terms of this linear function:</p> <p><em>y = slope * x + intercept + noise</em></p> <p>where <code>slope</code> and <code>intercept</code> are the regression parameters that are estimated, and <code>noise</code> represents the variability of the data that was not explained by the linear regression (it is equal to the difference between predicted and actual values of the dependent variable).</p> <p>Both inputs must be of the same length (no less than two), and the independent variable <em>x</em> cannot be constant; otherwise a <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> is raised.</p> <p>For example, we can use the <a class="reference external" href="https://en.wikipedia.org/wiki/Monty_Python#Films">release dates of the Monty Python films</a> to predict the cumulative number of Monty Python films that would have been produced by 2019 assuming that they had kept the pace.</p> <pre data-language="pycon3">>>> year = [1971, 1975, 1979, 1982, 1983] +>>> films_total = [1, 2, 3, 4, 5] +>>> slope, intercept = linear_regression(year, films_total) +>>> round(slope * 2019 + intercept) +16 +</pre> <p>If <em>proportional</em> is true, the independent variable <em>x</em> and the dependent variable <em>y</em> are assumed to be directly proportional. The data is fit to a line passing through the origin. Since the <em>intercept</em> will always be 0.0, the underlying linear function simplifies to:</p> <p><em>y = slope * x + noise</em></p> <p>Continuing the example from <a class="reference internal" href="#statistics.correlation" title="statistics.correlation"><code>correlation()</code></a>, we look to see how well a model based on major planets can predict the orbital distances for dwarf planets:</p> <pre data-language="pycon3">>>> model = linear_regression(period_squared, dist_cubed, proportional=True) +>>> slope = model.slope + +>>> # Dwarf planets: Pluto, Eris, Makemake, Haumea, Ceres +>>> orbital_periods = [90_560, 204_199, 111_845, 103_410, 1_680] # days +>>> predicted_dist = [math.cbrt(slope * (p * p)) for p in orbital_periods] +>>> list(map(round, predicted_dist)) +[5912, 10166, 6806, 6459, 414] + +>>> [5_906, 10_152, 6_796, 6_450, 414] # actual distance in million km +[5906, 10152, 6796, 6450, 414] +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.10.</span></p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.11: </span>Added support for <em>proportional</em>.</p> </div> </dd> +</dl> </section> <section id="exceptions"> <h2>Exceptions</h2> <p>A single exception is defined:</p> <dl class="py exception"> <dt class="sig sig-object py" id="statistics.StatisticsError"> +<code>exception statistics.StatisticsError</code> </dt> <dd> +<p>Subclass of <a class="reference internal" href="exceptions#ValueError" title="ValueError"><code>ValueError</code></a> for statistics-related exceptions.</p> </dd> +</dl> </section> <section id="normaldist-objects"> <h2>NormalDist objects</h2> <p><a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a> is a tool for creating and manipulating normal distributions of a <a class="reference external" href="http://www.stat.yale.edu/Courses/1997-98/101/ranvar.htm">random variable</a>. It is a class that treats the mean and standard deviation of data measurements as a single entity.</p> <p>Normal distributions arise from the <a class="reference external" href="https://en.wikipedia.org/wiki/Central_limit_theorem">Central Limit Theorem</a> and have a wide range of applications in statistics.</p> <dl class="py class"> <dt class="sig sig-object py" id="statistics.NormalDist"> +<code>class statistics.NormalDist(mu=0.0, sigma=1.0)</code> </dt> <dd> +<p>Returns a new <em>NormalDist</em> object where <em>mu</em> represents the <a class="reference external" href="https://en.wikipedia.org/wiki/Arithmetic_mean">arithmetic mean</a> and <em>sigma</em> represents the <a class="reference external" href="https://en.wikipedia.org/wiki/Standard_deviation">standard deviation</a>.</p> <p>If <em>sigma</em> is negative, raises <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a>.</p> <dl class="py attribute"> <dt class="sig sig-object py" id="statistics.NormalDist.mean"> +<code>mean</code> </dt> <dd> +<p>A read-only property for the <a class="reference external" href="https://en.wikipedia.org/wiki/Arithmetic_mean">arithmetic mean</a> of a normal distribution.</p> </dd> +</dl> <dl class="py attribute"> <dt class="sig sig-object py" id="statistics.NormalDist.median"> +<code>median</code> </dt> <dd> +<p>A read-only property for the <a class="reference external" href="https://en.wikipedia.org/wiki/Median">median</a> of a normal distribution.</p> </dd> +</dl> <dl class="py attribute"> <dt class="sig sig-object py" id="statistics.NormalDist.mode"> +<code>mode</code> </dt> <dd> +<p>A read-only property for the <a class="reference external" href="https://en.wikipedia.org/wiki/Mode_(statistics)">mode</a> of a normal distribution.</p> </dd> +</dl> <dl class="py attribute"> <dt class="sig sig-object py" id="statistics.NormalDist.stdev"> +<code>stdev</code> </dt> <dd> +<p>A read-only property for the <a class="reference external" href="https://en.wikipedia.org/wiki/Standard_deviation">standard deviation</a> of a normal distribution.</p> </dd> +</dl> <dl class="py attribute"> <dt class="sig sig-object py" id="statistics.NormalDist.variance"> +<code>variance</code> </dt> <dd> +<p>A read-only property for the <a class="reference external" href="https://en.wikipedia.org/wiki/Variance">variance</a> of a normal distribution. Equal to the square of the standard deviation.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.from_samples"> +<code>classmethod from_samples(data)</code> </dt> <dd> +<p>Makes a normal distribution instance with <em>mu</em> and <em>sigma</em> parameters estimated from the <em>data</em> using <a class="reference internal" href="#statistics.fmean" title="statistics.fmean"><code>fmean()</code></a> and <a class="reference internal" href="#statistics.stdev" title="statistics.stdev"><code>stdev()</code></a>.</p> <p>The <em>data</em> can be any <a class="reference internal" href="../glossary#term-iterable"><span class="xref std std-term">iterable</span></a> and should consist of values that can be converted to type <a class="reference internal" href="functions#float" title="float"><code>float</code></a>. If <em>data</em> does not contain at least two elements, raises <a class="reference internal" href="#statistics.StatisticsError" title="statistics.StatisticsError"><code>StatisticsError</code></a> because it takes at least one point to estimate a central value and at least two points to estimate dispersion.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.samples"> +<code>samples(n, *, seed=None)</code> </dt> <dd> +<p>Generates <em>n</em> random samples for a given mean and standard deviation. Returns a <a class="reference internal" href="stdtypes#list" title="list"><code>list</code></a> of <a class="reference internal" href="functions#float" title="float"><code>float</code></a> values.</p> <p>If <em>seed</em> is given, creates a new instance of the underlying random number generator. This is useful for creating reproducible results, even in a multi-threading context.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.pdf"> +<code>pdf(x)</code> </dt> <dd> +<p>Using a <a class="reference external" href="https://en.wikipedia.org/wiki/Probability_density_function">probability density function (pdf)</a>, compute the relative likelihood that a random variable <em>X</em> will be near the given value <em>x</em>. Mathematically, it is the limit of the ratio <code>P(x <= +X < x+dx) / dx</code> as <em>dx</em> approaches zero.</p> <p>The relative likelihood is computed as the probability of a sample occurring in a narrow range divided by the width of the range (hence the word “density”). Since the likelihood is relative to other points, its value can be greater than <code>1.0</code>.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.cdf"> +<code>cdf(x)</code> </dt> <dd> +<p>Using a <a class="reference external" href="https://en.wikipedia.org/wiki/Cumulative_distribution_function">cumulative distribution function (cdf)</a>, compute the probability that a random variable <em>X</em> will be less than or equal to <em>x</em>. Mathematically, it is written <code>P(X <= x)</code>.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.inv_cdf"> +<code>inv_cdf(p)</code> </dt> <dd> +<p>Compute the inverse cumulative distribution function, also known as the <a class="reference external" href="https://en.wikipedia.org/wiki/Quantile_function">quantile function</a> or the <a class="reference external" href="https://web.archive.org/web/20190203145224/https://www.statisticshowto.datasciencecentral.com/inverse-distribution-function/">percent-point</a> function. Mathematically, it is written <code>x : P(X <= x) = p</code>.</p> <p>Finds the value <em>x</em> of the random variable <em>X</em> such that the probability of the variable being less than or equal to that value equals the given probability <em>p</em>.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.overlap"> +<code>overlap(other)</code> </dt> <dd> +<p>Measures the agreement between two normal probability distributions. Returns a value between 0.0 and 1.0 giving <a class="reference external" href="https://www.rasch.org/rmt/rmt101r.htm">the overlapping area for the two probability density functions</a>.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.quantiles"> +<code>quantiles(n=4)</code> </dt> <dd> +<p>Divide the normal distribution into <em>n</em> continuous intervals with equal probability. Returns a list of (n - 1) cut points separating the intervals.</p> <p>Set <em>n</em> to 4 for quartiles (the default). Set <em>n</em> to 10 for deciles. Set <em>n</em> to 100 for percentiles which gives the 99 cuts points that separate the normal distribution into 100 equal sized groups.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="statistics.NormalDist.zscore"> +<code>zscore(x)</code> </dt> <dd> +<p>Compute the <a class="reference external" href="https://www.statisticshowto.com/probability-and-statistics/z-score/">Standard Score</a> describing <em>x</em> in terms of the number of standard deviations above or below the mean of the normal distribution: <code>(x - mean) / stdev</code>.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.9.</span></p> </div> </dd> +</dl> <p>Instances of <a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a> support addition, subtraction, multiplication and division by a constant. These operations are used for translation and scaling. For example:</p> <pre data-language="pycon3">>>> temperature_february = NormalDist(5, 2.5) # Celsius +>>> temperature_february * (9/5) + 32 # Fahrenheit +NormalDist(mu=41.0, sigma=4.5) +</pre> <p>Dividing a constant by an instance of <a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a> is not supported because the result wouldn’t be normally distributed.</p> <p>Since normal distributions arise from additive effects of independent variables, it is possible to <a class="reference external" href="https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables">add and subtract two independent normally distributed random variables</a> represented as instances of <a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a>. For example:</p> <pre data-language="pycon3">>>> birth_weights = NormalDist.from_samples([2.5, 3.1, 2.1, 2.4, 2.7, 3.5]) +>>> drug_effects = NormalDist(0.4, 0.15) +>>> combined = birth_weights + drug_effects +>>> round(combined.mean, 1) +3.1 +>>> round(combined.stdev, 1) +0.5 +</pre> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> </dd> +</dl> <section id="normaldist-examples-and-recipes"> <h3> +<a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a> Examples and Recipes</h3> <section id="classic-probability-problems"> <h4>Classic probability problems</h4> <p><a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a> readily solves classic probability problems.</p> <p>For example, given <a class="reference external" href="https://nces.ed.gov/programs/digest/d17/tables/dt17_226.40.asp">historical data for SAT exams</a> showing that scores are normally distributed with a mean of 1060 and a standard deviation of 195, determine the percentage of students with test scores between 1100 and 1200, after rounding to the nearest whole number:</p> <pre data-language="pycon3">>>> sat = NormalDist(1060, 195) +>>> fraction = sat.cdf(1200 + 0.5) - sat.cdf(1100 - 0.5) +>>> round(fraction * 100.0, 1) +18.4 +</pre> <p>Find the <a class="reference external" href="https://en.wikipedia.org/wiki/Quartile">quartiles</a> and <a class="reference external" href="https://en.wikipedia.org/wiki/Decile">deciles</a> for the SAT scores:</p> <pre data-language="pycon3">>>> list(map(round, sat.quantiles())) +[928, 1060, 1192] +>>> list(map(round, sat.quantiles(n=10))) +[810, 896, 958, 1011, 1060, 1109, 1162, 1224, 1310] +</pre> </section> <section id="monte-carlo-inputs-for-simulations"> <h4>Monte Carlo inputs for simulations</h4> <p>To estimate the distribution for a model than isn’t easy to solve analytically, <a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a> can generate input samples for a <a class="reference external" href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo simulation</a>:</p> <pre data-language="pycon3">>>> def model(x, y, z): +... return (3*x + 7*x*y - 5*y) / (11 * z) +... +>>> n = 100_000 +>>> X = NormalDist(10, 2.5).samples(n, seed=3652260728) +>>> Y = NormalDist(15, 1.75).samples(n, seed=4582495471) +>>> Z = NormalDist(50, 1.25).samples(n, seed=6582483453) +>>> quantiles(map(model, X, Y, Z)) +[1.4591308524824727, 1.8035946855390597, 2.175091447274739] +</pre> </section> <section id="approximating-binomial-distributions"> <h4>Approximating binomial distributions</h4> <p>Normal distributions can be used to approximate <a class="reference external" href="https://mathworld.wolfram.com/BinomialDistribution.html">Binomial distributions</a> when the sample size is large and when the probability of a successful trial is near 50%.</p> <p>For example, an open source conference has 750 attendees and two rooms with a 500 person capacity. There is a talk about Python and another about Ruby. In previous conferences, 65% of the attendees preferred to listen to Python talks. Assuming the population preferences haven’t changed, what is the probability that the Python room will stay within its capacity limits?</p> <pre data-language="pycon3">>>> n = 750 # Sample size +>>> p = 0.65 # Preference for Python +>>> q = 1.0 - p # Preference for Ruby +>>> k = 500 # Room capacity + +>>> # Approximation using the cumulative normal distribution +>>> from math import sqrt +>>> round(NormalDist(mu=n*p, sigma=sqrt(n*p*q)).cdf(k + 0.5), 4) +0.8402 + +>>> # Solution using the cumulative binomial distribution +>>> from math import comb, fsum +>>> round(fsum(comb(n, r) * p**r * q**(n-r) for r in range(k+1)), 4) +0.8402 + +>>> # Approximation using a simulation +>>> from random import seed, choices +>>> seed(8675309) +>>> def trial(): +... return choices(('Python', 'Ruby'), (p, q), k=n).count('Python') +... +>>> mean(trial() <= k for i in range(10_000)) +0.8398 +</pre> </section> <section id="naive-bayesian-classifier"> <h4>Naive bayesian classifier</h4> <p>Normal distributions commonly arise in machine learning problems.</p> <p>Wikipedia has a <a class="reference external" href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Person_classification">nice example of a Naive Bayesian Classifier</a>. The challenge is to predict a person’s gender from measurements of normally distributed features including height, weight, and foot size.</p> <p>We’re given a training dataset with measurements for eight people. The measurements are assumed to be normally distributed, so we summarize the data with <a class="reference internal" href="#statistics.NormalDist" title="statistics.NormalDist"><code>NormalDist</code></a>:</p> <pre data-language="pycon3">>>> height_male = NormalDist.from_samples([6, 5.92, 5.58, 5.92]) +>>> height_female = NormalDist.from_samples([5, 5.5, 5.42, 5.75]) +>>> weight_male = NormalDist.from_samples([180, 190, 170, 165]) +>>> weight_female = NormalDist.from_samples([100, 150, 130, 150]) +>>> foot_size_male = NormalDist.from_samples([12, 11, 12, 10]) +>>> foot_size_female = NormalDist.from_samples([6, 8, 7, 9]) +</pre> <p>Next, we encounter a new person whose feature measurements are known but whose gender is unknown:</p> <pre data-language="pycon3">>>> ht = 6.0 # height +>>> wt = 130 # weight +>>> fs = 8 # foot size +</pre> <p>Starting with a 50% <a class="reference external" href="https://en.wikipedia.org/wiki/Prior_probability">prior probability</a> of being male or female, we compute the posterior as the prior times the product of likelihoods for the feature measurements given the gender:</p> <pre data-language="pycon3">>>> prior_male = 0.5 +>>> prior_female = 0.5 +>>> posterior_male = (prior_male * height_male.pdf(ht) * +... weight_male.pdf(wt) * foot_size_male.pdf(fs)) + +>>> posterior_female = (prior_female * height_female.pdf(ht) * +... weight_female.pdf(wt) * foot_size_female.pdf(fs)) +</pre> <p>The final prediction goes to the largest posterior. This is known as the <a class="reference external" href="https://en.wikipedia.org/wiki/Maximum_a_posteriori_estimation">maximum a posteriori</a> or MAP:</p> <pre data-language="pycon3">>>> 'male' if posterior_male > posterior_female else 'female' +'female' +</pre> </section> <section id="kernel-density-estimation"> <h4>Kernel density estimation</h4> <p>It is possible to estimate a continuous probability density function from a fixed number of discrete samples.</p> <p>The basic idea is to smooth the data using <a class="reference external" href="https://en.wikipedia.org/wiki/Kernel_(statistics)#Kernel_functions_in_common_use">a kernel function such as a normal distribution, triangular distribution, or uniform distribution</a>. The degree of smoothing is controlled by a single parameter, <code>h</code>, representing the variance of the kernel function.</p> <pre data-language="python">import math + +def kde_normal(sample, h): + "Create a continuous probability density function from a sample." + # Smooth the sample with a normal distribution of variance h. + kernel_h = NormalDist(0.0, math.sqrt(h)).pdf + n = len(sample) + def pdf(x): + return sum(kernel_h(x - x_i) for x_i in sample) / n + return pdf +</pre> <p><a class="reference external" href="https://en.wikipedia.org/wiki/Kernel_density_estimation#Example">Wikipedia has an example</a> where we can use the <code>kde_normal()</code> recipe to generate and plot a probability density function estimated from a small sample:</p> <pre data-language="pycon3">>>> sample = [-2.1, -1.3, -0.4, 1.9, 5.1, 6.2] +>>> f_hat = kde_normal(sample, h=2.25) +>>> xarr = [i/100 for i in range(-750, 1100)] +>>> yarr = [f_hat(x) for x in xarr] +</pre> <p>The points in <code>xarr</code> and <code>yarr</code> can be used to make a PDF plot:</p> <img alt="Scatter plot of the estimated probability density function." src="https://docs.python.org/3.12/_images/kde_example.png"> </section> </section> </section> <div class="_attribution"> + <p class="_attribution-p"> + © 2001–2023 Python Software Foundation<br>Licensed under the PSF License.<br> + <a href="https://docs.python.org/3.12/library/statistics.html" class="_attribution-link">https://docs.python.org/3.12/library/statistics.html</a> + </p> +</div> |