Scipy Significance Testing

A significance test is to make an assumption about the parameters of a population (random variable) or the form of the population distribution in advance, and then use sample information to judge whether this assumption (alternative hypothesis) is reasonable, that is, to determine whether there is a significant difference between the true situation of the population and the null hypothesis. In other words, the significance test is used to determine whether the difference between the sample and our assumption about the population is purely due to chance variation, or is caused by the inconsistency between our assumption and the true situation of the population. The significance test tests the assumptions we make about the population, and its principle is to accept or reject the hypothesis based on the "principle of practical impossibility of small-probability events".

Significance testing is a method used to determine whether there is a difference between the experimental treatment group and the control group, or between the effects of two different treatments, and whether this difference is significant.

SciPy provides the scipy.stats module to perform Scipy significance testing functions.

Statistical Hypothesis

A statistical hypothesis is a hypothesis about the unknown distribution of one or more random variables. A statistical hypothesis in which the form of the distribution of the random variable is known and only involves one or several unknown parameters in the distribution is called a parametric hypothesis. The process of testing statistical hypotheses is called hypothesis testing, and the test used to judge parametric hypotheses is called a parametric test.

Null Hypothesis

Null hypothesis, a statistical term, also called the original hypothesis, refers to the hypothesis established in advance when conducting statistical tests. When the null hypothesis holds, the relevant statistic should follow a known probability distribution.

When the calculated value of the statistic falls into the rejection region, it can be known that a small-probability event has occurred, and the null hypothesis should be rejected.

A hypothesis to be tested is often denoted as H0, called the null hypothesis, and the hypothesis opposite to H0 is denoted as H1, called the alternative hypothesis.

When the null hypothesis is true, deciding to abandon the null hypothesis is called a Type I error, and its probability of occurrence is usually denoted asα；
When the null hypothesis is not true, deciding not to abandon the null hypothesis is called a Type II error, and its probability of occurrence is usually denoted asβ
α+β does not necessarily equal 1.

Usually only the maximum probability of making a Type I error is limitedα, and the probability of making a Type II error is not consideredβ. Such a hypothesis test is also called a significance test, and the probabilityαis called the significance level.

The most commonly usedαvalues are 0.01, 0.05, 0.10, etc. In general, depending on the research problem, if abandoning a true hypothesis causes a large loss, in order to reduce such errors,αtake a smaller value; conversely,αtake a larger value.

Alternative Hypothesis

The alternative hypothesis is one of the basic concepts of statistics. It contains all propositions about the population distribution that make the null hypothesis invalid. The alternative hypothesis is also called the opposite hypothesis or the alternative hypothesis.

The alternative hypothesis can replace the null hypothesis.

For example, for student evaluation, we will adopt:

"Students are worse than average" — as the null hypothesis

One-Sided Test

One-sided test, also called one-tailed test, also known as one-sided test. In hypothesis testing, the method of using the one-sided tail area in the area enclosed by the density curve of the test statistic and the axis to construct the critical region for testing is called a one-sided test.

When our hypothesis tests only one side of the value, it is called a "one-tailed test".

Example:

For the null hypothesis:

“均值等于 k”

We can have the alternative hypothesis:

“平均值小于 k”
或
“平均值大于 k”

Two-Sided Test

Two-sided test, also called two-tailed test or two-sided test. In hypothesis testing, a method that uses the tail areas on the left and right sides of the area enclosed by the density curve of the test statistic and the x-axis to construct the critical region for testing.

When our hypothesis tests both sides of the value.

Example:

For the null hypothesis:

“均值等于 k”

We can have the alternative hypothesis:

“均值不等于k”

In this case, the mean is less than or greater than k, and both sides need to be checked.

Alpha Value

The alpha value is the significance level.

The significance level is the probability of making an error when estimating that the population parameter falls within a certain interval, denoted byαindicated.

How close to extreme the data must be to reject the null hypothesis.

Usually taken as 0.01, 0.05, or 0.1.

P-Value

The P-value indicates how close to extreme the data actually is.

Compare the P-value with the alpha value to determine the level of statistical significance.

If the p-value <= alpha, we reject the null hypothesis and say the data is statistically significant; otherwise, we accept the null hypothesis.

T-Test

The T-test is used to determine whether there is a significant difference between the means of two variables and to determine whether they belong to the same distribution.

This is a two-tailed test.

The function ttest_ind() takes two samples of the same size and generates a tuple of the t statistic and the p-value.

Find whether the given values v1 and v2 come from the same distribution:

Example

import numpy as np
from scipy.stats import ttest_ind

v1 = np.random.normal(size=100)
v2 = np.random.normal(size=100)

res = ttest_ind(v1, v2)

print(res)

The output result is:

Ttest_indResult(statistic=0.40833510339674095, pvalue=0.68346891833752133)

If you only want to return the p-value, use the pvalue property:

Example

import numpy as np
from scipy.stats import ttest_ind

v1 = np.random.normal(size=100)
v2 = np.random.normal(size=100)

res = ttest_ind(v1, v2).pvalue

print(res)

The output result is:

0.68346891833752133

KS Test

The KS test is used to check whether a given value conforms to a distribution.

This function takes two parameters: the value to be tested and the CDF.

CDF is the Cumulative Distribution Function, also called the distribution function.

The CDF can be a string or a callable function that returns a probability.

It can be used as a one-tailed or two-tailed test.

By default it is a two-tailed test. We can pass the alternative parameter as one of the strings "two-sided", "less", or "greater".

Find whether a given value conforms to a normal distribution:

Example

import numpy as np
from scipy.stats import kstest

v = np.random.normal(size=100)

res = kstest(v, 'norm')

print(res)

The output result is:

KstestResult(statistic=0.047798701221956841, pvalue=0.97630967161777515)

Data Statistical Description

Using the describe() function, you can view information about an array, including the following values:

nobs -- number of observations
minmax -- minimum and maximum values
mean -- mathematical mean
variance -- variance
skewness -- skewness
kurtosis -- kurtosis

Display the statistical description information of the array:

Example

import numpy as np
from scipy.stats import describe

v = np.random.normal(size=100)
res = describe(v)

print(res)

The output result is:

DescribeResult(
    nobs=100,
    minmax=(-2.0991855456740121, 2.1304142707414964),
    mean=0.11503747689121079,
    variance=0.99418092655064605,
    skewness=0.013953400984243667,
    kurtosis=-0.671060517912661
  )

Normality Test (Skewness and Kurtosis)

The test that uses observed data to determine whether a population follows a normal distribution is called a normality test. It is an important special goodness-of-fit hypothesis test in statistical decision.

The normality test is based on skewness and kurtosis.

The normaltest() function returns the p-value of the null hypothesis:

“x 来自正态分布”

Skewness

A measure of the symmetry of data.

For a normal distribution, it is 0.

If it is negative, it means the data is skewed to the left.

If it is positive, it means the data is skewed to the right.

Kurtosis

A measure of whether data has heavy tails or light tails compared to a normal distribution.

Positive kurtosis means heavy tails.

Negative kurtosis means light tails.

Find the skewness and kurtosis of values in an array:

Example

import numpy as np
from scipy.stats import skew, kurtosis

v = np.random.normal(size=100)

print(skew(v))
print(kurtosis(v))

The output result is:

 0.11168446328610283
  -0.1879320563260931

Find whether the data comes from a normal distribution:

Example

import numpy as np
from scipy.stats import normaltest

v = np.random.normal(size=100)

print(normaltest(v))

The output result is:

NormaltestResult(statistic=4.4783745697002848, pvalue=0.10654505998635538)

Other Extensions