Central Limit Theorem in Machine Learning

Bayes Theorem in Machine learning

The Central Limit Theorem (CLT) is a fundamental concept in statistics. It states that when you take a large enough sample size from any population with any distribution, the distribution of the sample means will be approximately normal, regardless of the original distribution of the population. In simpler terms, it suggests that if you repeatedly take samples from any population and calculate the average of each sample, those averages will tend to follow a normal (bell-shaped) distribution, even if the original data doesn’t follow a normal distribution.

Mathematical Formulation of Central Limit Theorem

Consider a random variable ? with a mean ? and a standard deviation σ. According to the Central Limit Theorem (CLT), when we repeatedly sample from this population and calculate the mean of each sample, denoted as x̄, the distribution of these sample means will approximate a normal distribution. Specifically, the sample mean x̄ follows a normal distribution with a mean σ and a standard deviation σ/√n where n represents the sample size.

The characteristic function of the sample mean

Where ?₁, ?₂,……, ?_n are independent and identically distributed random variables with mean ? and variance σ².

Use the moment generating function Moment Generating Function(MGF) to characterize the distribution of x̄. The MGF of x̄ is the product of the MGFs of the individual random variables ?₁, ?₂,……, ?_n.
By applying the properties of MGFs, we manipulate the MGF of x̄ to simplify it and obtain a function that resembles the MGF of a normal distribution.
As n approaches infinity, we take the limit of the MGF of x̄. Through this limit, we show that the MGF of x̄ converges to the MGF of a normal distribution.
By the properties of MGFs and convergence, we conclude that the distribution of x̄ approaches a normal distribution as n becomes sufficiently large.

Assumptions and conditions of Central Limit Theorem

The Central Limit Theorem (CLT) is a fundamental concept in statistics, but it relies on certain assumptions and conditions to hold true. Here are the key assumptions and conditions required for the CLT to apply effectively:

Random Sampling: The samples must be selected randomly from the population. This ensures that each member of the population has an equal chance of being included in the sample, which helps in generalizing the results to the entire population.
Independence: The samples must be independent of each other. In other words, the outcome of one sample should not affect the outcome of another sample. This ensures that each observation contributes unique information to the overall sample distribution.
Sample Size: The sample size should be “sufficiently large.” While there is no strict rule for what constitutes a large enough sample size, as a general guideline, a sample size of at least 30 is often considered sufficient for the CLT to apply. However, smaller sample sizes may also suffice depending on the shape of the population distribution.
Population Distribution: The population from which the samples are drawn should have a finite mean (?) and variance (σ²). While the CLT is quite robust and can often apply even when this condition is not strictly met, it is generally assumed for the theorem to hold true.
Identically Distributed: The samples should be drawn from the same population and should have identical distributions. This ensures that each sample provides consistent information about the population distribution.
Finite Moments: The population distribution should have finite moments, particularly the first and second moments (mean and variance). This condition ensures that the sample mean and sample variance are well-defined.

Central Limit Theorem and Machine Learning

In the context of machine learning, the CLT is relevant in various ways:

Model Evaluation: CLT assures us that evaluation metrics from different samples tend towards a normal distribution with larger sample sizes, validating model performance.
Parameter Estimation: Techniques like maximum likelihood estimation benefit from CLT, ensuring parameter estimates approach normal distribution as sample size increases, enhancing accuracy.
Hypothesis Testing: CLT supports statistical tests by ensuring test statistics approach normality with larger sample sizes, enabling robust hypothesis testing in machine learning.
Bootstrapping: Central Limit Theorem justifies bootstrapping by ensuring the distribution of bootstrap sample means converges to normality, enhancing the reliability of estimated statistics.
Ensemble Methods: Ensemble methods leverage CLT by combining models, relying on the theorem to ensure aggregated predictions converge to a more accurate representation, enhancing predictive performance.

How CLT helps in generalizing large datasets?

The Central Limit Theorem (CLT) plays a crucial role in generalizing large datasets in statistics and data analysis.

Normalization of Sample Means: CLT states that the distribution of sample means tends towards a normal distribution as the sample size increases, regardless of the shape of the population distribution. This property allows us to make inferences about the population mean based on the sample mean, assuming that the sample size is sufficiently large.
Improved Estimation Accuracy: For large datasets, calculating the mean or other statistics from the entire dataset might be computationally intensive or impractical. CLT allows us to estimate population parameters (such as the mean) more efficiently by using sample statistics. The larger the dataset, the closer the sample mean will be to the population mean, according to CLT.
Confidence Intervals: CLT facilitates the construction of confidence intervals for population parameters. With large datasets, we can rely on CLT to approximate the distribution of sample means, enabling the calculation of confidence intervals with a high degree of accuracy.
Hypothesis Testing: In hypothesis testing, CLT provides a theoretical foundation for making inferences about the population based on sample data. It allows us to assess the likelihood of observing certain sample statistics under the null hypothesis and make decisions accordingly.
Generalization to the Population: By leveraging CLT, we can generalize insights obtained from analyzing large datasets to the broader population from which the data was sampled. This is particularly useful in fields like machine learning, where models trained on large datasets can be generalized to unseen data or populations.

Code Implementation

This code generates a population with a mean of 100 and a standard deviation of 15. It then takes 1000 samples of size 1000 each from this population, calculates the mean of each sample, and plots a histogram of these sample means.

Python3

import numpy as np
import matplotlib.pyplot as plt

# Parameters
population_mean = 100
population_stddev = 15
sample_size = 1000
num_samples = 1000

# Generate population data
population_data = np.random.normal(population_mean, population_stddev, 100000)

# Generate sample means
sample_means = [np.mean(np.random.choice(population_data, sample_size)) for _ in range(num_samples)]

# Plotting
plt.hist(sample_means, bins=30, density=True, alpha=0.6, color='b', edgecolor='black')
plt.xlabel('Sample Mean')
plt.ylabel('Density')
plt.title('Central Limit Theorem')
plt.show()

Output:

As per the Central Limit Theorem, the histogram should approximate a normal distribution, regardless of the underlying distribution of the population data.

Tags:

#AI-ML-DS With Python #ML-Statistics #AI-ML-DS #Machine Learning #Machine Learning