What is Cross Decomposition?

Cross Decomposition is a supervised machine learning technique used in multivariate data analysis through which the data set could be divided into matrices which could help in capturing different aspects of the data. The data might contain features that are correlated to each other and fitting all the original features to a normal data modelling technique might not be an efficient way to model the data with correlated features. Cross decomposition helps in the dimensionality reduction of such data sets without losing the insights of the original data. What cross decomposition does is that it divides the input features and different targets into component matrices and then finds out the relationship between the input features and the target matrix. When it divides the input features in another matrix it reduces the number of features in the matrix the same way principal component regression (PCR) does, the only difference here being that the PCR is a unsupervised learning algorithm whereas cross decomposition uses supervised learning algorithms considering the target values also. In the next section we will be discussing about the components of cross decomposition, what it actually comprises of and how is it useful for modelling complex datasets.

Different Components of Cross Decomposition

Two of the most popular methods of cross decomposition are Canonical Correlation Analysis (CCA) and Partial Least Squares (PLS) which have several important applications in the cross-decomposition process.

Partial Least Squares:

PLS is a type of Cross Decomposition method which is used for predictive modelling and finding the relationships between two different matrices of data. It is useful in the case where the data is multicollinear or the number of features are more than the number of observations. The working principle of PLS is to find the best fitting linear regression model by projecting the features and targets to a new space, it extracts new components of the predictors called the latent variables which have the best predictive capabilities for the target variables.

How Does PLS Works:

• PLS starts with standardizing the data for both the predictor and target matrices, it helps stabilizing the scales of the variables.
• After the first step it converts the predictor and target matrices into latent variables which are not correlated to each other in the same set, and which do not lose the information of the original set. These latent variables are chosen in such a way that the predictor and target set must have a maximized correlation with each other.
• This model is constructed in a new space with fewer variables such that the latent variables of the same set are uncorrelated to each other.
• At last, linear regression is performed on this new space of latent variables.

The three main algorithms that are used under the Partial Least Squares method are PLSCanonical, PLSSVD, and PLSRegression. All of these works in a similar way just that they have their own advantages and disadvantages. Let’s discuss them one by one:

• PLSCanonical Algorithm: PLSCanonical is the most commonly used PLS algorithm it is used to find out the relationship between two different matrices of data whether they are numerical or categorical. PLSCanonical algorithm finds out the latent variables from the predictor and target variables and maximises the covarience between them.
• Partial Least Squares Singular Value Decomposition (PLSSVD): PLSSVD is a more specialized algorithm which works with numerical data only, it has the combination of Partial Least Squares and Singular Value Decomposition.
• PLSRegression: PLSRegression is a regression algorithm that can be used to predict a continuous target variable from a set of predictor variables.

Canonical Correlation Analysis (CCA):

Canonical Correlation Analysis is similar to the PLSCanonical Algorithm, but CCA finds the projections that maximizes the correlation between linear combinations of variables. Suppose that the feature matrix is X with different variables in it as and the target matrix is Y with variables , then CCa finds out the linear combinations:

Canonical Variate of X =

Canonical Variate of Y =

The values of and are chosen such that the correlation between the canonical variate of X and the canonical variate of Y is maximum.

Usually, in real-world datasets, some of the features of the data are highly correlated with each other. Applying normal regression methods to highly correlated data is not an effective way to analyze such data, since multicollinearity makes the estimates highly sensitive to any change in the model. In this article, we will be diving deep into Cross Decomposition which will help us understand the optimal solutions to problems like multicollinearity in the data.