Biostatistics — 13. Theme 9. Principal Component Analysis

←

D. Shabanov, M. Kravchenko. Статистичний аналіз даних у зоології та екології

→

Topic 8. Cluster analysis

Topic 9. Principal Component Analysis

Topic 10. Discriminant Analysis

Biostatistics-12

Біостатистика‑13

Biostatistics-14

{"translated_text": "[IMG_1]\n\n\n\n←\n\nD.A. Shabanov, M.A. Kravchenko. Statistical Data Analysis in Zoology and Ecology\n\n→\n\n\n\nTopic 8. Cluster Analysis\n\nTopic 9. Principal Component Method\n\nTopic 10. Discriminant Analysis\n\n\n\nBiostatistics-12\n\nBiostatistics-13\n\nBiostatistics-14\n\n9.1. Essence of the Method (Using a Two-Dimensional Example)\n\nThe principal component method or component analysis (PCA) is one of the most important methods in the arsenal of a zoologist or ecologist. Unfortunately, in cases where the application of component analysis would be quite appropriate, cluster analysis is applied all too often.\n\nA typical task for which component analysis is useful is as follows: there is a certain set of objects, each of which is characterized by a certain (sufficiently large) number of features. The researcher is interested in the patterns reflected in the diversity of these objects. In the case when there are grounds to assume that objects are distributed among hierarchically subordinate groups, cluster analysis can be used — a method of classification (distribution into groups). If there are no grounds to expect that some hierarchy is reflected in the diversity of objects, it is logical to use ordination (ordered arrangement). If each object is characterized by a sufficiently large number of features (at least by a number of features that cannot be adequately represented on a single graph), it is optimal to begin the study of data with principal component analysis. The point is that this method is simultaneously a method of dimensionality reduction (reduction of the number of measurements) of data.\n\nIf the group of objects under consideration is characterized by the values of one feature, a histogram (for continuous features) or a bar chart (for characterizing the frequencies of a discrete feature) can be used to characterize their diversity. If objects are characterized by two features, a two-dimensional scatter plot can be used; if three features — a three-dimensional one. And if there are many features? One can attempt to reflect the mutual arrangement of objects relative to each other in multidimensional space on a two-dimensional graph. Usually such dimensionality reduction is associated with loss of information. Among the various possible ways of such representation, one must choose the one in which the loss of information will be minimal.\n\nLet us explain the above using the simplest example: transition from two-dimensional space to one-dimensional space. The minimum number of points that defines a two-dimensional space (plane) is 3. Figure 9.1.1 shows the arrangement of three points on a plane. The coordinates of these points can be easily read from the figure itself. How to choose the line that will carry the maximum information about the mutual arrangement of points?\n[IMG_2]\nFig. 9.1.1. Three points on a plane defined by two features. Onto which line will the maximum variance of these points be projected?\n\nLet us consider the projections of points onto line A (shown in blue). The coordinates of projections of these points onto line A are: 2, 8, 10. The mean value is 62/3. The variance is (2-62/3)+ (8-62/3)+ (10-62/3)=342/3.\n\nNow let us consider line B (shown in green). The coordinates of points are 2, 3, 7; the mean value is 4, the variance is 14. Thus, a smaller proportion of variance is reflected onto line B than onto line A.\n\nWhat is this proportion? Since lines A and B are orthogonal (perpendicular), the proportions of total variance projecting onto A and B do not intersect. Therefore, the total variance of arrangement of the points of interest can be calculated as the sum of these two components: 342/3+14=482/3. At that, 71.2% of the total variance is projected onto line A, and 28.8% onto line B.\n\nAnd how to determine onto which line the maximum proportion of variance will be reflected? This line will correspond to the regression line for the points of interest, which is denoted as C (red color). 77.2% of the total variance is reflected onto this line, and this is the maximum possible value for this arrangement of points. Such a line, onto which the maximum proportion of total variance is projected, is called the first principal component.\n\nAnd onto which line should the remaining 22.8% of total variance be reflected? Onto the line perpendicular to the first principal component. This line will also be a principal component, since the maximum possible proportion of variance (naturally, not counting that reflected onto the first principal component) is reflected onto it. Thus, this is the second principal component.\n\nHaving calculated these principal components using Statistica (we will describe the dialogue a bit later), we obtain the picture shown in Fig. 9.1.2. The coordinates of points on principal components are shown in standard deviations.\n[IMG_3]\nFig. 9.1.2. Arrangement of the three points shown in Fig. 9.1.1 on the plane of two principal components. Why are these points arranged differently relative to each other than in Fig. 9.1.1?\n\nIn Fig. 9.1.2, the mutual arrangement of points turns out to be changed. In order to correctly interpret such pictures in the future, one should consider in more detail the reasons for the differences in point arrangement in Fig. 9.1.1 and 9.1.2. Point 1 in both cases is to the right (has a larger coordinate on the first feature and the first principal component) than point 2. But, for some reason, point 3 in the original arrangement is below the other two points (has the smallest value of feature 2), and above the other two points on the plane of principal components (has a larger coordinate on the second component). This is because the principal component method optimizes exactly the variance of the original data projecting onto the axes it selects. If a principal component is correlated with some original axis, the component and the axis may be directed in the same direction (have positive correlation) or in opposite directions (have negative correlations). Both variants are equivalent. The principal component algorithm may \"flip\" or not \"flip\" any plane; no conclusions should be drawn from this.\n\nHowever, the points in Fig. 9.1.2 are not just \"flipped\" compared to their mutual arrangement in Fig. 9.1.1; their mutual arrangement has also changed in a certain way. The differences between points on the second principal component appear enhanced. 22.76% of the total variance, accounted for by the second component, \"spread\" the points to the same distance as 77.24% of the variance accounted for by the first principal component.\n\nIn order for the arrangement of points on the principal component plane to correspond to their actual arrangement, this plane should be distorted. Fig. 9.1.3 shows two concentric circles; their radii are related as the proportions of variances reflected by the first and second principal components. The picture corresponding to Fig. 9.1.2 is distorted so that the standard deviation on the first principal component corresponds to the larger circle, and on the second — to the smaller one.\n[IMG_4]\nFig. 9.1.3. We took into account that a larger proportion of variance falls on the first principal component than on the second. For this, we distorted Fig. 9.1.2, fitting it to two concentric circles whose radii are related as the proportions of variances falling on the principal components. But the arrangement of points still does not correspond to the original, shown in Fig. 9.1.1!\n\nAnd why does the mutual arrangement of points in Fig. 9.1.3 not correspond to that in Fig. 9.1.1? In the original figure, Fig. 9.1, points are arranged according to their coordinates, not according to the proportions of variance falling on each axis. In Fig. 9.1.1, a distance of 1 unit along the first feature (on the abscissa axis) accounts for a smaller proportion of point variance along this axis than a distance of 1 unit along the second feature (on the ordinate axis). And in Fig. 9.1.1, distances between points are determined precisely by the units in which the features by which they are described are measured.\n\nLet us slightly complicate the task. Table 9.1.1 shows the coordinates of 10 points in 10-dimensional space. The first three points and the first two measurements are the example we have just considered.\n\nTable 9.1.1. Coordinates of Points for Further Analysis\n\nPoints\n\nCoordinates\n\n\n\nCharacter 1\n\nCharacter 2\n\nCharacter 3\n\nCharacter 4\n\nCharacter 5\n\nCharacter 6\n\nCharacter 7\n\nCharacter 8\n\nCharacter 9\n\nCharacter 10\n\n\n\n1\n\n10\n\n7\n\n6\n\n9\n\n5\n\n8\n\n6\n\n1\n\n9\n\n7\n\n\n\n2\n\n2\n\n3\n\n2\n\n1\n\n7\n\n1\n\n8\n\n5\n\n7\n\n1\n\n\n\n3\n\n8\n\n2\n\n3\n\n6\n\n9\n\n2\n\n2\n\n1\n\n5\n\n8\n\n\n\n4\n\n2\n\n7\n\n4\n\n1\n\n5\n\n3\n\n9\n\n8\n\n3\n\n2\n\n\n\n5\n\n8\n\n6\n\n5\n\n1\n\n6\n\n4\n\n3\n\n9\n\n2\n\n7\n\n\n\n6\n\n8\n\n10\n\n7\n\n8\n\n2\n\n1\n\n3\n\n5\n\n2\n\n4\n\n\n\n7\n\n2\n\n5\n\n3\n\n5\n\n5\n\n6\n\n9\n\n9\n\n2\n\n1\n\n\n\n8\n\n6\n\n3\n\n3\n\n6\n\n7\n\n8\n\n2\n\n4\n\n1\n\n8\n\n\n\n9\n\n6\n\n1\n\n2\n\n4\n\n9\n\n6\n\n1\n\n3\n\n7\n\n3\n\n\n\n10\n\n4\n\n4\n\n3\n\n7\n\n8\n\n2\n\n1\n\n7\n\n5\n\n8\n\n\n\nFor educational purposes, let us first consider only part of the data from Table 9.1.1. In Fig. 9.1.4, we see the position of ten points on the plane of the first two features. Note that the first principal component (line C) passed somewhat differently than in the previous case. There is nothing surprising in this: its position is influenced by all the points under consideration.\n[IMG_5]\nFig. 9.1.4. We increased the number of points. The first principal component passes somewhat differently now, as the added points have influenced it\n\nFig. 9.1.5 shows the position of the 10 points we considered on the plane of the first two components. Note: everything has changed, not only the proportion of variance accounted for by each principal component, but also the position of the first three points!\n[IMG_6]\nFig. 9.1.5. Ordination in the plane of the first principal components of 10 points characterized in Table 9.1.1. Only the values of the first two features were considered; the last 8 columns of Table 9.1.1 were not used\n\nIn general, this is natural: since the principal components are arranged differently, the mutual arrangement of points has also changed.\n\nDifficulties in comparing the arrangement of points on the principal component plane and on the original plane of their feature values may cause bewilderment: why use such a difficult-to-interpret method? The answer is simple. In the case when the compared objects are described by only two features, their ordination by these original features can be used quite well. All the advantages of the principal component method manifest in the case of multidimensional data. The principal component method in such a case turns out to be an effective way of dimensionality reduction of data.\n\n9.2. Transition to Initial Data with a Large Number of Measurements\n\nLet us consider a more complex case: let us analyze the data presented in Table 9.1.1 by all ten features. Fig. 9.2.1 shows how to call up the window of the method we need.\n[IMG_7]\nFig. 9.2.1. Launching the principal component method\n\nWe will be interested only in selecting features for analysis, although the Statistica dialogue allows much finer tuning (Fig. 9.2.2).\n[IMG_8]\nFig. 9.2.2. Selecting variables for analysis\n\nAfter performing the analysis, a results window appears with several tabs (Fig. 9.2.3). All main windows are already accessible from the first tab.\n[IMG_9]\nFig. 9.2.3. The first tab of the principal component analysis results dialogue\n\nOne can see that the analysis extracted 9 principal components, and described 100% of the variance reflected in the 10 initial features. This means that one feature was redundant, excessive.\n\nLet us begin reviewing the results from the button \"Plot case factor voordinates, 2D\": it will show the arrangement of points on the plane defined by two principal components. Pressing this button, we will get to a dialogue where we need to specify which components we will use; naturally, we should begin the analysis with the first and second components. The result is in Fig. 9.2.4.\n[IMG_10]\nFig. 9.2.4. Ordination of the objects under consideration on the plane of the first two principal components\n\nThe position of points has changed, and this is natural: new features are involved in the analysis. Fig. 9.2.4 reflects more than 65% of the entire diversity in the mutual arrangement of points, and this is already a non-trivial result. For example, returning to Table 9.1.1, one can verify that points 4 and 7, as well as 8 and 10, are indeed quite close to each other. However, the differences between them may concern other principal components not shown in the figure: after all, they still account for a third of the remaining variability.\n\nBy the way, when analyzing the placement of points on the principal component plane, there may be a need to analyze the distances between them. The easiest way to obtain a distance matrix between points is to use the cluster analysis module.\n\nAnd how are the extracted principal components related to the original features? This can be found out by pressing the button (Fig. 9.2.3) Plot var. factor coordinates, 2D. The result is in Fig. 9.2.5.\n[IMG_11]\nFig. 9.2.5. Projections of original features onto the plane of the first two principal components\n\nWe are looking at the plane of two principal components from above. Original features that are not related to the principal components at all will be perpendicular (or almost perpendicular) to them and will be reflected as short segments ending near the origin. Thus, feature No. 6 is least related to the first two principal components (although it shows a certain positive correlation with the first component). The segments corresponding to features that will be fully reflected on the principal component plane will end on the circle of unit radius encompassing the center of the figure.\n\nFor example, one can see that features 10 (related by positive correlation), as well as 7 and 8 (related by negative correlation), had the strongest influence on the first principal component. In order to examine the structure of such correlations in more detail, one can press the button Factor coordinates of variables, and obtain the table shown in Fig. 9.2.6.\n[IMG_12]\nFig. 9.2.6. Correlations between original features and extracted principal components (Factors)\n\nThe button Eigenvalues outputs the values called eigenvalues of principal components. In the upper part of the window shown in Fig. 9.2.3, such values are output for several first components; the button Scree plot shows them in a convenient form (Fig. 9.2.7).\n[IMG_13]\nFig. 9.2.7. Eigenvalues of extracted principal components and the proportions of total variance they reflect\n\nFirst, one needs to understand what the eigenvalue shows. This is a measure of variance reflected onto a principal component, measured in the amount of variance that pertained to each feature in the initial data. If the eigenvalue of the first principal component equals 3.4, this means that more variance is reflected onto it than onto three features from the initial set. Eigenvalues are linearly related to the proportion of variance pertaining to a principal component, the only difference being that the sum of eigenvalues equals the number of original features, and the sum of variance proportions equals 100%.\n\nAnd what does it mean that information about variability across 10 features was possible to reflect in 9 principal components? That one of the initial features was redundant, did not add any new information. This was indeed the case; Fig. 9.2.8 shows how the set of points reflected in Table 9.1.1 was generated.\n[IMG_14]\nFig. 9.2.8 The data in Table 9.1.1 are artificial, and here one can see how they were generated"}

Точки

Координати

Character 1

Character 2

Character 3

Character 4

Character 5

Character 6

Character 7

Character 8

Character 9

Character 10