Biostatistics — 07. Topic 4 (continuation). Non‑parametric tests for comparing samples

←

D. Shabanov, M.A. Kravchenko. Statistical Analysis of Data in Zoology and Ecology

→

Topic 4 (continued). Multiple comparisons

Topic 4 (continued). Non-parametric tests for sample comparison

Topic 5. A brief introduction to analysis of variance

Biostatistics-06

Biostatistics-07

Biostatistics-08

4.8. Non-parametric analogues of parametric methods Parametric tests, which we have considered so far, are based on the fact that the samples being compared can be characterized by two parameters: the mean and the standard deviation (or some other measure of variability). What if the distribution in the samples (or, more precisely, in the population from which these samples were obtained) is completely different? If the size of each of the compared samples is sufficiently large (more than a hundred), parametric tests can still be used. Whatever the distribution of these samples, their means "behave" approximately the same as the means of samples from a normal distribution. However, if the sample size is smaller, non-parametric tests should be used. For example, the non-parametric analogue of Student's t-test is the Mann-Whitney U-test. The Student's t-test is based on a distribution that describes the deviation of the sample mean of a certain size around the population mean of a normally distributed variable. The greater the deviation from , the lower the probability that it occurred randomly during sample formation. But what if we know nothing about the nature of the distribution of the populations? Let's consider a fairly simple example that explains how a large group of non-parametric methods works – rank tests. We have two samples. Let's arrange their elements in ascending order: the first is a1, a2, a3, a4, a5; the second is b1, b2, b3, b4, b5, b6. Let's combine the elements of these samples into a general series, arranged in ascending order of their values. Let's compare three different cases: No. 1: a1, a2, a3, a4, a5, b1, b2, b3, b4, b5, b6; No. 2: a1, a2, a3, a4, b1, a5, b2, b3, b4, b5, b6; No. 3: b1, a1, b2, a2, b3, a3, b4, b5, a4, a5, b6. In case No. 1, all elements of one sample are located on one side of the general series, and all elements of the other series are on the other side. In case No. 2, one permutation (elements b1 and a5) would be enough to make the order of elements as in case No. 1. Finally, in case No. 3, the elements of the two samples are mixed, and to arrange them in a series where first one and then the other will be located, 5 permutations are needed. We need to choose between the alternative hypothesis (according to which samples a and b are taken from different populations) and the null hypothesis (according to which these samples are taken from the same population). Are the probabilities of the alternative and null hypotheses the same for the three different cases we have shown? No; the alternative hypothesis is more probable in the first case, and the null hypothesis in the third. The idea of a non-parametric rank test is that we can use the number of necessary permutations as a measure to evaluate the null and alternative hypotheses. The specific values calculated when applying non-parametric tests turn out to be different, but the logic of comparison approximately corresponds to the example we considered. Thus, thanks to the application of clever approaches, non-parametric analogues have been found for parametric methods of sample comparison (Table 4.8.1). Most often, non-parametric methods have lower power (i.e., they more often reject the alternative hypothesis when it is actually true), but they allow working with differently distributed data and are less sensitive to small sample sizes. Table 4.8.1. Non-parametric analogues of parametric methods

Type of comparison

Parametric methods

Non‑parametric methods

Comparison of values of a variable in two independent samples

Student's t-test; Analysis of variance (ANOVA)

Mann-Whitney U-test; Wald-Wolfowitz runs test; Two-sample Kolmogorov-Smirnov test

Comparison of values of a variable in two dependent samples

Student's t-test for paired comparisons

Sign test; Wilcoxon signed-rank test

Comparison of values of a variable in several independent samples

Analysis of variance (ANOVA)

Kruskal-Wallis one-way analysis of variance by ranks; Median test

{"translated_text":"[IMG_1]\n \n \n\n←\n\nD.A. Shabanov, M.A. Kravchenko. Statistical analysis of data in zoology and ecology\n\n→\n\nTopic 4 (continuation). Multiple comparisons\n\nTopic 4 (continuation). Non‑parametric criteria for comparing samples\n\nTopic 5. Brief introduction to analysis of variance\n\nBiostatistics‑06\n\nBiostatistics‑07\n\nBiostatistics‑08\n\n4.8. Non‑parametric analogues of parametric methods\nParametric criteria that we have considered so far are based on the assumption that the samples being compared can be characterized by two parameters: the mean and the standard deviation (or some other measure of variability). What should we do if the distribution in the samples (or, more precisely, in the population from which these samples were drawn) is completely different?\nIf the size of each of the compared samples is sufficiently large (more than one hundred), parametric criteria can still be used. Whatever distribution these samples have, their means \"behave\" roughly like the means of samples with a normal distribution. However, if the sample sizes are smaller, non‑parametric criteria should be used.\nFor example, the non‑parametric analogue of Student's t‑test is the Mann‑Whitney U‑test. Student's t‑test is based on a distribution that describes the deviations of the sample mean of a certain size [IMG_2] around the population mean of a normally distributed variable [IMG_3]. The stronger the deviation [IMG_4] from [IMG_5], the lower the probability that it arose by chance during sampling. But how to proceed if we know nothing about the nature of the population distributions?\nLet us consider a fairly simple example that illustrates how a large group of non‑parametric methods works — rank‑based tests. We have two samples. Arrange their elements in ascending order: the first — a1, a2, a3, a4, a5; the second — b1, b2, b3, b4, b5, b6. Combine the elements of these samples into a single series ordered by increasing values. Compare three different cases:\nNo. 1: a1, a2, a3, a4, a5, b1, b2, b3, b4, b5, b6;\nNo. 2: a1, a2, a3, a4, b1, a5, b2, b3, b4, b5, b6;\nNo. 3: b1, a1, b2, a2, b3, a3, b4, b5, a4, a5, b6.\nIn case 1 all elements of one sample are on one side of the combined series, and all elements of the other sample are on the opposite side. In case 2 a single transposition (elements b1 and a5) would be enough to make the order of elements the same as in case 1. Finally, in case 3 the elements of the two samples are intermingled, and to arrange them so that all of one sample come first and then all of the other, five transpositions are required. We need to choose between the alternative hypothesis (that samples a and b are drawn from different populations) and the null hypothesis (that they are drawn from the same population). Are the probabilities of the alternative and null hypotheses the same for the three cases we have shown? No; the alternative hypothesis is more probable in the first case, and the null hypothesis in the third.\nThe idea of a rank‑based non‑parametric test is that we can use the number of required transpositions as a measure for evaluating the null and alternative hypotheses. The specific statistics calculated when applying non‑parametric tests are different, but the logic of comparison roughly corresponds to the example we considered.\nThus, thanks to clever approaches, non‑parametric analogues have been selected for parametric methods of sample comparison (Table 4.8.1). Most often non‑parametric methods have lower power (i.e., they reject the alternative hypothesis less often when it is actually true), but they allow work with diversely distributed data and are less sensitive to small sample sizes.\nTable 4.8.1. Non‑parametric analogues of parametric methods\n\nType of comparison\n\nParametric methods\n\nNon‑parametric methods\n\nComparison of a variable in two independent samples\n\nt‑test (Student);\nAnalysis of variance (ANOVA)\n\nMann‑Whitney U‑test;\nWald‑Wolfowitz runs test;\nTwo‑sample Kolmogorov‑Smirnov test\n\nComparison of a variable in two dependent samples\n\tt‑test for paired samples\n\nSign test\nWilcoxon test\n\nComparison of a variable in several independent samples\n\nAnalysis of variance (ANOVA)\n\nKruskal‑Wallis rank‑sum test;\nMedian test\n\n4.9. Mann‑Whitney U‑test\nTo examine the application of the Mann‑Whitney test on our example file Pelophylax_example.sta we will have to use a somewhat artificial example. As an example of a variable whose distribution differs markedly from normal, we can use the trait called DNA — DNA content per cell (in picograms, pg), measured by flow DNA cytometry.\n[IMG_6]\nFig. 4.9.1. The \"DNA\" trait has a distribution sharply different from normal\nWe will determine whether females and males of Pelophylax esculentus differ in this trait. To use the Mann‑Whitney test, go to the menu Statistics / Nonparametrics. Note the icons in the menu: they correspond to those used for analogous comparisons with the t‑test.\n[IMG_7]\nFig. 4.9.2. Mann‑Whitney U‑test is computed here\nIn the dialog box you must specify the dependent (Dependent) and grouping (Grouping) variables; if the grouping variable has more than two values, select the two values that will correspond to the samples being compared. To select only representatives of Pelophylax esculentus, use the Select cases window and the alphanumeric codes entered in section 3.1 when describing the example file.\n[IMG_8]\nFig. 4.9.3. Settings chosen for the described comparison\nYou can see that Statistica computes all three criteria listed in Table 4.9.1 that are used for comparing two independent samples, but it \"recommends\" (launches from the button in the upper left corner) the Mann‑Whitney test. We will compute it and verify that the differences between females and males in DNA amount per cell are not statistically significant.\n[IMG_9]\nFig. 4.9.4. Result of the Mann‑Whitney comparison\nIf a one‑tailed test is not of interest, it is advisable to use the p‑value computed with the adjustment (the one after the column \"Z adjusted\", i.e., 0.906780). This adjustment increases the test's power for samples larger than 20. In any case, no substantially significant difference between males and females was found.\nThe dialog we used for the Mann‑Whitney comparison provides the possibility of constructing box‑plots. Since we are using a non‑parametric method, the plot does not display sample parameters (e.g., its mean), but uses non‑parametric measures — median and quartiles (values that \"cut\" the distribution into quarters).\n[IMG_10]\nFig. 4.9.5. Graphical comparison of DNA trait distributions for females and males Pelophylax esculentus\nIt may seem odd why the first (Min to 25 %) and last (75 % to Max) quarters are so much narrower than the second and third. To understand this, we will construct a categorized histogram.\n[IMG_11]\nFig. 4.9.6. Histogram showing the distributions of the DNA trait values recorded for females and males Pelophylax esculentus\nIt becomes clear that the surprising property of the distributions shown in the previous figure is a consequence of the bimodality of the trait under consideration.\n4.10. Sign test for paired comparisons\nIn our example file Pelophylax_example.sta there are no data that require comparison of two related samples, so we will create them artificially. Imagine that a sample of 25 frogs was measured by two people. Their measurement results are in the columns First and Second. The size distribution in this sample was initially far from normal.\n[IMG_12]\nFig. 4.10.1. Distribution of frog sizes (in 0.1 mm) based on measurements performed by two people on the same material\nNevertheless, for many frogs the measurements made by the first and second researcher differ. Our task is to determine whether the two researchers measure frog length equally. To answer this, we will use the sign test.\n[IMG_13]\nFig. 4.10.2. Using the sign test to compare measurement results made by two different researchers\nThe sign test simply determines the proportion of cases in which a value from one sample is greater than a value from the other sample.\n[IMG_14]\nFig. 4.10.3. Differences are statistically significant!\nWe can establish that the second researcher significantly more often overestimated the measurement results compared with the first researcher.\nWe will compare this result with that obtained using a parametric method — the paired‑samples t‑test.\n[IMG_15]\nFig. 4.10.4. The parametric method gave the same result, but with somewhat higher reliability\nA lower p‑value obtained with the parametric criterion is fully consistent with the aforementioned fact that parametric methods have greater power than non‑parametric ones. But was it appropriate to use a parametric test? In fact, it was. Paired comparisons consider not the sets of values in the first and second samples, but the difference for each element between the first and second sample. Let us plot the distribution of the difference between the First and Second samples.\n[IMG_16]\nFig. 4.10.5. Distribution of the difference between the two researchers' measurements\nIt can be seen that the deviation of the difference distribution from normality is statistically insignificant. Using a parametric test was therefore entirely appropriate.\nCould we have used methods for comparing independent samples? When comparing independent samples, the fact that the distribution of the variables of interest differs markedly from normal becomes important. Thus, we should use not the t‑test but the U‑test. To use the Mann‑Whitney U‑test, the data file must be restructured: all measurements should be placed in one column, and the second column will become the grouping variable.\n[IMG_17]\nFig. 4.10.6. By Mann‑Whitney, measurements performed by two different people do not differ\nHow to explain this discrepancy? As in many other cases, the first thing to do when something is unclear is to look at the distribution of the variables of interest.\n[IMG_18]\nFig. 4.10.7. The distributions of measurement results obtained by the two people are practically identical. Yet, as Fig. 4.10.3 shows, for 75 % of the frogs the second researcher’s measurements are larger than the first’s!\nOf course, the obtained result is quite logical. By using the Mann‑Whitney test instead of the sign test (or the Wilcoxon test), we lost crucial information characterizing the patterns of change in the variable under study.\nBy the way, the data we used were generated artificially. The First column was a fragment from the file Pelophylax_example.sta, containing mainly the smallest and largest individuals, and the Second column was obtained with the formula =Trunc(First‑2.4+Rnd(8)). It should be clear what this formula \"does\".\n4.11. Kruskal‑Wallis rank‑based analysis of variance\nUntil now we have used only pairwise comparisons of samples. Now we will consider a method that allows simultaneous comparison of several samples. The Kruskal‑Wallis test is a non‑parametric analogue of analysis of variance (ANOVA), which is discussed in detail in the next section of this manual. From a computational point of view it is a multivariate generalization of the Mann‑Whitney test. Although the Kruskal‑Wallis test in some respects is inferior to ANOVA (e.g., it does not allow simultaneous assessment of two or more factors), it is a powerful tool suitable for solving many problems.\nWe will demonstrate the Kruskal‑Wallis test using our file Pelophylax_example.sta (see section 3.1). We need to find out whether representatives of different genotypes differ in the length of the internal heel bulge in a statistically significant way. This is a meaningful task, as the size and shape of the internal heel bulge are important diagnostic traits useful for distinguishing different forms of green frogs.\n[IMG_19]\nFig. 4.11.1. Note the highlighted icon corresponding to the comparison of several independent groups\nNaturally, the dependent variable is the heel‑bulge length (Ci), and the grouping variable is genotype.\n[IMG_20]\nFig. 4.11.2. Settings selected. If you need to compare not all values of the grouping variable, use the dialog invoked by the Code button\nClicking the Summary button yields the results of two tests: the non‑parametric Kruskal‑Wallis ANOVA and the median test, which is based on Pearson's [IMG_21] method. The use of [IMG_22] is discussed in more detail in a later chapter of this manual; here it suffices to say that this method is used for non‑parametric comparison of distributions. If the distributions of the dependent variable for different groups defined by the grouping trait are different, this indicates that the grouping and dependent variables are related. The Kruskal‑Wallis method, as you recall, belongs to rank‑based non‑parametric methods. These two methods operate on different principles and often yield substantially different results.\n[IMG_23]\nFig. 4.11.3. Both methods demonstrate a statistically significant effect of the grouping variable on the dependent variable. The Kruskal‑Wallis method gives p = 0.0047, and the median test — p = 0.0112\nNote: due to some unexplained snobbery, in some Statistica windows the leading zero before the decimal separator (with system settings using a comma) is omitted.\nPressing the Multiple comparisons of mean ranks for all groups button provides pairwise comparison results for all groups. In fact, this is equivalent to performing Mann‑Whitney comparisons for all possible group pairs. The program then displays two windows: the z‑value used in Mann‑Whitney calculations and the calculated significance level for each pair.\n[IMG_24]\nFig. 4.11.4. Pairwise group comparisons in the Kruskal‑Wallis test dialog are equivalent to multiple comparisons using the Mann‑Whitney test\nNote that when conducting multiple comparisons there is a risk of committing a Type I error (accepting the alternative hypothesis when the null is true). To avoid this risk, the multiple‑comparison correction described above should be used.\nFinally, the Box & whisker button allows a visual comparison of the distributions of different groups.\n[IMG_25]\nFig. 4.11.5. Comparison of heel‑bulge length distributions among representatives of different genotypes"}