Biostatistics — 06. Topic 4 (continued). Multiple Comparisons
One of the most important problems associated with statistical analysis is the problem of multiple comparisons. How many errors have been made from not understanding this problem!
←
D.A. Shabanov, M.A. Kravchenko. Statistical Analysis of Data in Zoology and Ecology
→
Topic 4. Comparison of Samples
Topic 4 (continued). Multiple Comparisons
Topic 4 (continued). Non-parametric tests for sample comparison
Biostatistics-05
Biostatistics-06
Biostatistics-07
4.5. The problem of multiple comparisons In section 4.2, we compared male and female green frogs from the file Pelophylax_example.sta by their body length. However, this file also contains measurement results for six other morphometric traits. Males and females can be compared not only for all seven (including body length) morphometric traits simultaneously. Along with these measurements, it is also advisable to consider various indices – partial ratios of one measurement to another. How many of them can be calculated based on this data without considering the same pair of traits twice? Six measurements can be divided by body length, five by the first of these six, and so on. Thus, 6 + 5 + 4 + 3 + 2 + 1 = 21 indices can be calculated. Let's add 21 columns to the Pelophylax_example.sta file. The easiest way to do this is to double-click the mouse on the gray area outside the columns (to the right of the Cs variable in Fig. 4.5.1). Naturally, the Vars / Add method also works. Fig. 4.5.1. Double-clicking the area to the right of the columns opened the window for adding rows and columns. In the Add variables window, you need to specify the required number of columns. For the added columns, you need to specify names and set the formulas by which they will be calculated. It is more convenient to do this in the All Specs window (Vars / All Specs).
Fig. 4.5.2. Formulas for a group of columns have been added in the All Specs window. After pressing the OK button, the program sequentially checks and recalculates the formulas, which it reports in a special window. By checking the box Apply to all variable with valid expression, you can make this process automatic. Data in the new columns are displayed with excessive precision. You can make their display more adequate as follows: in the column header mode (Specs), change the data display format (Display format), selecting the Number mode. After that, you can specify the required number of decimal places (Decimal places), for example, three.
Fig. 4.5.3. You can move from one column to another using the arrow buttons (in the upper right corner of the window). Now, it seems, we can compare the values of all metric traits and indices for males and females in our file.
{"translated_text": "[IMG_1]\n\n\n\n←\n\n\n\nD.A. Shabanov, M.A. Kravchenko. Statistical Analysis of Data in Zoology and Ecology\n\n\n\n→\n\n\n\n\nTopic 4. Comparison of Samples\n\nTopic 4 (continued). Multiple Comparisons\n\nTopic 4 (continued). Non-parametric Tests for Sample Comparison\n\n\n\nBiostatistics-05\n\nBiostatistics-06\n\nBiostatistics-07\n\n4.5. The Problem of Multiple Comparisons\nIn section 4.2, we compared male and female green frogs from the file Pelophylax_example.sta by their body length. However, this file contains measurement results for six additional morphometric characters. Males and females can be compared not only by all seven (including body length) morphometric characters simultaneously. Along with these measurements, it is also reasonable to consider various indices—quotients of one measurement divided by another. How many of these can be calculated from these data without examining the same pair of characters twice? Six measurements can be divided by body length, five by the first of these six, and so on. Thus, it is possible to calculate 6 + 5 + 4 + 3 + 2 + 1 = 21 indices.\nAdd 21 more columns to the file Pelophylax_example.sta. The simplest way to do this is to double-click on the gray area outside the columns (to the right of the variable Cs in Fig. 4.5.1). Naturally, the Vars / Add method also works.\n[IMG_2]\nFig. 4.5.1. Double-clicking on the field to the right of the columns brought up the window for adding rows and columns. In the Add variables box, you need to specify the required number of columns\nFor the added columns, you need to specify names and define formulas by which they will be calculated. The easiest way to do this is in the All Specs window (Vars / All Specs).\n\n[IMG_3]\nFig. 4.5.2. Formulas have been added for a group of columns in the All Specs window. After pressing the OK button, the program begins to sequentially check and recalculate the formulas, as indicated in a special window. By checking the box Apply to all variables with valid expression, you can make this process automatic\nThe data in the new columns is displayed with excessive precision. You can make their display more adequate. To do this, in the column header mode (Specs), you need to change the data display format (Display format) by selecting the Number mode. After that, you can specify the required number of decimal places, for example, three.\n\n[IMG_4]\nFig. 4.5.3. You can move from one column to another using the arrow buttons (in the upper right part of the window)\nNow, it seems, we can compare the values of all metric characters and indices for males and females in our file.\n\n[IMG_5]\nFig. 4.5.4. Results of comparing twenty-eight characters by Student's t-test. Characters for which p-values less than 0.05 were recorded are highlighted in red\nThus, \"significant\" differences were recorded for three variables. Females and males differ \"significantly\" in the variability of the T/Ltc index (the ratio of tibia length to head width); for the Dp/Fm index (the ratio of the first toe length of the hind limb to femur length), the mean values characteristic of males and females differ \"significantly\"; for the Dp/T index (the ratio of the first toe length of the hind limb to tibia length), the two sexes differ both in mean value and in the level of variability.\nIn many works, including those published in reputable sources, such reasoning is presented, and far-reaching conclusions are drawn from the fact that differences between males and females were found precisely for these characters. However, we must consider what the obtained result means.\nWhat does it mean that for some character the level of statistical significance for the recorded differences is less than 0.05? It means that such differences, as those recorded for this character, could have arisen by chance during sampling from a single population no more frequently than in one case out of 20. We conducted comparisons for 28 characters; for each of them, we conducted a comparison using two tests. In total, we conducted 56 comparisons. How many times out of 56 attempts can we expect repetition of outcomes that occur with a probability of one case per twenty attempts? Two to three times.\n\n4.6. An Experiment on Obtaining False \"Significant\" Differences in Multiple Comparisons\nLet us confirm what was said in the previous section with a special experiment. Let us create a file in which multiple comparisons of samples will be performed that are known to belong to one population (random data generated according to the same distribution). Will we obtain false \"significant\" results?\nCreate a new statistics file: File / New. Specify 102 columns (Number of variables) and 100 rows (Number of cases). Click on the header of the first column and in the formula line specify: =Rnd(1). This formula generates a random variable uniformly distributed between 0 and 1 in each cell of the column. In the All Specs mode, \"stretch\" this formula to all columns, including column No. 101. In the formula line of column No. 102, enter the formula =(v101>1/2). In this column, there will be 0 (the value in brackets is incorrect) when the random variable in column 101 is less than 0.5 (i.e., with a probability of 50%), and 1 when this variable is greater than the threshold value.\n\n\n\n[IMG_6]\nFig. 4.6.1. Experimental file with data\nLet us perform for this file a comparison of rows divided into groups depending on the value in the 102nd column, by 100 characters (the 101st column cannot be used because it is related by dependence to the value in the 102nd column).\n[IMG_7]\nFig. 4.6.2. Student's t-test performed for the first 100 columns\nHaving performed the comparison of two groups by 100 characters, we will see a sufficiently high number of \"significant\" (i.e., those for which p < 0.05) results. But do not forget: our file is filled with data belonging to one general population (noise generated by the formula =Rnd(1), a uniformly distributed random variable). Recalculating the data (using the command Vars / Recalculate Spreadsheet Formulas..., the keyboard shortcut Shift+F9, or the special button labeled x=?) will result in the set of \"significant\" differences changing.\n[IMG_8]\nFig. 4.6.3. Results. Note the \"significant\" differences highlighted in red for some characters. This is an artifact of the method, a consequence of using multiple comparisons without the necessary corrections\nIn fact, if in the case of multiple comparisons we act the same way as in a single comparison, the danger of committing a Type I statistical error sharply increases: erroneously rejecting the null hypothesis when it is actually true.\n\n4.7. Corrections for Multiple Comparisons\nHow then should we act in order to avoid reaching false conclusions? The simplest solution is to use the Bonferroni correction, proposed by the Italian mathematician Carlo Emilio Bonferroni. Calculating the Bonferroni correction is very simple. You need to determine the total number of statistical hypotheses being tested (choices between the null and alternative hypothesis), which can be denoted as m. For each of these comparisons, you should use not the statistical significance level α, which was adopted for single comparisons, but the quotient of its division by the number of hypotheses: α/m. In the example considered, this means that as the critical level of statistical significance, you should take not 0.05, but 0.05/100=0.0005. As you understand, in this case, there will be no statistically significant differences in the examples we considered.\nIn the case of a sufficiently large number of multiple comparisons, applying the Bonferroni correction has a significant drawback. By lowering the critical level of statistical significance, we increase the probability of committing a Type II statistical error—accepting the null hypothesis when the alternative is true.\nA more adequate (though somewhat more complex to implement) method is one that is easiest to describe by quoting the r-analytics blog.\n\"To overcome the problems associated with the low power of the Bonferroni method, in 1978, Stur Holm (Holm 1978) proposed a much more powerful modification of it (this method is often called the Holm-Bonferroni method). This modified method is based on an algorithm that includes the following steps:\nThe initial P-values are ordered in ascending order: p(1)≤p(2)≤⋯≤p(m). These P-values correspond to the hypotheses being tested H(1),H(2),…H(m).\nIf p(1)≥α/m, all null hypotheses H(1),H(2),…H(m) are accepted and the procedure stops. Otherwise, reject hypothesis H(1) and continue.\nIf p(2)≥α/(m−1), null hypotheses H(2),H(3),…H(m) are accepted and the procedure stops. Otherwise, hypothesis H(2) is rejected and the procedure continues.\n...\nIf p(m)≥α, the null hypothesis H(m) is accepted and the procedure stops.\nThe described procedure is called step-down (English: step-down): it starts with the smallest P-value in the ordered series and sequentially \"descends\" to higher values. At each step, the corresponding p(i) value is compared with the adjusted significance level α/(m-i+1)\".\n(the last formula in the cited source contains an error; this error is corrected here; moreover, the Russian spelling of the author's surname should probably be \"Holm\" rather than \"Kholm\").\nAn example of implementing the Holm-Bonferroni procedure using Statistica is shown in Fig. 4.5.8. In this example, 11 comparisons were performed using a certain non-parametric test. The calculated levels of statistical significance (p) were recorded in the US_UB column and sorted in ascending order. In the i_US_UB column, the ordinal numbers of these comparisons are placed. In the a_US_UB column, the critical values for each of these comparisons are calculated. Finally, in the formula line of the Holm_US_UP column, an expression is entered that takes the value 1 if the recorded effect should be considered statistically significant, and 0 if the result is recognized as statistically insignificant. It can be seen that the differences recorded in comparisons No. 7, 8, and 9 would have been recognized as statistically significant if each of these comparisons were the only one, and the same differences should be considered statistically insignificant in the case where we are conducting 11 comparisons simultaneously and using the Holm-Bonferroni procedure.\n[IMG_9]\nFig. 4.7.1. Implementation of the Holm-Bonferroni procedure using Statistica\nIt should be particularly emphasized that the problem of multiple comparisons concerns not only the comparison of samples but also all cases of multiple use of statistical tests. To convince yourself of this, you can try to determine the correlation between all possible pairs of characters from the experimental file made in the previous section. The result will be the same...\nWhatever methods you use, if you are conducting several statistical tests simultaneously, you must use corrections for multiple comparisons!"}
{"translated_text": "[IMG_1]\n\n\n\n←\n\n\n\nD.A. Shabanov, M.A. Kravchenko. Statistical Analysis of Data in Zoology and Ecology\n\n\n\n→\n\n\n\n\nTopic 4. Comparison of Samples\n\nTopic 4 (continued). Multiple Comparisons\n\nTopic 4 (continued). Non-parametric Tests for Sample Comparison\n\n\n\nBiostatistics-05\n\nBiostatistics-06\n\nBiostatistics-07\n\n4.5. The Problem of Multiple Comparisons\nIn section 4.2, we compared male and female green frogs from the file Pelophylax_example.sta by their body length. However, this file contains measurement results for six additional morphometric characters. Males and females can be compared not only by all seven (including body length) morphometric characters simultaneously. Along with these measurements, it is also reasonable to consider various indices—quotients of one measurement divided by another. How many of these can be calculated from these data without examining the same pair of characters twice? Six measurements can be divided by body length, five by the first of these six, and so on. Thus, it is possible to calculate 6 + 5 + 4 + 3 + 2 + 1 = 21 indices.\nAdd 21 more columns to the file Pelophylax_example.sta. The simplest way to do this is to double-click on the gray area outside the columns (to the right of the variable Cs in Fig. 4.5.1). Naturally, the Vars / Add method also works.\n[IMG_2]\nFig. 4.5.1. Double-clicking on the field to the right of the columns brought up the window for adding rows and columns. In the Add variables box, you need to specify the required number of columns\nFor the added columns, you need to specify names and define formulas by which they will be calculated. The easiest way to do this is in the All Specs window (Vars / All Specs).\n\n[IMG_3]\nFig. 4.5.2. Formulas have been added for a group of columns in the All Specs window. After pressing the OK button, the program begins to sequentially check and recalculate the formulas, as indicated in a special window. By checking the box Apply to all variables with valid expression, you can make this process automatic\nThe data in the new columns is displayed with excessive precision. You can make their display more adequate. To do this, in the column header mode (Specs), you need to change the data display format (Display format) by selecting the Number mode. After that, you can specify the required number of decimal places, for example, three.\n\n[IMG_4]\nFig. 4.5.3. You can move from one column to another using the arrow buttons (in the upper right part of the window)\nNow, it seems, we can compare the values of all metric characters and indices for males and females in our file.\n\n[IMG_5]\nFig. 4.5.4. Results of comparing twenty-eight characters by Student's t-test. Characters for which p-values less than 0.05 were recorded are highlighted in red\nThus, \"significant\" differences were recorded for three variables. Females and males differ \"significantly\" in the variability of the T/Ltc index (the ratio of tibia length to head width); for the Dp/Fm index (the ratio of the first toe length of the hind limb to femur length), the mean values characteristic of males and females differ \"significantly\"; for the Dp/T index (the ratio of the first toe length of the hind limb to tibia length), the two sexes differ both in mean value and in the level of variability.\nIn many works, including those published in reputable sources, such reasoning is presented, and far-reaching conclusions are drawn from the fact that differences between males and females were found precisely for these characters. However, we must consider what the obtained result means.\nWhat does it mean that for some character the level of statistical significance for the recorded differences is less than 0.05? It means that such differences, as those recorded for this character, could have arisen by chance during sampling from a single population no more frequently than in one case out of 20. We conducted comparisons for 28 characters; for each of them, we conducted a comparison using two tests. In total, we conducted 56 comparisons. How many times out of 56 attempts can we expect repetition of outcomes that occur with a probability of one case per twenty attempts? Two to three times.\n\n4.6. An Experiment on Obtaining False \"Significant\" Differences in Multiple Comparisons\nLet us confirm what was said in the previous section with a special experiment. Let us create a file in which multiple comparisons of samples will be performed that are known to belong to one population (random data generated according to the same distribution). Will we obtain false \"significant\" results?\nCreate a new statistics file: File / New. Specify 102 columns (Number of variables) and 100 rows (Number of cases). Click on the header of the first column and in the formula line specify: =Rnd(1). This formula generates a random variable uniformly distributed between 0 and 1 in each cell of the column. In the All Specs mode, \"stretch\" this formula to all columns, including column No. 101. In the formula line of column No. 102, enter the formula =(v101>1/2). In this column, there will be 0 (the value in brackets is incorrect) when the random variable in column 101 is less than 0.5 (i.e., with a probability of 50%), and 1 when this variable is greater than the threshold value.\n\n\n\n[IMG_6]\nFig. 4.6.1. Experimental file with data\nLet us perform for this file a comparison of rows divided into groups depending on the value in the 102nd column, by 100 characters (the 101st column cannot be used because it is related by dependence to the value in the 102nd column).\n[IMG_7]\nFig. 4.6.2. Student's t-test performed for the first 100 columns\nHaving performed the comparison of two groups by 100 characters, we will see a sufficiently high number of \"significant\" (i.e., those for which p < 0.05) results. But do not forget: our file is filled with data belonging to one general population (noise generated by the formula =Rnd(1), a uniformly distributed random variable). Recalculating the data (using the command Vars / Recalculate Spreadsheet Formulas..., the keyboard shortcut Shift+F9, or the special button labeled x=?) will result in the set of \"significant\" differences changing.\n[IMG_8]\nFig. 4.6.3. Results. Note the \"significant\" differences highlighted in red for some characters. This is an artifact of the method, a consequence of using multiple comparisons without the necessary corrections\nIn fact, if in the case of multiple comparisons we act the same way as in a single comparison, the danger of committing a Type I statistical error sharply increases: erroneously rejecting the null hypothesis when it is actually true.\n\n4.7. Corrections for Multiple Comparisons\nHow then should we act in order to avoid reaching false conclusions? The simplest solution is to use the Bonferroni correction, proposed by the Italian mathematician Carlo Emilio Bonferroni. Calculating the Bonferroni correction is very simple. You need to determine the total number of statistical hypotheses being tested (choices between the null and alternative hypothesis), which can be denoted as m. For each of these comparisons, you should use not the statistical significance level α, which was adopted for single comparisons, but the quotient of its division by the number of hypotheses: α/m. In the example considered, this means that as the critical level of statistical significance, you should take not 0.05, but 0.05/100=0.0005. As you understand, in this case, there will be no statistically significant differences in the examples we considered.\nIn the case of a sufficiently large number of multiple comparisons, applying the Bonferroni correction has a significant drawback. By lowering the critical level of statistical significance, we increase the probability of committing a Type II statistical error—accepting the null hypothesis when the alternative is true.\nA more adequate (though somewhat more complex to implement) method is one that is easiest to describe by quoting the r-analytics blog.\n\"To overcome the problems associated with the low power of the Bonferroni method, in 1978, Stur Holm (Holm 1978) proposed a much more powerful modification of it (this method is often called the Holm-Bonferroni method). This modified method is based on an algorithm that includes the following steps:\nThe initial P-values are ordered in ascending order: p(1)≤p(2)≤⋯≤p(m). These P-values correspond to the hypotheses being tested H(1),H(2),…H(m).\nIf p(1)≥α/m, all null hypotheses H(1),H(2),…H(m) are accepted and the procedure stops. Otherwise, reject hypothesis H(1) and continue.\nIf p(2)≥α/(m−1), null hypotheses H(2),H(3),…H(m) are accepted and the procedure stops. Otherwise, hypothesis H(2) is rejected and the procedure continues.\n...\nIf p(m)≥α, the null hypothesis H(m) is accepted and the procedure stops.\nThe described procedure is called step-down (English: step-down): it starts with the smallest P-value in the ordered series and sequentially \"descends\" to higher values. At each step, the corresponding p(i) value is compared with the adjusted significance level α/(m-i+1)\".\n(the last formula in the cited source contains an error; this error is corrected here; moreover, the Russian spelling of the author's surname should probably be \"Holm\" rather than \"Kholm\").\nAn example of implementing the Holm-Bonferroni procedure using Statistica is shown in Fig. 4.5.8. In this example, 11 comparisons were performed using a certain non-parametric test. The calculated levels of statistical significance (p) were recorded in the US_UB column and sorted in ascending order. In the i_US_UB column, the ordinal numbers of these comparisons are placed. In the a_US_UB column, the critical values for each of these comparisons are calculated. Finally, in the formula line of the Holm_US_UP column, an expression is entered that takes the value 1 if the recorded effect should be considered statistically significant, and 0 if the result is recognized as statistically insignificant. It can be seen that the differences recorded in comparisons No. 7, 8, and 9 would have been recognized as statistically significant if each of these comparisons were the only one, and the same differences should be considered statistically insignificant in the case where we are conducting 11 comparisons simultaneously and using the Holm-Bonferroni procedure.\n[IMG_9]\nFig. 4.7.1. Implementation of the Holm-Bonferroni procedure using Statistica\nIt should be particularly emphasized that the problem of multiple comparisons concerns not only the comparison of samples but also all cases of multiple use of statistical tests. To convince yourself of this, you can try to determine the correlation between all possible pairs of characters from the experimental file made in the previous section. The result will be the same...\nWhatever methods you use, if you are conducting several statistical tests simultaneously, you must use corrections for multiple comparisons!"}
{"translated_text": "[IMG_1]\n\n\n\n←\n\n\n\nD.A. Shabanov, M.A. Kravchenko. Statistical Analysis of Data in Zoology and Ecology\n\n\n\n→\n\n\n\n\nTopic 4. Comparison of Samples\n\nTopic 4 (continued). Multiple Comparisons\n\nTopic 4 (continued). Non-parametric Tests for Sample Comparison\n\n\n\nBiostatistics-05\n\nBiostatistics-06\n\nBiostatistics-07\n\n4.5. The Problem of Multiple Comparisons\nIn section 4.2, we compared male and female green frogs from the file Pelophylax_example.sta by their body length. However, this file contains measurement results for six additional morphometric characters. Males and females can be compared not only by all seven (including body length) morphometric characters simultaneously. Along with these measurements, it is also reasonable to consider various indices—quotients of one measurement divided by another. How many of these can be calculated from these data without examining the same pair of characters twice? Six measurements can be divided by body length, five by the first of these six, and so on. Thus, it is possible to calculate 6 + 5 + 4 + 3 + 2 + 1 = 21 indices.\nAdd 21 more columns to the file Pelophylax_example.sta. The simplest way to do this is to double-click on the gray area outside the columns (to the right of the variable Cs in Fig. 4.5.1). Naturally, the Vars / Add method also works.\n[IMG_2]\nFig. 4.5.1. Double-clicking on the field to the right of the columns brought up the window for adding rows and columns. In the Add variables box, you need to specify the required number of columns\nFor the added columns, you need to specify names and define formulas by which they will be calculated. The easiest way to do this is in the All Specs window (Vars / All Specs).\n\n[IMG_3]\nFig. 4.5.2. Formulas have been added for a group of columns in the All Specs window. After pressing the OK button, the program begins to sequentially check and recalculate the formulas, as indicated in a special window. By checking the box Apply to all variables with valid expression, you can make this process automatic\nThe data in the new columns is displayed with excessive precision. You can make their display more adequate. To do this, in the column header mode (Specs), you need to change the data display format (Display format) by selecting the Number mode. After that, you can specify the required number of decimal places, for example, three.\n\n[IMG_4]\nFig. 4.5.3. You can move from one column to another using the arrow buttons (in the upper right part of the window)\nNow, it seems, we can compare the values of all metric characters and indices for males and females in our file.\n\n[IMG_5]\nFig. 4.5.4. Results of comparing twenty-eight characters by Student's t-test. Characters for which p-values less than 0.05 were recorded are highlighted in red\nThus, \"significant\" differences were recorded for three variables. Females and males differ \"significantly\" in the variability of the T/Ltc index (the ratio of tibia length to head width); for the Dp/Fm index (the ratio of the first toe length of the hind limb to femur length), the mean values characteristic of males and females differ \"significantly\"; for the Dp/T index (the ratio of the first toe length of the hind limb to tibia length), the two sexes differ both in mean value and in the level of variability.\nIn many works, including those published in reputable sources, such reasoning is presented, and far-reaching conclusions are drawn from the fact that differences between males and females were found precisely for these characters. However, we must consider what the obtained result means.\nWhat does it mean that for some character the level of statistical significance for the recorded differences is less than 0.05? It means that such differences, as those recorded for this character, could have arisen by chance during sampling from a single population no more frequently than in one case out of 20. We conducted comparisons for 28 characters; for each of them, we conducted a comparison using two tests. In total, we conducted 56 comparisons. How many times out of 56 attempts can we expect repetition of outcomes that occur with a probability of one case per twenty attempts? Two to three times.\n\n4.6. An Experiment on Obtaining False \"Significant\" Differences in Multiple Comparisons\nLet us confirm what was said in the previous section with a special experiment. Let us create a file in which multiple comparisons of samples will be performed that are known to belong to one population (random data generated according to the same distribution). Will we obtain false \"significant\" results?\nCreate a new statistics file: File / New. Specify 102 columns (Number of variables) and 100 rows (Number of cases). Click on the header of the first column and in the formula line specify: =Rnd(1). This formula generates a random variable uniformly distributed between 0 and 1 in each cell of the column. In the All Specs mode, \"stretch\" this formula to all columns, including column No. 101. In the formula line of column No. 102, enter the formula =(v101>1/2). In this column, there will be 0 (the value in brackets is incorrect) when the random variable in column 101 is less than 0.5 (i.e., with a probability of 50%), and 1 when this variable is greater than the threshold value.\n\n\n\n[IMG_6]\nFig. 4.6.1. Experimental file with data\nLet us perform for this file a comparison of rows divided into groups depending on the value in the 102nd column, by 100 characters (the 101st column cannot be used because it is related by dependence to the value in the 102nd column).\n[IMG_7]\nFig. 4.6.2. Student's t-test performed for the first 100 columns\nHaving performed the comparison of two groups by 100 characters, we will see a sufficiently high number of \"significant\" (i.e., those for which p < 0.05) results. But do not forget: our file is filled with data belonging to one general population (noise generated by the formula =Rnd(1), a uniformly distributed random variable). Recalculating the data (using the command Vars / Recalculate Spreadsheet Formulas..., the keyboard shortcut Shift+F9, or the special button labeled x=?) will result in the set of \"significant\" differences changing.\n[IMG_8]\nFig. 4.6.3. Results. Note the \"significant\" differences highlighted in red for some characters. This is an artifact of the method, a consequence of using multiple comparisons without the necessary corrections\nIn fact, if in the case of multiple comparisons we act the same way as in a single comparison, the danger of committing a Type I statistical error sharply increases: erroneously rejecting the null hypothesis when it is actually true.\n\n4.7. Corrections for Multiple Comparisons\nHow then should we act in order to avoid reaching false conclusions? The simplest solution is to use the Bonferroni correction, proposed by the Italian mathematician Carlo Emilio Bonferroni. Calculating the Bonferroni correction is very simple. You need to determine the total number of statistical hypotheses being tested (choices between the null and alternative hypothesis), which can be denoted as m. For each of these comparisons, you should use not the statistical significance level α, which was adopted for single comparisons, but the quotient of its division by the number of hypotheses: α/m. In the example considered, this means that as the critical level of statistical significance, you should take not 0.05, but 0.05/100=0.0005. As you understand, in this case, there will be no statistically significant differences in the examples we considered.\nIn the case of a sufficiently large number of multiple comparisons, applying the Bonferroni correction has a significant drawback. By lowering the critical level of statistical significance, we increase the probability of committing a Type II statistical error—accepting the null hypothesis when the alternative is true.\nA more adequate (though somewhat more complex to implement) method is one that is easiest to describe by quoting the r-analytics blog.\n\"To overcome the problems associated with the low power of the Bonferroni method, in 1978, Stur Holm (Holm 1978) proposed a much more powerful modification of it (this method is often called the Holm-Bonferroni method). This modified method is based on an algorithm that includes the following steps:\nThe initial P-values are ordered in ascending order: p(1)≤p(2)≤⋯≤p(m). These P-values correspond to the hypotheses being tested H(1),H(2),…H(m).\nIf p(1)≥α/m, all null hypotheses H(1),H(2),…H(m) are accepted and the procedure stops. Otherwise, reject hypothesis H(1) and continue.\nIf p(2)≥α/(m−1), null hypotheses H(2),H(3),…H(m) are accepted and the procedure stops. Otherwise, hypothesis H(2) is rejected and the procedure continues.\n...\nIf p(m)≥α, the null hypothesis H(m) is accepted and the procedure stops.\nThe described procedure is called step-down (English: step-down): it starts with the smallest P-value in the ordered series and sequentially \"descends\" to higher values. At each step, the corresponding p(i) value is compared with the adjusted significance level α/(m-i+1)\".\n(the last formula in the cited source contains an error; this error is corrected here; moreover, the Russian spelling of the author's surname should probably be \"Holm\" rather than \"Kholm\").\nAn example of implementing the Holm-Bonferroni procedure using Statistica is shown in Fig. 4.5.8. In this example, 11 comparisons were performed using a certain non-parametric test. The calculated levels of statistical significance (p) were recorded in the US_UB column and sorted in ascending order. In the i_US_UB column, the ordinal numbers of these comparisons are placed. In the a_US_UB column, the critical values for each of these comparisons are calculated. Finally, in the formula line of the Holm_US_UP column, an expression is entered that takes the value 1 if the recorded effect should be considered statistically significant, and 0 if the result is recognized as statistically insignificant. It can be seen that the differences recorded in comparisons No. 7, 8, and 9 would have been recognized as statistically significant if each of these comparisons were the only one, and the same differences should be considered statistically insignificant in the case where we are conducting 11 comparisons simultaneously and using the Holm-Bonferroni procedure.\n[IMG_9]\nFig. 4.7.1. Implementation of the Holm-Bonferroni procedure using Statistica\nIt should be particularly emphasized that the problem of multiple comparisons concerns not only the comparison of samples but also all cases of multiple use of statistical tests. To convince yourself of this, you can try to determine the correlation between all possible pairs of characters from the experimental file made in the previous section. The result will be the same...\nWhatever methods you use, if you are conducting several statistical tests simultaneously, you must use corrections for multiple comparisons!"}