Current Referee Status

Referee status: Not Approved, Not Approved, Approved, Approved with reservations

Article Reports

Referee Report: https://doi.org/10.5256/f1000research.7499.r10159, Daniel Lakens, School of Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands, 30 Oct 2015, version 1, 2 not approved

Referee Comment: <b>Cyril Pernet</b>; <i>Posted: 06 Jul 2016</i>; <i>“investigate if an effect is likely” – ambiguous statement. I think you mean, whether the observed DATA is probable, assuming there is no effect?</i> This sentence was changed, following as well the other reviewer, to ‘null hypothesis significance testing is the statistical method of choice in biological, biomedical and social sciences to investigate if an effect is likely, even though it actually tests whether the observed data are probable, assuming there is no effect’ &nbsp; <i>The Fisher (1959) reference is not correct – Fischer developed his method much earlier.</i> Changed, refers to Fisher 1925 &nbsp; <i>“This p-value thus reflects the conditional probability of achieving the observed outcome or larger, p(Obs|H0)” – please add 'assuming the null-hypothesis is true'. “p(Obs|H0)” – explain this notation for novices.</i> I changed a little the sentence structure, which should make explicit that this is the condition probability. &nbsp; <i>“Following Fisher, the smaller the p-value, the greater the likelihood that the null hypothesis is false.”&nbsp; This is wrong, and any statement about this needs to be much more precise. I would suggest direct quotes.</i> This sentence has been removed &nbsp; <i>“there is something in the data that deserves further investigation” –unclear sentence. “The reason for this” – unclear what ‘this’ refers to.</i> This has been changed to ‘[…] to decide whether the evidence is worth additional investigation and/or replication (Fisher, 1971 p13)’ &nbsp; <i>“not the probability of the null hypothesis of being true, p(H0)” – second of can be removed?</i> my mistake – the sentence structure is now ‘ <i>not the probability of the null hypothesis p(H0), of being true,’ ; hope this makes more sense (and this way refers back to p(Obs&gt;t|H0)</i> &nbsp; <i>“Any interpretation of the p-value in relation to the effect under study (strength, reliability, probability) is indeed wrong, since the p-value is conditioned on H0” &nbsp;- incorrect. A big problem is that it depends on the sample size, and that the probability of a theory depends on the prior.</i> Fair enough – my point was to stress the fact that p value and effect size or H1 have very little in common, but yes that the part in common has to do with sample size. I left the conditioning on H0 but also point out the dependency on sample size. &nbsp; <i>“If there is no effect, we should replicate the absence of effect with a probability equal to 1-p.” I don’t understand this, but I think it is incorrect.</i> Removed &nbsp; <i>“The total probability of false positives can also be obtained by aggregating results (Ioannidis, 2005).” Unclear, and probably incorrect.</i> Removed &nbsp; <i>“By failing to reject, we simply continue to assume that H0 is true, which implies that one cannot, from a nonsignificant result, argue against a theory” – according to which theory? From a NP perspective, you can ACT as if the theory is false.</i> The whole paragraph was changed to reflect a more philosophical take on scientific induction/reasoning. I hope this is clearer. &nbsp; <i>“(Lakens &amp; Evers, 2014”) – we are not the original source, which should be cited instead.</i> done &nbsp; <i>“ Typically, if a CI includes 0, we cannot reject H0.”&nbsp; - when would this not be the case? This assumes a CI of 1-alpha. “If a critical null region is specified rather than a single point estimate, for instance [-2 +2] and the CI is included within the critical null region, then H0 can be accepted.” – you mean practically, or formally? I’m pretty sure only the former.</i> Changed to refer to equivalence testing &nbsp; <i>The section on ‘The (correct) use of NHST’ seems to conclude only Bayesian statistics should be used. I don’t really agree.</i> I rewrote this, as to show frequentist analysis can be used&nbsp; - I’m trying to sell Bayes more than any other approach. &nbsp; <i><b>“</b>we can always argue that effect size, power, etc. must be reported.” – which power? Post-hoc power? Surely not? Other types are unknown. So what do you mean? The recommendation on what to report remains vague, and it is unclear why what should be reported.</i> I’m arguing we should report it all, that’s why there is no exhausting list – I can if needed.

Referee Report: https://doi.org/10.5256/f1000research.7499.r11036, Marcel ALM van Assen, Department of Methodology and Statistics, Tilburgh University, Tilburg, The Netherlands, 10 Nov 2015, version 1, 2 not approved

Referee Comment: <b>Cyril Pernet</b>; <i>Posted: 06 Jul 2016</i>; <i>Null hypothesis significance testing (NHST) is a difficult topic, with misunderstandings arising easily. Many texts, including basic statistics books, deal with the topic, and attempt to explain it to students and anyone else interested. I would refer to a good basic text book, for a detailed explanation of NHST, or to a specialized article when wishing an explaining the background of NHST. So, what is the added value of a new text on NHST? In any case, the added value should be described at the start of this text. Moreover, the topic is so delicate and difficult that errors, misinterpretations, and disagreements are easy. I attempted to show this by giving comments to many sentences in the text.</i> The idea of this short review was to point to common interpretation errors (stressing again and again that we are under H0) being in using p-values or CI, and also proposing reporting practices to avoid bias. This is now stated at the end of abstract. Regarding text books, it is clear that many fail to clearly distinguish Fisher/Pearson/NHST, see Glinet et al (2012) J. Exp Education 71, 83-92. If you have 1 or 2 in mind that you know to be good, I’m happy to include them. &nbsp; <i>Abstract: “null hypothesis significance testing is the statistical method of choice in biological, biomedical and social sciences to investigate if an effect is likely”. No, NHST is the method to test the hypothesis of no effect.</i> I agree – yet people use it to investigate (not test) if an effect is likely. The issue here is wording. What about adding this distinction at the end of the sentence?: ‘null hypothesis significance testing is the statistical method of choice in biological, biomedical and social sciences used to investigate if an effect is likely, even though it actually tests for the hypothesis of no effect’. &nbsp; <i>Intro: “Null hypothesis significance testing (NHST) is a method of statistical inference by which an observation is tested against a hypothesis of no effect or no relationship.” What is an ‘observation’? NHST is difficult to describe in one sentence, particularly here. I would skip this sentence entirely, here.</i> I think a definition is needed, as it offers a starting point. What about the following: ‘NHST is a method of statistical inference by which an experimental factor is tested against a hypothesis of no effect or no relationship based on a given observation’ &nbsp; <i>Section on Fisher; also explain the one-tailed test.</i> <i>Section on Fisher; p(Obs|H0) does not reflect the verbal definition (the ‘or more extreme’ part).</i> <i>Section on Fisher; use a reference and citation to Fisher’s interpretation of the p-value</i> <i>Section on Fisher; “This was however only intended to be used as an indication that there is something in the data that deserves further investigation. The reason for this is that only H0 is tested whilst the effect under study is not itself being investigated.” First sentence, can you give a reference? Many people say a lot about Fisher’s intentions, but the good man is dead and cannot reply… Second sentence is a bit awkward, because the effect <u>is</u> investigated in a way, by testing the H0.</i> The section on Fisher has been modified (more or less) as suggested: (1) avoiding talking about one or two tailed tests (2) updating for p(Obs≥t|H0) and (3) referring to Fisher more explicitly (ie pages from articles and book) ; I cannot tell his intentions but these quotes leave little space to alternative interpretations. &nbsp; <i>Section on p-value; Layout and structure can be improved greatly, by first again stating what the p-value is, and then statement by statement, what it is not, using separate lines for each statement. Consider adding that the p-value is randomly distributed under H0 (if all the assumptions of the test are met), and that under H1 the p-value is a function of population effect size and N; the larger each is, the smaller the p-value generally is.</i> Done &nbsp; <i>Skip the sentence “If there is no effect, we should replicate the absence of effect with a probability equal to 1-p”. Not insightful, and you did not discuss the concept ‘replicate’ (and do not need to).</i> Done &nbsp; <i>Skip the sentence “The total probability of false positives can also be obtained by aggregating results (</i> <i>Ioannidis, 2005</i> <i>).” Not strongly related to p-values, and introduces unnecessary concepts ‘false positives’ (perhaps later useful) and ‘aggregation’.</i> Done &nbsp; <i>Consider deleting; “If there is an effect however, the probability to replicate is a function of the (unknown) population effect size with no good way to know this from a single experiment (</i> <i>Killeen, 2005</i> <i>).”</i> Done &nbsp; <i>The following sentence; “&nbsp;Finally, a (small) p-value&nbsp;is not an indication favouring a hypothesis. A low p-value indicates a misfit of the null hypothesis to the data and cannot be taken as evidence in favour of a specific alternative hypothesis more than any other possible alternatives such as measurement error and selection bias (Gelman, 2013).” is surely not mainstream thinking about NHST; I would surely delete that sentence. In NHST, a p-value is used for testing the H0. Why did you not yet discuss significance level? Yes, before discussing what is not a p-value, I would explain NHST (i.e., what it is and how it is used).&nbsp;</i> <i>Also the next sentence “The more (a priori) implausible the alternative hypothesis, the greater the chance that a finding is a false alarm (Krzywinski &amp; Altman, 2013;&nbsp;Nuzzo, 2014).“ is not fully clear to me. This is a Bayesian statement. In NHST, no likelihoods are attributed to hypotheses; the reasoning is “IF H0 is true, then…”.</i> The reasoning here is as you state yourself, part 1: ‘a p-value is used for testing the H0; and part 2: ‘no likelihoods are attributed to hypotheses’ it follows we cannot favour a hypothesis. It might seems contentious but this is the case that all we can is to reject the null – how could we favour a specific alternative hypothesis from there? This is explored further down the manuscript (and I now point to that) – note that we do not need to be Bayesian to favour a specific H1, all I’m saying is this cannot be attained with a p-value. &nbsp; <i>Last sentence: “As&nbsp;</i> <i>Nickerson (2000)</i> <i>&nbsp;puts it ‘theory corroboration requires the testing of multiple predictions because the chance of getting statistically significant results for the wrong reasons in any given case is high’.” What is relation of this sentence to the contents of this section, precisely?</i> The point was to emphasise that a p value is not there to tell us a given H1 is true and can only be achieved through multiple predictions and experiments. I deleted it for clarity. &nbsp; <i>Next section: “For instance, we can estimate that the probability of a given F value to be in the critical interval [+2 +∞] is less than 5%” This depends on the degrees of freedom.</i> This sentence has been removed &nbsp; <i>“When there is no effect (H0 is true), the erroneous rejection of H0 is known as type I error and is equal to the p-value.” Strange sentence. The Type I error is the probability of erroneously rejecting the H0 (so, when it is true). The p-value is … well, you explained it before; it surely does not equal the Type I error.</i> Indeed, you are right and I have modified the text accordingly. When there is no effect (H0 is true), the erroneous rejection of H0 is known as type 1 error. Importantly, the type 1 error rate, or alpha value is determined a priori. It is a common mistake but the level of significance (for a given sample) is not the same as the frequency of acceptance alpha found on repeated sampling (Fisher, 1955). &nbsp; <i>Consider adding a figure explaining the distinction between Fisher’s logic and that of Neyman and Pearson.</i> A figure is now presented – with levels of acceptance, critical region, level of significance and p-value. &nbsp; “ <i>When the test statistics falls outside the critical region(s)” What is outside? “There is a profound difference between accepting the null hypothesis and simply failing to reject it (Killeen, 2005)” I agree with you, but perhaps you may add that some statisticians simply define “accept H0’” as obtaining a p-value larger than the significance level. Did you already discuss the significance level, and it’s mostly used values? “To accept or reject equally the null hypothesis, Bayesian approaches (Dienes, 2014;&nbsp;Kruschke, 2011) or confidence intervals must be used.” Is ‘reject equally’ appropriate English? Also using Cis, one cannot accept the H0.</i> I should have clarified further here – as I was having in mind tests of equivalence. To clarify, I simply states now: ‘To accept the null hypothesis, tests of equivalence or Bayesian approaches must be used.’ &nbsp; <i>Do you start discussing alpha only in the context of Cis?</i> It is now presented in the paragraph before. &nbsp; <i>“CI also indicates the precision of the estimate of effect size, but unless using a percentile bootstrap approach, they require assumptions about distributions which can lead to serious biases in particular regarding the symmetry and width of the intervals (Wilcox, 2012).” Too difficult, using new concepts. Consider deleting.</i> Done &nbsp; <i>“Assuming the CI (a)symmetry and width are correct, this gives some indication about the likelihood that a similar value can be observed in future studies, with 95% CI giving about 83% chance of replication success (Lakens &amp; Evers, 2014).” This statement is, in general, completely false. It very much depends on the sample sizes of both studies. If the replication study has a much, much, much larger N, then the probability that the original CI will contain the effect size of the replication approaches (1-alpha)*100%. If the original study has a much, much, much larger N, then the probability that the original Ci will contain the effect size of the replication study approaches 0%.</i> Yes, you are right, I completely overlooked this problem. The corrected sentence (with more accurate ref) is now <i>“Assuming the CI (a)symmetry and width are correct, this gives some indication about the likelihood that a similar value can be observed in future studies. For future studies of the same sample size, 95% CI giving about 83% chance of replication success (Cumming and Mallardet, 2006). If sample sizes differ between studies, CI do not however warranty any a priori coverage”. </i> &nbsp; <i>“Finally, contrary to p-values, CI can be used to accept H0. Typically, if a CI includes 0, we cannot reject H0. If a critical null region is specified rather than a single point estimate, for instance [-2 +2] and the CI is included within the critical null region, then H0 can be accepted. Importantly, the critical region must be specified a priori and cannot be determined from the data themselves.” No. H0 cannot be accepted with Cis.</i> Again, I had in mind equivalence testing, but in both cases you are right we can only reject and I therefore removed that sentence. &nbsp; <i>“The (posterior) probability of an effect can however not be obtained using a frequentist framework.” Frequentist framework? You did not discuss that, yet.</i> Removed &nbsp; <i>“X% of times the CI obtained will contain the same parameter value”. The same? True, you mean? “e.g. X% of the times the CI contains the same mean” I do not understand; which mean? “The alpha value has the same interpretation as when using H0, i.e. we accept that 1-alpha CI are wrong in alpha percent of the times.&nbsp;“ What do you mean, CI are wrong? Consider rephrasing. “To make a statement about the probability of a parameter of interest, likelihood intervals (maximum likelihood) and credibility intervals (Bayes) are better suited.” ML gives the likelihood of the data given the parameter, not the other way around.</i> corrected &nbsp; “ <i>Many of the disagreements are not on the method itself but on its use.” Bayesians may disagree.</i> removed &nbsp; <i>“If the goal is to establish the likelihood of an effect and/or establish a pattern of order, because both requires ruling out equivalence, then NHST is a good tool (Frick, 1996)” NHST does not provide evidence on the likelihood of an effect.</i> “If the goal is to establish some quantitative values, then NHST is not the method of choice.” P-values are also quantitative… this is not a precise sentence. And NHST may be used in combination with effect size estimation (this is even recommended by, e.g., the American Psychological Association (APA)). Yes, p-values must be interpreted in context with effect size, but this is not what people do. The point here is to be pragmatic, does and don’t. The sentence was changed. &nbsp; <i>“Because results are conditioned on H0, NHST cannot be used to establish beliefs.” It can reinforce some beliefs, e.g., if H0 or any other hypothesis, is true. “To estimate the probability of a hypothesis, a Bayesian analysis is a better alternative.” It is the only alternative?</i> Not for testing, but for probability, I am not aware of anything else. &nbsp; <i>“Note however that even when a specific quantitative prediction from a hypothesis is shown to be true (typically testing H1 using Bayes), it does not prove the hypothesis itself, it only adds to its plausibility.” How can we show something is true?</i> Cumulative evidence is, in my opinion, the only way to show it. Even in hard science like physics multiple experiments. In the recent CERN study on finding Higgs bosons, 2 different and complementary experiments ran in parallel – and the cumulative evidence was taken as a proof of the true existence of Higgs bosons.

Referee Report: https://doi.org/10.5256/f1000research.10487.r17400, Stephen J. Senn, Luxembourg Institute of Health, Strassen, L-1445, Luxembourg, 03 Nov 2016, version 3, 1 approved, 1 approved with reservations, 2 not approved

Referee Report: https://doi.org/10.5256/f1000research.10487.r19282, Dorothy Vera Margaret Bishop, Department of Experimental Psychology, University of Oxford, Oxford, UK, 03 Feb 2017, version 3, 1 approved, 1 approved with reservations, 2 not approved

Referee Comment: <b>Cyril Pernet</b>; <i>Posted: 19 Sep 2017</i>; I wondered about changing the focus slightly and modifying the title to reflect this to say something like: Null hypothesis significance testing: a guide to commonly misunderstood concepts and recommendations for good practice As one previous reviewer noted, it’s questionable that there is a need for a tutorial introduction, and the limited length of this article does not lend itself to a full explanation. So it might be better to just focus on explaining as clearly as possible the problems people have had in interpreting key concepts. I think a title that made it clear this was the content would be more appealing than the current one. &nbsp; <i>Thank you for the suggestion – you indeed saw the intention behind the ‘tutorial’ style of the paper. &nbsp;</i> P 3, col 1, para 3, last sentence. Although statisticians always emphasise the arbitrary nature of p &lt; .05, we all know that in practice authors who use other values are likely to have their analyses queried. I wondered whether it would be useful here to note that in some disciplines different cutoffs are traditional, e.g. particle physics. Or you could cite David Colquhoun’s paper in which he recommends using p &lt; .001 (http://rsos.royalsocietypublishing.org/content/1/3/140216)&nbsp;- just to be clear that the traditional p &lt; .05 has been challenged. <i>I have added a sentence on this citing Colquhoun 2014 and the new Benjamin 2017 on using .005. &nbsp;</i> Having read the section on the Fisher approach and Neyman-Pearson approach I felt confused. I have to confess that despite years of doing stats, this distinction had eluded me (which is why I am a good target reader), but I wasn’t really entirely enlightened after reading this. As I understand it, I have been brought up doing null hypothesis testing, so am adopting a Fisher approach. But I also talk about setting alpha to .05, and understand that to come from the Neyman-Pearson approach. If I have understood this correctly, these do amount to the same thing (as the author states, they are assimilated in practice), but we are then told this is a ‘common mistake’. But the explanation of the difference was hard to follow and I found myself wondering whether it would actually make any difference to what I did in practice. In order to understand the last sentence before ‘Acceptance or rejection of H0’ I would need some good analogy. Maybe it would be possible to explain this better with the tried-and-tested example of tossing a coin. So in Fisher approach, you do a number of coin tosses to test whether the coin is unbiased (Null hypothesis); you can then work out p as the probability of the null given a specific set of observations, which is the p—value. What I can’t work out is how you would explain the alpha from Neyman-Pearson in the same way (though I can see from Figure 1 that with N-P you could test an alternative hypothesis, such as the idea that the coin would be heads 75% of the time). <i>I agree that this point is always hard to appreciate, especially because it seems like in practice it makes little difference. I added a paragraph but using reaction times rather than a coin toss – thanks for the suggestion. &nbsp;</i> The section on acceptance or rejection of H0 was good, though I found the first sentence a bit opaque and wondered if it could be made clearer. Also I wondered if this rewording would be accurate (as it is clearer to me): instead of: ‘By failing to reject, we simply continue to assume that H0 is true, which implies that one cannot….’ have ‘In failing to reject, we do not assume that H0 is true; one cannot argue against a theory from a non-significant result.’ I felt most readers would be interested to read about tests of equivalence and Bayesian approaches, but many would be unfamiliar with these and might like to see an example of how they work in practice – if space permitted. <i>Added an example based on new table 1, following figure 1 – giving CI, equivalence tests and Bayes Factor (with refs to easy to use tools) &nbsp;</i> Confidence intervals: I simply could not understand the first sentence – I wondered what was meant by ‘builds’ here. I understand about difficulties in comparing CI across studies when sample sizes differ, but I did not find the last sentence on p 4 easy to understand. <i>Changed builds to constructs (this simply means they are something we build) and added that the implication that probability coverage is not warranty when sample size change, is that we cannot compare CI. &nbsp;</i> P 5: The sentence starting: ‘The alpha value has the same interpretation’ was also hard to understand, especially the term ‘1-alpha CI’. Here too I felt some concrete illustration might be helpful to the reader. And again, I also found the reference to Bayesian intervals tantalising – I think many readers won’t know how to compute these and something like a figure comparing a traditional CI with a Bayesian interval and giving a source for those who want to read on would be very helpful. The reference to ‘credible intervals’ in the penultimate paragraph is very unclear and needs a supporting reference – most readers will not be familiar with this concept. <i>I changed ‘</i> <i>i.e. we accept that 1-alpha CI are wrong in alpha percent of the times in the long run’ to ‘, ‘e.g. a 95% CI is wrong in 5% of the times in the long run (i.e. if we repeat the experiment many times).’ – for Bayesian intervals I simply re-cited </i>Morey &amp; Rouder, 2011. &nbsp; Typos &nbsp; P 3, col 1, para 2, line 2; “allows us to compute” P 3, col 2, para 2, ‘probability of replicating’ P 3, col 2, para 2, line 4 ‘informative about’ P 3, col 2, para 4, line 2 delete ‘of’ P 3, col 2, para 5, line 9 – ‘conditioned’ is either wrong or too technical here: would ‘based’ be acceptable as alternative wording P 3, col 2, para 5, line 13 ‘This dichotomisation allows one to distinguish’ P 3, col 2, para 5, last sentence, delete ‘Alternatively’. P 3, col 2, last para line 2 ‘first’ P 4, col 2, para 2, last sentence is hard to understand; not sure if this is better: ‘If sample sizes differ between studies, the distribution of CIs cannot be specified a priori’ &nbsp; <i>It is not the CI cannot be specified, it’s that the interval is not predictive of anything anymore! I changed it to ‘If sample </i> <i>sizes, however,</i> <i> differ between studies, there is no warranty that a CI from one study will be true at the rate alpha in a different study, which implies that </i> <i>CI</i> <i> cannot be compared across studies at this is rarely the same sample sizes’</i> &nbsp; P 5, col 1, para 2, ‘a pattern of order’ – I did not understand what was meant by this &nbsp; <i>I added (i.e. establish that A &gt; B) – we test that conditions are ordered, but without further specification of the probability of that effect nor its size</i> P 5, col 1, para 2, last sentence unclear: possible rewording: “If the goal is to test the size of an effect then NHST is not the method of choice, since testing can only reject the null hypothesis.’ (??) &nbsp; <i>Yes it works – thx</i> P 5, col 1, para 3, line 1 delete ‘that’ P 5, col 1, para 3, line 3 ‘on’ -&gt; ‘by’ P 5, col 2, para 1, line 4 , rather than ‘Here I propose to adopt’ I suggest ‘I recommend adopting’ P 5, col 2, para 1, line 13 ‘with’ -&gt; ‘by’ P 5, col 2, para 1 – recommend deleting last sentence P 5, col 2, para 2, line 2 ‘consider’ -&gt; ‘anticipate’ P 5, col 2, para 2, delete ‘should always be included’ P 5, col 2, para 2, ‘type one’ -&gt; ‘Type I’&nbsp; &nbsp; <i>Typos fixed, and suggestions accepted – thanks for that.</i> &nbsp;

Grant Information: The author(s) declared that no grants were involved in supporting this work

Copyright: This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.