#false-positive

Když si #Google splete #Vanoce a #Halloween

...děsí uživatele #GoogleSearchConsole tím, že ne-#index.uje celý #web

UK’s Met Police’s facial recognition technology isn’t, 98% of the time

The security camera commissioner has said he is concerned about quantity of false positives Getty

‘Intrinsically Orwellian’ systems must be scrapped, campaigners say as biometrics commissioner brands them ‘not yet fit for use’

14 May 2018 | Jon Sharman | Independent

Facial recognitionsoftware used by the UK’s biggest police force has returned false positives in more than 98 per cent of…

How to: Malwarebytes gives trojan warning for basic C# "Hello World!" program

Malwarebytes gives trojan warning for basic C# "Hello World!" program

Basically, I just ran a scan of my computer with Malwarebytes (updated the definitions before running), and it said my “helloworld” program written in C# has a trojan.

I know for a fact this is a false positive, as I only wrote the program 2-3 days ago and followed a small tutorial website to make the program that I trust. I…

One of the few things that I recently come across and grow a big interest on is about how the accountability of scientific research findings has been criticized widely (and harshly, I must say, according to a few papers and articles that I have read).

Why is this important, of course, have everything to do with maintaining the level of integrity and credibility of scientific field as well as protecting the trust of public for any scientific finding.

Criticisms flew in from both directions –inside the science field itself and of course, from outside the scientific circle but related fields such as science journalism.

This paper explores the factors from within the practice of research that could lead to wrongful conclusion about the research findings.

The author summarizes that:

“a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical models; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.”

Similar to the notion held by Simmons, et al (2011) that I have covered before, one major issue about research findings being false is due to the effect of false-positive findings and its persistence once it is being published.

Ioannaidis argued that “high rate of nonreplication of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming a conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for p-value less than .05".

Moreover, he also pointed out that biases also play a part in wrongfully claiming research finding to be true. Apart from biases that is involved in the construction of research design, analysis of collected data, and presentation of findings; a bigger issue also lies in the bias held by scientists that “resist accepting that whole field in which they have spent their careers is a ‘null field’”, or in short, a field that yield no scientific information (although the author puts a disclaimer of “based on our current understanding”).

Like Simmons et, al (2011), this paper also provides ways to improve the current situation in science research:

Focusing on finding better powered evidence;

Focusing on the totality of the evidence instead of looking only into isolated findings;

Moving away from chasing statistical significance but instead improving our understanding of the range of possible relationship of the variables of the research;

This paper explores and offers ways to tackle the issue of false-positive error in scientific research, especially in the fields of social sciences.

False-positive is an error of incorrectly rejecting a null hypothesis.

The authors of this paper argued that false-positive is actually a very costly error —both for the researchers as well as the field of science they are working on.

Their argument breaks down into several points:

False-positive results are usually very persistent once they got published. This is partly because null results have many possible causes, causing replications to be hard to do. Furthermore, the likelihood of null findings or exact replication studies to get published on prestigious journals also lessen the incentive of researchers to perform them.

It is a waste of resource and could mislead in the application of the findings.

A field known for publishing false-positives could lose their credibility over time.

They pointed out that the main cause for this error is “researcher degrees of freedom” in terms of making every single decision for their researches.

This would lead to them being relatively flexible in exploring analytic alternatives, that would cause them to (sometimes) only report things that “worked” —instead of presenting the research as it is (although it is also very hard to define the research finding as it really is, in my opinion. This is where objectivity really should play a big part).

Researchers exploration usually caused by two factors: ambiguity on how to best make decisions for the steps in their researches and their desire to find significant results (they referred this to as result of self-serving bias when facing ambiguous information and tendency to pick what best fit their desires).

They explored the impact of four common degree of freedom on producing false-positive results:

Choosing among dependent variables

Choosing sample size

Using covariate

Reporting subsets of experimental conditions.

They found that combination of the four could lead to a shocking 61% of false-positive results.

Hence, they proposed simple solutions for this flexibility-ambiguity problem —both for the researchers as well as for the reviewers.

source: the paper.

They claimed that these solutions are “simple, low-cost, and straightforwardly effective”.

However, this is of course not without limitations, which they also presented in the paper. The limitations went both ways as being “not to far enough”--which means that this solutions still have loopholes that can be manipulated; as well as being “too far” --which referred to the solutions might dampen researchers excitement in actually carrying out their studies..

Hence, they explored a few other solutions, which are less practical and effective (but should be worth to try, for the sake of better practices):

Correcting alpha levels (Main criticism for this point is adding another level of ambiguity).

Using Bayesian statistics (However, this could lead to higher degrees of freedom for researchers in trying out their data and taxing because they need to make additional judgement on a case-by-case basis)

Conceptual replications (But, they could be misleading)

Posting materials and data (well, ironically, it would be too hard too examine a paper when you have ALL the data; especially when you have to download, store, and load all the raw data provided into specific analytic tools, cross-check the results by running the statistical analysis independently, reading all additional materials, and so on —in short, too overwhelming. Also, there would still be an issue where raw data get manipulated).

At least, according to my ThreatSTOP.com email report, which flagged the SublimeText update IP (209.20.75.76) as 'Feed Level Danger 9'. A network monitoring tool that flags Sublime update traffic as a level 9 danger, is equivalent to anti-virus flagging a text editor as a malicious executable. I understand that false-positives are bound to happen, but this is just ham-handed. ThreatSTOP, I am disappoint.