I was reading an interesting (if slightly, sometimes incredibly, technical) site dealing with the methodologies used by environmentalists to argue for climate change. It is delightfully skeptical, while providing very technical analysis. However, there was an interesting article on a presentation about the possibility of failure inherent in peer review and the likely impact on public policy:

Empirical research in academic journals is often cited as the basis for public policy decisions, in part because people think that the journals have checked the accuracy of the research. Yet such work is rarely subjected to independent checks for accuracy during the peer review process, and the data and computational methods are so seldom disclosed that post-publication verification is equally rare. This study argues that researchers and journals have allowed habits of secrecy to persist that severely inhibit independent replication. Non-disclosure of essential research materials may have deleterious scientific consequences, but our concern herein is something different: the possible negative effects on public policy formation. When a piece of academic research takes on a public role, such as becoming the basis for public policy decisions, practices that obstruct independent replication, such as refusal to disclose data, or the concealment of details about computational methods, prevent the proper functioning of the scientific process and can lead to poor public decision making. This study shows that such practices are surprisingly common, and that researchers, users of research, and the public need to consider ways to address the situation. We offer suggestions that journals, funding agencies, and policy makers can implement to improve the transparency of the publication process and enhance the replicability of the research that is published.

This is especially interesting to me, as it is an argument I have made before, though without that specific argument.

My past arguments, starting with my analysis of journalism, have argued that certain professions may either autoselect or gradually develop an orthodoxy, which causes an inherent bias in what they produce*. We can see this clearly among journalists, many of whom enter the profession to "save the world", and thus tend to have a more activist, left-leaning political bias than the general public. Similarly, either because of the need to seek approval to obtain tenure, to pass peer review for one's articles, or just to enjoy the approval of peers, many academic disciplines have a departmental orthodoxy. This orthodoxy may vary from university to university, and clearly varies between disciplines, but I think it would not be unfair to say, especially in the humanities, and ertainly since the era of draft exemptions in the 1960's, the university orthodoxy, by and large, has been shades of left-wing ideology.

And there is one other influence, more common to the physical sciences. Though scientists will deny it, funding can influence the orthodoxy of an agency. Even if every scientists is scrupulously honest, the funding agencies are not. So a scientist who produces consistent reports disproving global warming is unlikely to find such funds, while one who does the opposite will garner additional grants. As a result the second will get better offers, will be desired at more prestigious universities, and in general will have more influence on orthodoxy and the training of future scientists. In addition, as those studies will be promoted, they will color the public impression of the discipline, causing the self-selection I mentioned earlier for journalists. As climate science seems, in the public eyes, to be all about the dangers of global warming, those who deny global warming are less likely to become climate scientists. And so the public impression becomes the reality over time.

However, none of this relates to the article above, which tends to focus on the rather lax standards of peer review when it comes to methods and data used. And that too is a problem, as the essay describes. Though, when combined with the possibility of peer review being undermined by accepted orthodoxies within disciplines, it does serve to highlight just how little credence we should give to peer review, or to peer review alone.

All of which brings me back to a point I made before, the need to personally examine claims, or, if you feel unqualified, to at least seek a review from more than one source, and one outside the circle of "insiders". I know that is the theoretical purpose of peer review, and it works as such in some disciplines, but in too many cases, peer review has developed some weaknesses, meaning that either personal review, or review by additional experts, is needed.

----------------------------------------------------------------------------

* Actually, this theory came from personal experience.  Back when I was younger, I applied for a retail job and was given a test to see whether I would steal or not. Apparently I failed. However, when I thought about it, I realized the reason. My father was a career police officer, and throughout my life, on every trip near Baltimore, I would hear "I remember that guy who staggered out of there holding his intestines", or "that's where X got caught trying to carry out six TVs". Every family trip became a police blotter. As a result, I came to think of everyone outside of my immediate family as potential criminals. And so, on my test, I answered all the "would you steal X" questions "no", but all the questions phrased as "given the opportunity, do you think people would steal X" as "yes". It was the logical outcome of my youth, but to the simple-minded psychologists who developed the test, it was proof that I was "projecting" my desire to steal onto others. So it is no surprise that I give little credence to most psychological theories (I do like Fromm sometimes, when he stays out of politics, and Branden, as he was excommunicated by the Objectivists, some of Maslow, some of Laingg, obviously Szasz, but not most others). What is odd is that I would become the proponent of assuming other people are basically competent and more like me than not. But I guess we all can change our beliefs given time.

------------------------------------------------------------

POSTSCRIPT

Before someone tells me that I am not competent to review everything and should defer to committees of experts, they need to recall how many errors have been endorsed, even championed quite aggressively by established authorities. And I don't just mean in the distant past. In the last century, the best medical authorities championed bleeding and mercury purgatives, and then "medical nihilism", going so far as enacting laws to shut out competing theories. Or in this century we saw physicists convinced of the existence of aether, psychiatrists and doctors convinced of the benefits of lobotomies and leukotomies, doctors performing routine hysterectomies for any "female problem" once one passed child bearing years, using thalidomide on fertile women, and so on. Just because an idea has been endorsed by recognized authorities is no promise of truth. Yes, experts may get the answer right, in fact they probably do it with great frequency, but they can also make mistakes. And once we introduce some sort of orthodoxy, the frequency of mistakes rises. And real scientists should welcome the scrutiny. A true scientists relies on challenging assertions to establish the truth. It is only demagogues, those afraid of inquiry, who feel the need to declare debate "closed" and enforce their opinions through appeals to authority.

Note: I am aware those who promote very dubious medical treatments often use the same arguments, but that does not render them invalid. Anyone who shuts out all alternatives and cuts off questioning, even for the best motives (such as doctors afraid of quackery) have also introduced the possibility of making errors permanent. Humans are liable to error, and the only cure is the constant scrutiny of others. Yes, that scrutiny also involves the possibility of people making errors (choosing quacks rather than doctors, believing Keynesians rather than subjective value theorists, voting in Obama), but the alternative is to give preferential treatment to our present errors, insulating them from wide scale examination. In other words, for fear of new errors we decide to make the old ones permanent.  That is not science, it is stagnation.

NOTE 2: In response to the first note, I am sure doctors and others will argue they will correct their own errors through free inquiry. However, the problem is, by closing the profession and narrowing the range of allowable approaches, they have also created a set of professionals with a uniform set of beliefs, at least to a degree. They may be able to correct some errors, in fact I am sure they will. But there are also likely some errors that the established views will make more difficult to discern, perhaps fundamental errors in their assumptions, which will be much harder, if not impossible, to even discover, much less correct. And those who do challenge those assumptions may be accused of having deviated from the acceptable beliefs, and the rules excluding inquiry may be used to stop them as well. In fact, the existence of such rules may prevent them from even asking those questions. And thus, the self-policing may not be nearly enough to correct the errors that presently exist.

NOTE 3: The odd thing is scientists recognize how religious belief, and dogma concerning what was or was not an acceptable belief closed many avenues of research in the middle ages. Yet they themselves, at least in some fields, now provide the same restrictions, just using a different set of guidelines. Nor is this limited to the hard sciences, social science and humanities are even more bound by institutional dogmas.

POSTSCRIPT II

I know many of my readers do not accept my belief in a totally free market in the practice of medicine, and I accept that. However, you do have to see how limiting what is and is not an acceptable approach could make certain discoveries, or the elimination of certain mistakes, less likely. Whether or not you think that is offset by the benefit of keeping people from frequenting quacks is up to you, but we should at least be honest about the costs of licensing (which also includes the higher costs inherent in any barrier to entry), rather than pretending it is entirely beneficial with no costs.