30% of IPCC Report References Not Peer-Reviewed

The chair of the IPCC has made the claim that the IPCC reports are based on peer-reviewed scientific papers. An audit found that 30% of the references cited in the 2007 report were not, in fact, peer-reviewed at all. Half the chapters in the report were so thin on actual references that the audit gave them an "F" - less than 60% of the references in that chapter were actual peer-reviewed science.

Yes, the person who ran the audit is promoting her forthcoming book on the site. Fortunately the audit results and methodology are available. The criteria for peer-reviewed status were very simple - the article simply had to be from a journal, with no judgements made on the quality of that journal.

And so *which* scientific conclusions do they invalidate?

Just a note - I disagree with their assertion that a PhD thesis is not peer-reviewed. When you write your thesis (or at least, this is how it works At the University of Ottawa), you submit it to 3 reviewers. 2 of them are from inside the university, and the third is an external examiner from outside the university with whom the author (or their supervisor) has no ties (no co-authored papers, etc.)

No that is fairly standard. I know at my school it was a panel of 5.

I tried to pair down in there, but could not find references to the articles in question, just numbers. Is that in there, actual links or references to articles I could at least nab on Lexis?

Robear wrote:

And so *which* scientific conclusions do they invalidate?

Even if the answer is none, that doesn't mean this is OK. Though I'll also agree that the woman's methodology is somewhat suspect.

So Lobster, if an article is a *summary* of peer-reviewed work - that's not peer-reviewed. If it is something that does not *require* peer review (say, an assay of bird species in an area), that counts as a negative. And so forth. Also, this does not show that any of the conclusions *depend entirely* on non-peer-reviewed work. Nor does the scale they use to create the "grades" make any kind of sense, it's completely arbitrary because, well, how many peer-reviewed articles does it take to validate a conclusion? And what are the qualifications of the reviewers? Have they done this for, say, Watt's Up With That or some other sources, for comparison? Is this a standard method used to judge scientific accuracy?

What method of analysis was used to prevent bias?

If the science is not brought into question, what's the use? Besides FUD and propaganda, of course. The methodology is more than "somewhat suspect", it's entirely inappropriate for the supposed conclusion.

Which is why I was hoping for some mentions of the references in question. Those UN compendiums on any subject are huge documents. I am most familiar with those dealing with crime-mostly rape and the dusparity in justice around the world for victims. And they will include mentions to newspaper articles, BBC broadcasts, university polls. Such documents are not feasible to be peer reviewed, but must be sourced if mentioned. Now if there are data sets, conclusory studies without being vetted we have an entirely different story.

But on the face, I see little more than the typical anomoly hunting.

Robear wrote:

So Lobster, if an article is a *summary* of peer-reviewed work - that's not peer-reviewed. If it is something that does not *require* peer review (say, an assay of bird species in an area), that counts as a negative. And so forth. Also, this does not show that any of the conclusions *depend entirely* on non-peer-reviewed work. Nor does the scale they use to create the "grades" make any kind of sense, it's completely arbitrary because, well, how many peer-reviewed articles does it take to validate a conclusion? And what are the qualifications of the reviewers? Have they done this for, say, Watt's Up With That or some other sources, for comparison? Is this a standard method used to judge scientific accuracy?

What method of analysis was used to prevent bias?

If the science is not brought into question, what's the use? Besides FUD and propaganda, of course. The methodology is more than "somewhat suspect", it's entirely inappropriate for the supposed conclusion.

If an article is a summary of peer-reviewed work it is not peer-reviewed. Usually it doesn't need to be. On the other hand, if the summary gets anything important wrong then the criticism is valid. It might not negate the scientific assertion but it can weaken it, and that's significant.

I'm not saying that this invalidates the IPCC report. I'm not saying that this is anything but FUD. What I am saying is that we can't let frustration or emotion keep us from examining and re-examining our beliefs. That isn't how science works. The appropriate response to this kind of criticism is to examine the sources marked "non-reviewed" and determine their validity. It very well may be that they're all valid. Again, that's how science works: you have an idea, you challenge it, and if your idea turned out to be right then you don't really learn anything new. You just validate your conjecture. The "eureka" is when you come up with the theory. Not when you test it.

I'm sick all to hell of obstructionist "skeptic" propaganda too. In fact, I am very well confident that these objections are total bullsh*t expressly for the purpose of FUD, as you noted.

When you're trying to stand on the side of science, you don't get the luxury of saying, "my theory has held up to all your previous objections, therefore I don't need to listen to your objections anymore." One of those objections might have some merit, and if you ignored them just because you were sick of being contradicted, you aren't doing science. That's not to say we need to start over from scratch every time someone has a question. Simply looking at the objection and saying, "you're using an incorrect definition of 'peer-review' and placing undue importance upon it" might be enough. As is holding up any previous retort to any previous phrasing of the same question.

It's frustrating and it's stupid but science is not about PR. Any scientist worth their salt needs to be able to say that the body of evidence presently available says that climate change is a real thing, but that it is possible that it's not. You might love an idea, you might see nothing but support for it, you might be willing to fight for it, but as soon as you marry it and refuse to acknowledge the possibility that it could be wrong, that's not science anymore. It's faith.

...And I really hate saying that because seriously. Sick to death of arguing about this instead of trying to fix it.

LobsterMobster wrote:

If an article is a summary of peer-reviewed work it is not peer-reviewed. Usually it doesn't need to be. On the other hand, if the summary gets anything important wrong then the criticism is valid. It might not negate the scientific assertion but it can weaken it, and that's significant.

I'm not saying that this invalidates the IPCC report. I'm not saying that this is anything but FUD. What I am saying is that we can't let frustration or emotion keep us from examining and re-examining our beliefs. That isn't how science works. The appropriate response to this kind of criticism is to examine the sources marked "non-reviewed" and determine their validity. It very well may be that they're all valid. Again, that's how science works: you have an idea, you challenge it, and if your idea turned out to be right then you don't really learn anything new. You just validate your conjecture. The "eureka" is when you come up with the theory. Not when you test it.

I'm sick all to hell of obstructionist "skeptic" propaganda too. In fact, I am very well confident that these objections are total bullsh*t expressly for the purpose of FUD, as you noted.

When you're trying to stand on the side of science, you don't get the luxury of saying, "my theory has held up to all your previous objections, therefore I don't need to listen to your objections anymore." One of those objections might have some merit, and if you ignored them just because you were sick of being contradicted, you aren't doing science. That's not to say we need to start over from scratch every time someone has a question. Simply looking at the objection and saying, "you're using an incorrect definition of 'peer-review' and placing undue importance upon it" might be enough. As is holding up any previous retort to any previous phrasing of the same question.

It's frustrating and it's stupid but science is not about PR. Any scientist worth their salt needs to be able to say that the body of evidence presently available says that climate change is a real thing, but that it is possible that it's not. You might love an idea, you might see nothing but support for it, you might be willing to fight for it, but as soon as you marry it and refuse to acknowledge the possibility that it could be wrong, that's not science anymore. It's faith.

...And I really hate saying that because seriously. Sick to death of arguing about this instead of trying to fix it.

And having someone say an article wasn't peer-reviewed isn't science either. Science would be "hey, I did an experiment and the data I got from it disproved or will likely change your theory". As usual, the anti-AGW crowd isn't doing any science, they are simply attacking the data, how that data was collected, and other procedural items. I'm not saying we shouldn't pay attention to those things and make sure everything is as above board as possible, but at a certain point you *do* get to ignore the folks who aren't bringing their own peer-reviewed science to support their claims.

It's just as easy to cast doubt on the so-called Citizen's Audit of the IPCC report. Who audited just one of the sections, say the first chapter? A guy named Remond Weissenberger of Toronto, Ontario. What are his scientific credentials? Well, in his own words he's "been following the climate debate for the past five years." That's it. Well, he must have some deep scientific training that would allow him to sort through all the references, understand the all the various papers, and know which refer to actual scientific studies and which just reference a previous body of well understood research, right? Nope. He's an industrial designer.

So who is auditing the auditors to make sure they're doing an honest job and following rigorous standards and procedures?

Take the first non-peer reviewed reference Weissneberger quotes: the paper "Avoiding Dangerous Climate Change by Inducing Technological Progress".

This paper addresses the question of the costs of stabilising atmospheric concentrations of carbon dioxide at three levels 550ppm, 500ppm and 450ppm, with the emissions projected to 2100. Two policy instruments are used to achieve these targets: emission trading permits for the energy industries and carbon taxes for the rest of the economy, with the revenues recycled to prevent extra inflation.

It's not even a scientific paper, it's an economic policy paper. Of course it's not going to be peer-reviewed.

According to the Citizen's Audit crowd, this is supposed to show how the IPCC isn't to be believed, but they also don't quite understand what the IPCC report actually is. It's not a scientific article, it's a public policy support document written for one audience: policy makers and government officials to inform them about the science behind AGW and the possible social, economic, and political consequences of doing nothing to mitigate it.

The problem I see with your position, Lobster, is that it's vulnerable to denial of service attacks through repeated forcing of reviews of the same data. The IPCC process *did* this kind of review, both scientific and political, by people who were legitimately experts in their fields, and on the political side highly skeptical to boot. This group is simply saying "you need to go through several years and several hundred thousand man-hours of review again, only this time with people who think you are liars dogging you every step of the way, simply because we think you should." At some point, we have to acknowledge that there *is* no way to make these people happy, and drop it. They already lie about the data; why would they not take this opportunity to simply create more fud?

They are not questioning the results of any of the peer-reviewed work, or the conclusions of the IPCC, that I can see. Maybe I'm wrong, but this seems to be just a way to get scientists not to do work, instead forcing them to be second-guessed by people who don't mind lying to make their points. It's not useful to think of this as part of the scientific process, because it's not (as was pointed out by OG.)

This is not a sincere attempt to improve the science. It has one goal, to discredit the field of climate science. That's it. They can do what they like with the data - and they are - but we should not give them the respect that legitimate inquiries would receive.

Slinger, Robear, I see where you're both coming from. I agree with you both. As I said...

It's frustrating and it's stupid but science is not about PR.

The truth isn't on a deadline. Science doesn't care if it's constantly questioned with the same questions.

You can't force science to serve politics, even if it's in response to someone trying to do the same.

I see what you're saying, Lobster. It's just that this "study" is not science, and we should not give it that much respect.

Robear wrote:

This is not a sincere attempt to improve the science. It has one goal, to discredit the field of climate science. That's it. They can do what they like with the data - and they are - but we should not give them the respect that legitimate inquiries would receive.

It appears to me that their goal is to discredit the IPCC's claims about their analysis being entirely supported by peer-reviewed research and some of the IPCC report conclusions, not to discredit the field of climate research - unless you're reading a different page than I am. That would most definitely fall under science, since the IPCC report is supposed to be a scientific report. Was it science when the IPCC was making the claim about the Himalayan glaciers that was recently shown to be based on nothing more than a magazine article? The only sad thing about this is that the press and the IPCC obviously didn't do their homework.

There are more examples of what happens when you remove the non-peer-reviewed references. Whole sections are unsupported or weakly supported, and some are based solely on press releases. You don't think this undermines the quality of the analysis?

And, as far as the credentials of the auditors goes, how hard is it to determine if something was published in a journal or not? We're not talking a review of the sources, just a determination of whether or not they were actual, published, peer-reviewed research, as was claimed by the IPCC. I think most people could handle that.

You really think the IPCC report conclusions are invalidated because someone found a reference that described possible effects on an industry, from that industry, that they believe would happen anyway? Really?

That's some powerful invalidation there.

It appears to me that their goal is to discredit the IPCC's claims about their analysis being entirely supported by peer-reviewed research and some of the IPCC report conclusions, not to discredit the field of climate research - unless you're reading a different page than I am.

The IPCC report is a summary of the field. It's not a scientific paper itself. Here's the telling sentence, which was emphasized in the original:
"If the IPCC has misled the public about a matter this straightforward surely its ability to accurately describe more complex topics is called into question." If the IPCC report is wrong and misleading, then the science behind it is not convincing, and the whole field of climate science is discredited.

But hey, let's take your position. The review is science. Please explain why it was not peer-reviewed, and how it's used standard statistical methodology to determine the relationships between the papers and the results. I'm eager to see the prior use of numerical counting of references and the assignment of American school grading systems in peer-reviewed analyses of scientific papers.

Aetius wrote:
Robear wrote:

This is not a sincere attempt to improve the science. It has one goal, to discredit the field of climate science. That's it. They can do what they like with the data - and they are - but we should not give them the respect that legitimate inquiries would receive.

It appears to me that their goal is to discredit the IPCC's claims about their analysis being entirely supported by peer-reviewed research and some of the IPCC report conclusions, not to discredit the field of climate research - unless you're reading a different page than I am. That would most definitely fall under science, since the IPCC report is supposed to be a scientific report. Was it science when the IPCC was making the claim about the Himalayan glaciers that was recently shown to be based on nothing more than a magazine article? The only sad thing about this is that the press and the IPCC obviously didn't do their homework.

There are more examples of what happens when you remove the non-peer-reviewed references. Whole sections are unsupported or weakly supported, and some are based solely on press releases. You don't think this undermines the quality of the analysis?

No, the group is trying to discredit climate research by attacking a policy document. They can't fight the science so they're trying to discredit the policy report and thereby discredit the climate research behind it.

At the heart of the matter, the group failed. All the assertions in Chapter 3 that the earth is getting warmer and we humans are responsible for it were found by all three reviewers to be peer-reviewed science.

So if the science is right, then it just becomes an argument over what to do about it. That's why there are all the economic impact assessments and projections of how much it will cost to fix things. The IPCC report is just like every other governmental policy document in that manner: it's trying to educate policy makers and let them know what the possible costs and consequences are for various responses.

If you want to argue over abatement costs or the ins and outs of emissions trading, fine. Let's do it. But don't try to claim that because a report by the UK Treasury isn't peer-reviewed that it magically invalidates decades of climate research. It just means that we're down to arguing over the best way to solve the problem we've collectively gotten ourselves into.

OG_Slinger wrote:

If you want to argue over abatement costs or the ins and outs of emissions trading, fine. Let's do it. But don't try to claim that because a report by the UK Treasury isn't peer-reviewed that it magically invalidates decades of climate research. It just means that we're down to arguing over the best way to solve the problem we've collectively gotten ourselves into.

I agree. Just because something is not peer-reviewed does not mean it couldn't pass peer review. Now it doesn't necessarily mean it could either, but we can't automatically treat "not peer-reviewed" as "definitely untrue." It's just not nearly as powerful as peer-reviewed material, assuming it's appropriate to review in such a way.

Robear wrote:

The IPCC report is a summary of the field. It's not a scientific paper itself. Here's the telling sentence, which was emphasized in the original:
"If the IPCC has misled the public about a matter this straightforward surely its ability to accurately describe more complex topics is called into question." If the IPCC report is wrong and misleading, then the science behind it is not convincing, and the whole field of climate science is discredited.

You're misreading that. They are talking about the reliability and trustworthiness of the IPCC report. If the IPCC's process permits basic mistakes and weak references, then the document as a whole is not what it is purported to be - a reliable analysis based on peer-reviewed research.

LobsterMobster wrote:
OG_Slinger wrote:

If you want to argue over abatement costs or the ins and outs of emissions trading, fine. Let's do it. But don't try to claim that because a report by the UK Treasury isn't peer-reviewed that it magically invalidates decades of climate research. It just means that we're down to arguing over the best way to solve the problem we've collectively gotten ourselves into.

I agree. Just because something is not peer-reviewed does not mean it couldn't pass peer review. Now it doesn't necessarily mean it could either, but we can't automatically treat "not peer-reviewed" as "definitely untrue." It's just not nearly as powerful as peer-reviewed material, assuming it's appropriate to review in such a way.

However, if something is claimed to be based on peer-reviewed science, and substantial parts of it are not, then it is right to question the claims that are being made.

If the IPCC's process permits basic mistakes and weak references, then the document as a whole is not what it is purported to be - a reliable analysis based on peer-reviewed research.

"Basic mistakes"? That's not what they were looking for. No one's found any in the summary conclusions, and only a few that did not affect the conclusions in the supporting materials. Further, if you read the report intros and prefaces, you won't find the claim to be based entirely on peer-reviewed research. That appears to come from the same sources the critical report disparages - press releases, interviews and statements, mostly by one person (who in my mind is pretty well discredited.)

Again, what matters is the results. The IPCC report has been subjected to intense scrutiny before and after it's release. It's 3 years old now. And your position is that if we just keep digging, we'll find some flaws no one else had seen, simply because someone tells you that if a certain percentage of sources are not peer-reviewed, the report is seriously flawed? I mean, consider the logic here.

Where is the connection between the amount of peer-reviewed references and the ultimate conclusions of the report? Please, demonstrate to me that this approach has any validity at all. Where else is it used in science? Can you think of other fields of hard science where this kind of review is used? Why is an A-F scale useful, and what is the basis for the scale assignments?

Please, help me understand why this is useful at all.

However, if something is claimed to be based on peer-reviewed science, and substantial parts of it are not, then it is right to question the claims that are being made.

I can't find that claim in the intros to the AR4 documents, although I'm still looking. Given that there were hundreds of reviewers before publication, and it's been out for three years now under intense scrutiny, I'll ask you - why were the flaws that this will supposedly make apparent not found before? Did all the skeptics just take it's accuracy for granted?

Now I'm pretty well pissed off at these guys. Take a look at the flow chart here, in the IPCC AR4 documentation about how the report was assembled. Note the text in the blue box denoting materials used in the IPCC Reports:

Peer reviewed and internationally available scientific technical and socio-economic literature, manuscripts made available for IPCC review and selected non-peer reviewed literature produced by other relevant institutions including industry.

So, basically, these guys are lying to us, claiming something that is directly contradicted by the methodology given in the report itself. Now do you believe this is a sincere effort? You really think these guys maybe forgot to read the rules for sources used by the very report whose sources they are contesting?

If so, that's pretty egregiously wrong.

Hold on a second. Segments in question, are properly cited bibliographically, and contextually?
So we are now denying global warming for proper scholarly bibliography and citation? I have posed this question before. But at what point do you unhitch from that wagon as it crashes into the Flat Earth Society, The Discovery Institute, and Christian Science Center?

Annex A at the end of this document details the conditions for use of non-peer-reviewed materials. It notes that the references to them will be integrated with the scientific and technical documents.

Nothing is perfect. There are undoubtedly some references that are less useful than others. But the idea that we can't trust the IPCC report because of some made up criteria, offered by a group that's *concealing* the nature of information provided in the report, that's just wrong. We're being lied to, yeah. But not, in this area, by the IPCC.

Aetius wrote:
LobsterMobster wrote:
OG_Slinger wrote:

If you want to argue over abatement costs or the ins and outs of emissions trading, fine. Let's do it. But don't try to claim that because a report by the UK Treasury isn't peer-reviewed that it magically invalidates decades of climate research. It just means that we're down to arguing over the best way to solve the problem we've collectively gotten ourselves into.

I agree. Just because something is not peer-reviewed does not mean it couldn't pass peer review. Now it doesn't necessarily mean it could either, but we can't automatically treat "not peer-reviewed" as "definitely untrue." It's just not nearly as powerful as peer-reviewed material, assuming it's appropriate to review in such a way.

However, if something is claimed to be based on peer-reviewed science, and substantial parts of it are not, then it is right to question the claims that are being made.

Aetius, please point out where the Citizen's Audit folks have invalidated the science behind climate change. All three of their so-called auditors found that the part of Chapter 3, which discussed the science of climate changed, all referenced peer-reviewed articles.

Notice how they aren't even touching the output of Working Group 1, which is all the science behind climate change. No, they're focusing on the output of Working Group 3, who focuses on the ways we can mitigate climate change. Why don't they audit this IPCC report? They don't because they'd look like the ignorant asses they are.

No, instead they picked on the policy document, which of course will contain non-peer-reviewed content because it deals with things like economic predictions and estimates, and hoped people wouldn't check to closely, ignore that the IPCC has different working groups focusing on different areas, and overlook the main report that is merely summarized by Working Group 3 as the basis for talking about ways to fix things.

Aetius, the frustration is not directed at you. One of the issues I have with this stuff is that it's designed to appeal to well-meaning people who in fact may not have made up their minds, as well as being reinforcement for those who have. It shares the same overall problem that the overwhelming majority of denialist literature has; while masquerading as "skepticism", it's in fact deliberately deceptive. The frustration comes from the fact that this stuff pops up like mushrooms, over and over again, each time with some new angle, some plausible argument, but full of flaws, mistakes and deliberate omissions. Yes, there is a campaign to obfuscate the science, and this is a good example of it.

To Lobster's point, the issue with treating *all* challenges as science is precisely this. These assaults are a massive time sink for researchers who could be doing actual science, but instead are pilloried in the press, insulted and denigrated by members of Congress, and forced to respond over and over again, in detail, to repeated "requests for information" designed simply to harass and obstruct. At some point, we just have to dismiss the obvious frauds and move on. The appearance of unfairness is nothing compared to the slanders already suffered by the researchers, anyway.

No good comes of this kind of "investigation".

KingGorilla wrote:

Hold on a second. Segments in question, are properly cited bibliographically, and contextually?
So we are now denying global warming for proper scholarly bibliography and citation? I have posed this question before. But at what point do you unhitch from that wagon as it crashes into the Flat Earth Society, The Discovery Institute, and Christian Science Center?

It sounds like nitpicking but there's a reason for scholarly bibliography and citation. In fact, it is precisely to keep people from making stuff up. Ridiculing motives doesn't change that.

Robear wrote:

To Lobster's point, the issue with treating *all* challenges as science is precisely this. These assaults are a massive time sink for researchers who could be doing actual science, but instead are pilloried in the press, insulted and denigrated by members of Congress, and forced to respond over and over again, in detail, to repeated "requests for information" designed simply to harass and obstruct. At some point, we just have to dismiss the obvious frauds and move on. The appearance of unfairness is nothing compared to the slanders already suffered by the researchers, anyway.

No good comes of this kind of "investigation".

I agree. I still maintain that this is a disconnect between science and emotion rather than the failure of science to be more useful.

And I maintain that it's sheer deception. Which should automatically negate any consideration of the study's claims.

Robear wrote:

And I maintain that it's sheer deception. Which should automatically negate any consideration of the study's claims.

I say it shouldn't matter why someone's made a claim. Motivation doesn't change the facts.

So, what proportion of their time do you consider it reasonable for people to spend to cover their asses against the folks who say "nuh-uh!"?

From what people have said here, the IPCC report referred to *had* proper citations, and a proper description of their methodology, the differences between criteria for including different sorts of sources, etc.

That's a pretty reasonable amount of documentation.

What should they have done on top of that?

If the next time a report like this is made, it has this additional documentation, how certain is it that someone won't ignore the methodology (including both the things they've ignored this time and whatever additional new work has been done) and nitpick it apart again?

At what point is it okay for the scientific community to say "Right, I've got some actual work to do" and walk away?

Hypatian wrote:

At what point is it okay for the scientific community to say "Right, I've got some actual work to do" and walk away?

I think science in general needs to hire a PR agency. Instead of waiting for Neil deGrasse Tyson, Bill Nye, or Carl Sagan to pop up every generation, there needs to be a team of folks who's sole job is to make science more understandable to the masses and who can use the same tactics of the deniers against them: attacking the credibility of individual deniers, looking into their funding and backing, and basically making them look like the morons they are.