[This guest post is written by Swordsmanus, printed here with the author’s permission. If you’re on Reddit, go give him an upvote or two.]

The conclusions of the study Risks and Benefit of a Gun in the House are held up as fact by anti-gun advocates. It is published by Harvard, after all. But there are several problems with how it goes about coming to those conclusions. I read the study and noted its citations wherever things did not add up. I then looked up the citations and cross referenced them with readily available information. My analysis is broken down by section within Dr. Hemenway’s study, so you can go back and forth between the two if you’d like.

Introduction

There’s some good background on gun ownership and demographics in the US. The problem is the blown-up quote featured on page one. An English, Communication, or Political Science major could have a lot of fun showing how the ostensibly even-handed quote is actually very biased, but I won’t get into it here.

Risks

Accidents

The data on accidental gun deaths presented in the study matches up completely with data from the Center for Disease Control and Prevention (CDC). However it omits the fact that accidental gun deaths per year have dropped steadily and significantly in the past 10 years, despite gun ownership going up and concealed and open carry laws becoming more lax across the nation.

In 2007 there were 613 accidental gun deaths, and again it’s part of a steady downward trend. For comparison, there were 29,846 accidental deaths by poisoning in 2007 according to the CDC [1].

So while the data presented was sound, it left out powerful yet easily-accessible context — the kind that any researcher on this subject should use as a starting point. But David Hemenway isn’t just any researcher. He’s aProfessor of Health Policy at the Harvard School of Public Health. So why leave it out?

Benefits

Deterrence

Dr. Hemenway points out that there’s no clear evidence that gun ownership deters crime.

It’s true, guns do not clearly deter crimes from being committed when viewed in the aggregate. The crime data from the FBI Bureau of Justice Statistics going from 1960-2009 supports that [2]. Restrictive gun legislation and lax gun laws have no clear effect, either [3].

There have been studies claiming otherwise both ways, but they have consistently only studied relatively short periods of time and relatively small sample sizes compared to a 50-year span for every state in the nation.

Thwarting Crimes

This is the real offender in the study. Up front, Dr. Hemenway acknowledges that aggregate data on gun use in self-defense is unreliable, while implying that some of those who are included in measurements of defensive gun use are actually criminals.

He then breaks down the sources of self-defense data: (a) police reports, (b) randomly sampled surveys that ask directly about self-defense gun use, and (c) surveys that ask about self-defense gun use only after respondents report that someone attempted to commit a crime against them (aka, NCVS Surveys).

(a) Police Reports: In this section, Dr. Hemenway implies that guns are almost never useful for self-defense from a home invasion. He cites only the Atlanta study taken from citation #12…A study done on one city, covering a 4-month period. There are tens of thousands of other cities to draw data from, across much greater spans of time than 4 months. There are decades of data to draw upon and view as a coherent whole!

That Dr. Hemenway chose just this single study, for a single city, going over just a 4-month period instead of the course of years, instead of aggregating available data across the nation, says loud and clear that citation #12 was cherry picked to suit his conclusions.

I really can’t overstate how much of a methodological and ethical issue this is, given that he uses this citation as though it accurately represents data on home invasions in the US, and uses it to support the study’s later conclusions against owning firearms.

(b) random sample surveys This section used citation #91 to support its claims, when that study merely gave inconclusive conclusions [4]. Dr. Hemenway also claims, “police report more total self-defense gun uses than all civilians combined“, yet citation #92 has the exact opposite data, and is from a study 5 years newer. The JustFacts link I cited earlier uses the same study, listing it as citation [17]. The relevant data is quoted there.

(c) NCVS Surveys This section opens with, “The National Crime Victimization Surveys (NCVS) obtain information about self-defense gun use only from those respondents who first report that a crime against them was threatened, attempted, or completed. This feature of the NCVS substantially reduces the problem of reporting incidents that were not true self-defense gun uses.”

There’s a critical flaw here. For the reporting period used in the study, the NCVS surveys are at least as likely to under-report defensive gun use as random sample surveys are to over-report them.

Why? The NCVS survey period covered in Dr. Hemenway’s study was from 1992 to 2001. Until 1996, in the majority of states, concealed carry was highly restricted or illegal rather than “shall issue” or unrestricted, and even then it wasn’t until 2002-2003 when 66% of states became “shall issue” states [5].

Thus, for the majority of the reporting period selected for this study, in most states it was illegal to have a handgun on your person outside the home. For the entire reporting period, a significant number of states had the same issue. And most of the NCVS crimes surveyed were for crimes outside the home:

“…Interviewers identify themselves to respondents as federal government employees, even displaying, in face-to-face contacts, an identification card with a badge. Respondents are told that the interviews are being conducted on behalf of the U.S. Department of Justice, the law enforcement branch of the federal government.”

“As a preliminary to asking questions about crime victimization experiences, interviewers establish the address, telephone number, and full names of all occupants, age twelve and over, in each household they contact.”

“…it is made very clear to respondents that they are, in effect, speaking to a law enforcement arm of the federal government, whose employees know exactly who the respondents and their family members are, where they live, and how they can be recontacted.”

“It is not hard for gun-using victims interviewed in the NCVS to withhold information about their use of a gun, especially since they are never directly asked whether they used a gun for self-protection…”

“…88% of the violent crimes which respondents reported to NCVS interviewers in 1992 were committed away from the victim’s home…in a location where it would ordinarily be a crime for the victim to even possess a gun, never mind use it defensively.”

“Because the question about location is asked before the self-protection questions, the typical violent crime victim respondent has already committed himself to having been victimized in a public place before being asked what he or she did for self-protection…respondents usually could not mention their defensive use of a gun without, in effect, confessing to a crime to a federal government employee.” [6]

There are known issues with the NCVS methodology for the period used, and yet its data is still used by Dr. Hemenway to strongly support his position. He wrote “Survey Research And Self-Defense Gun Use: An Explanation Of Extreme Overestimates” in 1997. He is no stranger to evaluating survey methods [7]. He should know well that survey inaccuracy can go both ways due to methodology, not just one way.

Overall the “Thwarting Crimes” section in the study has insufficient strength of citations used to support it vs. other available data/studies refuting it, including citations used within the study, like citations #12 and #92.

Shootings in the Home

Remember the part where Dr. Hemenway acknowledged that aggregate data on gun use in self-defense is unreliable?

In this part, he claims, “Home guns were 4 times more likely to be involved in an accident, 7 times more likely to be used in a criminal assault or homicide, and 11 times more likely to be used in an attempted or completed suicide than to be used to injure or kill in self-defense“.

Under “Thwarting Crime”, he says that self-defense data is unreliable. But elsewhere he turns around and implies it really is reliable, but only when the data sampled casts gun use in a bad light. An inconsistency like this is inexcusable, given the author’s credentials and experience.

Conclusion

Dr. Hemenway states, “…for those households where having a gun or not will matter this year, the evidence indicates that the costs will widely outweigh the benefits…”

I hope it’s clear by now that in this study, Dr. Hemenway started with the above conclusion and then cherry picked his data in order to support it.

What a researcher with integrity does is gather data and form conclusions based only on the most robust, reliable, and consistent data available, omitting or acknowledging the limitations of weaker data and leaving it out of one’s conclusions, especially when stronger data contradicts it. To do otherwise is to sully the reputation of all that you and your work are associated with.

RELATED ARTICLESMORE FROM AUTHOR

26 COMMENTS

  1. Sorry Nick, but you don’t sound too convincing. But, because I’m a fair guy, let’s say you’re right, and to compensate for all that sloppy surveying Hemenway did, we cut his concludisions in half and then halve them again.

    “In this part, he claims, “Home guns were 4 (1X or equal) times more likely to be involved in an accident, 7 (1.75) times more likely to be used in a criminal assault or homicide, and 11 (2.75) times more likely to be used in an attempted or completed suicide than to be used to injure or kill in self-defense“.”

    How’s that work for ya?

    • I like this section, “Home guns ……11 times more likely to be used in an attempted or completed suicide than to be used to injure or kill in self-defense“.
      I think homes are equally at fault here, we should legislate against the dangers of committing suicide in homes.

    • Bad information, lazy sampling procedures, and faulty analysis invalidate the entire study. There is no possible way to draw any conclusions whatsoever from the original study in a credible manner, no matter how many times you arbitrarily cut them in half.

      Show me a rigorous study and I’ll run with whatever conclusion it gives no matter if I agree or not, but crap like the original study makes me lose a little bit of faith in humanity.

    • You can’t really look at it that way mike. Using stats or regression analysis to forecast or trend anything, is impossible when you don’t have a good sample group and size; or good data. And even then, focused studies rarely give you any context from where we can draw a useful conclusion.

      Just halving what looks to be a poorly constructed study doesn’t give you margin of error to make a truly analytical hypothesis.

    • Plus it says “Kill or Injure” in self-defense. It doesn’t say “protect your life” in self-defense. Brandishing stats are even less reliable, but many a burglar has run away after seeing/hearing a homeowner with a gun. Or the homeowner could shoot and miss and scare him off without killing or injuring anyone. Also, “criminal assault” and “homicide” don’t belong in the same category (as Mr. Jackson would say, ain’t no ballpark neither. It ain’t the same league. It ain’t even the same sport)

      And the “well even if we divide it by four it’s still really bad” argument is weak. That has nothing to do with the validity of anything.

  3. My cousin told me the other day that he heard that 90% of guns were used against their owners. He’s scared to death of guns, but this is what he heard so it must be true.

  4. Working backwards from a conclusion is a bad career move for any researcher. Unless you do it with guns, pollution, or other stalwarts of Liberals, and then you can cherry pick data to your heart’s content and be applauded for it.

      • No it’s not Anthropogenic Global Warming is obvious to anyone. The ice caps have basically melted like they said they would and it’s too hot to survive anywhere. Wait, that’s hasn’t happened yet? Let me revise my estimate. It will happen in 10 years!

        It’s also clearly because we use oil and shoot guns. All that extra heat from firing off millions of rounds of ammo a year…

    • This is where I tend to part ways philsophically with some of my fellow 2A supporters.

      There are clearly times where studies are bunk. This one from Harvard seems to be one. Actually, any study that tries to identify or quantify a percieved human value, especially with such limited data sets…is starting off on the wrong foot. Particularly when it fails to account for rather abstract concepts. This study for example, fails to analyze impact of gun ownership and how it affects non-gun related legislation or threat of percieved tyranny.

      But there are plenty of other studies conducted that some may label as “liberal”, such as those concerning the environment, which are largely accepted by the scientific community. I don’t think we should ignore all scientific reports or conclusions, based on which political group has layed claim.

      A poorly conducted study is just that. A poor study. Doesn’t make its hypothesis less valid. Especially when there are plenty of other well developed studies with very sobering and largely accepted conclusions.

      • One should also keep in mind how the political climate of the times interacts with the science of the times. For example compare the forces at play in climate science with those in the gun debate. I have to think that perceptions of climate changes and its causes will provide far more power to those who manage to control the debate than to those that have influence in the firearms arena. In other words if you think the power play raging around guns is pitched, it’s nothing compared to climate change. It’s hard to think of anything that can’t be construed to have a “carbon footprint.” Regardless of the facts, climate change is the tyrant’s ever-loving dream.

  5. guns are almost never useful for self-defense from a home invasion.

    Keep and Bear Arms reports several examples of home invasions being repelled every goddam week, so Hemenway is obviously FOS.

    As PT notes: The Public Health School at Harvard is funded by the Joyce Foundation

    That’s what this “study” is really all about. It’s bought and paid for by the Joyce Foundation. Kick over any pile of wingnut bullsh!t and you will find the maggots from the Joyce Foundation. MAIG, the Brady Campaign Against America, mikey …. they’re all just mouthpieces for the Joyce Foundation.

    And POTUS? The Chosen One? A former director of the Joyce Foundation. Just in case you didn’t know.

  6. In 2009, 630 bicyclists were killed and 51,000 were injured. So, should we outlaw bicycles? They are obviously more dangerous than guns. We have to do something about this.

  7. I stopped an attempted home invasion in 1972 with a service revolver(I was an LEO)-no shots fired the would be perps fled in a car.
    Some years back my niece(an adult) used her dad’s 357 to confront a home invader who had already committed a number of sexual assaults in the area-he took off,but was caught the next day.Probably saved her life,because she wasn’t the type to just take it.
    Just two examples there.

  8. ATF Max 8 million CPL’s US, approximately 186 million age 21 or older or 4.3% of the people licensed for CPL.

    Possible deaths from CPL holders in 3 year time span from Violence Policy Center report last year, 137 or 45 per year equals .00000562 per concealed license holder. You can also review Florida’s data on CCW at http://licgweb.doacs.state.fl.us/stats/cw_monthly.html it says the same thing.

    JAMA http://jama.ama-assn.org/cgi/content/full/286/4/415 700,000 doctors in US kill 44,000 to 98,000 by medical malpractice every year or .14 per physician.

    Physician is .065 or .14 /.00000562 = 12,000 to 25,000 times more likely to harm you than a CPL holder.

    So where is the risk from concealed carry holders and why aren’t you antis crying to ban doctors?

  9. Just the other morning, September 5, 2011, at 0255hrs, I was awake working on spreadsheets and research. Suddenly five persons began banging on my door. They appeared to be Mexicans. I live in a predominantly white/black middle class neighborhood. Three cops live on my block. I limped my disabled veteran butt to the door to see what they wanted. Took me about a minute to get to the door from when they started banging on it. They stopped banging on it when I slammed my hand against the door. I yelled through the window next to the door, “What do you want?” Here I noticed they were wearing jackets, jackets in Arizona, in early September and its 90+ outside. They claimed to be looking for a person named Gilbert. I replied that there was no Gilbert here. Here they turned from the door and began arguing about going inside to see if he was there. I then limp-ran from the door, upstairs into my room and grabbed my Remington 870 express magnum. I grabbed six slugs. In the amount of time it took me to get back down stairs, I had only speed loaded 4 rounds. I shoved the other two in the sidesaddle and calmly walked to the door. I glanced out the window to see them fidgeting with something in their jackets. I then racked the slide hard to get a good, loud noise from it. I saw the five people visibly jump. I then proceeded to aim the weapon through the glass at the front man and pressed the safety off. Their leader then went and looked through the glass, all he saw was the end of my barrel and I was just waiting to put my finger on that trigger and squeeze. I knew that I was not afraid for my own safety but I could not guarantee the safety of the two sleeping persons upstairs should they breech the door. I knew without a doubt that should he even attempt to open the door in any way or should he attempt to draw a concealed firearm that I would squeeze and rack. Luckily, for me and mine, the sight of a shotgun barrel was enough to scare them off, this time. This incident marks the sixth time in the past 12 months that something like this has happened. Police have repeatedly instructed me “to not shoot them in the back”, and to “not wait for the police to arrive” in these situations.
    I have been formally told that this in the manner that people break into homes around here and shoot anyone sleeping in bed. They also tell me that people that have displayed a firearm have successfully turned away the criminals. It’s not often that The Police congratulate you for pulling a loaded weapon on a would be home invader.

  10. Dann,
    You heartless clod!! Don’t you realize there will now have to be 4 accidental shootings, 7 criminal assaults and 11 suicides to balance you one act of self defense!?! How can you possibly value your life so much as to inflict so much needless suffering on society? Obviously, if there were no self defense uses of firearms then the accidents, assaults, and suicides would simply evaporate.

  11. When I was in Vietnam in 1968-9,one of the guys in my unit extended his tour and got a 30 day leave home to Ohio.
    He came back and couldn’t conatin himself-he told us how he wound up killing a burglar in the middle of the bight with a deer rifle.He didn’t get any hassle from the cops.Seeing as how we were a maintenance unit,he got into more combat home on leave than in country.We only got shot at by people we couldn’t see with rockets and mortars.
    Nowadays he probably would’ve gone through hell over it.

  12. “What a researcher with integrity does …”

    There’s the problem – Harvard can’t seem to find any researchers with integrity.

Comments are closed.