Counting guns in early American

[Lynne Henderson]

For whoever's information:  Underlying databases are *not* usually requested
*or* examined by scholarly journals in any field.  It is assumed the
researcher is honest and honorable, and that s/he is not lying about her
data.  (What matters is the methodology, and of course supporting
appendices, etc. have to "work") As a reviewer for empirical studies, I have
never demanded data and I don't know anyone who has as a general matter.
Commentators, other researchers, etc. can write the author(s) and ask for
the data of course--and they do if the data set is not one that is generally
used.  If one wants to replicate findings, one may ask for data, and it is
considered bad form to refuse to share it *unless* as Bellesiles' earlier
post points out, one's data gets wiped out in a flood or fire.  And that
*hardly* deserves the kind of criticism I've seen posted on this list).
    As for statistics, they can show different things depending on what
tests you run, what you use for your regressions, what the size of your
sample is, etc. etc. etc.  Quantitative methods ar eimportant but not the
*ony* thing--you have to look at the purpose of the "tests" and what they
pick up and ignore in terms of factors.
    as for law review footnotes--I don't know of any l.revs that have asked
for original data sets much less examined them, though it is within the
realm of possibility, and I am sure someone out there has either done it or
had it done to them.  In fact, the accuracy/reliability of substantive
citechecking has gone downhill, I think, at least in terms of substance in
my 20 odd years in this business, though even back when i was a mere
citechecker, corners got cut (one famous empiricist made a claim the
articles he cited and his data *didn't* support, but when I pinted this out
to the president of the l.rev.he said it was "too late"'to fix it, and the
famous person was the famous person, etc.)
Happy new year to all
Lynne

hendersl at ix.netcom.com




on 1/2/01 7:07 AM, Leslie Goldstein at lesl at UDEL.EDU wrote:

> I have written a few books and dozens of articles and never once been
> asked for underlying databases (not that I generally use databases, but
> my most recent book was empirical and neither Cambridge, Routledge, nor
> Johns Hopkins press asked me for underlying factual evidence).  Indeed,
> my inclination was to insist on including all back-up evidence , at
> least in appendices, and there was some pressure (not too strong) on me
> to eliminate it.  Law reviews are notorious for checking footnotes but I
> know of no journals edited by professional scholars that actually look
> up all footnote references (people do not have time to do this).
> Leslei Goldstein
>
> James Maule wrote:
>>
>> First, there are at least two Jim's here, and though Ed quotes me (Maule) I
>> think his praise in "I think Jim has done..." is directed to Jim Lindgren.
>>
>> Second, Bellesiles wrote a book. Are editors no longer asking for underlying
>> data? I've been asked to provide copies of unpublished cited material, and to
>> provide or have available for provision the underlying databases (the latter
>> was also requested of me by the editors of a student law review). Given what
>> editors like to do to authors, I find it puzzling that there was no review...
>> even spot-checking, it seems to me from reading Jim Lindgren's article, would
>> have raised red flags.
>>
>> Third, I agree with Ed that the laying out of the research plan and the
>> methodologies is a necessary precursor to the laying out of interpretation
>> and conclusion. Back to taxes and accounting for a moment... leaving an audit
>> trail makes it more difficult for the numbers to be juggled.
>>
>> Fourth, has this sort of stark dispute over underlying numbers ever happened
>> with empirical studies on searches and seizures, stop and frisks, and other
>> events that are the subject of oft-debated Constitutional provisions?
>>
>> Jim Maule
>> Professor of Law, Villanova University School of Law
>> Villanova PA 19085
>> maule at law.villanova.edu
>> http://vls.law.vill.edu/prof/maule
>> President, TaxJEM Inc (computer assisted tax law instruction)
>> (www.taxjem.com)
>> Publisher, JEMBook Publishing Co. (www.jembook.com)
>> Maule Family Archivist & Genealogist (www.maulefamily.com)
>>
>>>>> RichardsE at UMKC.EDU 12/31/00 09:01PM >>>
>> Jim Maule wrote:
>>
>>> Finally, I confess my bias. I am an advocate of empirical
>>> analysis as a substantial component of legal research, I like
>>> numbers, I enjoy learning and teaching tax, and I read
>>> databases. So I am sure that learning more about statistics
>>> (which certainly is something I need to do) may be easier for
>>> me to propose for myself than it would be for others to pursue.
>>
>> (First, while I love empirical studies, I hate tax, so no one need think the
>> two
>> are related.)
>>
>> I think Jim has done an excellent piece of work.  It is not a statistical
>> tour
>> de force, in fact it represents something both simpler and harder - a good
>> analysis of the underlying research design.  While there are law professor
>> studies (and research studies in general) that founder on poorly done
>> statistical analysis, they are not as serious a problem as the more common
>> failure to do a proper research design.  With a proper research design,
>> others
>> can re-analyze the data and correct statistical problems.  Bad research
>> design,
>> however, means that the data themselves are unreliable.
>>
>> The worst studies, both from the sense of badly done and from being
>> dishonest,
>> start with a defective research design because mere statistical errors will
>> be
>> caught by all but the most ignorant reviewers.  Good research design does not
>> require much math, but it does require a lot of common sense.  Unfortunately,
>> it
>> is harder to teach common sense than to teach math.  It is also easier to
>> blame
>> bad math than bad sense when a study is found to be defective.
>> Unfortunately,
>> when a defective research design is discovered, it does not take a
>> statistician
>> to understand the problem.  This leads to the assumption that the work was
>> dishonest, rather than incompetent.  Sometimes that is the case, but I
>> suspect
>> that this example is more the second than the first.  I will disagree with
>> Greg
>> Sisk in his comment:
>>
>> "Lindgren's most devastating point is not that
>> Bellesiles's documentation was inadequate, unreported, or
>> unavailable, but that Bellesiles's reports of purportedly objective
>> figures on the frequency of guns being listed in probate records is
>> so wide of the mark, well beyond what would be expected by minor
>> errors in manipulating data."
>>
>> As with Alan Sokal's send up of Social Text (see
>> http://www.dartreview.com/issues/2.3.99/stflop.html) the real tragedy is that
>> material was and not caught by reviewers.  We know this is a problem with law
>> reviews because they generally have no adult supervision.  Unfortunately,
>> there
>> is little evidence that many social science and history journals are doing
>> much
>> better these days.  I will be the first to admit that scientific journals are
>> also having problems, but they generally are over MUCH more subtle issues, or
>> over political bias claims.  They usually do alright on this level of error,
>> except in politically  charged areas, where they often very cowardly. (That
>> said, I confess that this work would probably sneak past many prominent
>> medical
>> journals because of their bias against guns.)
>>
>> I suspect that any law professor, asked to analyze Bellesiles's study as if
>> he/she were going to attack it as opposing evidence in a trial, could have
>> figured out the problems. We are not encouraged to think about articles in
>> this
>> same way, especially if we agree with them.:-)  I think that this lack of
>> skepticism about one's allies data and articles is the most fundamental
>> difference I have seen in legal academics as compared to science.
>>
>> Ed
>>
>> Edward P. Richards
>> Executive Director - Center for Public Health Law
>> Professor of Law
>> University of Missouri Kansas City
>> (816)235-2370 Fax (816)235-5276
>> richardse at umkc.edu
>> http://plague.law.umkc.edu
>>
>> Ed