PHASE 3: GETTING THE CRITICAL SCRUTINY

Christopher Schmitt
U.S. News and World Report

Our impulse, and common practice in newsrooms, is to closely hold the information we gather and develop until it's ready for publication. But in CAR work, that can be just the wrong way to go.

That's because:


1) Our data analysis often puts us in unexplored territory, OR
2) Although we're clever, we're often not as expert as, well, the experts, OR
3) Both

Often, we're essentially doing social science-style research, akin to studies or dissertations.

So, in keeping with that theme, consider using a variant of the "peer review process"; using experts, expose your entire process from idea, to data source, to analysis and results - to critical scrutiny. By focusing as much on process as on results, this is a large step beyond the customary "get comment on what we found."

There are always experts to be found; some are too busy or attitude-challenged ("Come back and see me when you have a PhD...") but many others quite willing. (Believe it or not, some are even flattered to be asked.)

Many, if approached sincerely and responsibly, will be warm to the task, especially if they're leaders in their field.

Hint: If you talk their lingo, or are working in an area of particular interest, you'll do better. Short of that, do your homework and be as prepared as you can.

1. Look at professional journals. Beyond the standard methods, professional journals are a great way to find the leading lights - especially in technical areas.

2. Do it one-on-one or in a group, but the more minds brought to bear on your results, the better.

3. Convene a panel. Invite experts from all sides of an issue to meet. Make a formal presentation of your project. Ask for discussion, comments, feedback, etc.

4. Get reaction on the record. You need quotes on both sides of what you found. Don't be afraid to report the negative. Everyone else has to be criticized.

What you get:

1) Bullet-proofing and embarrassment-sparing ("That's bogus because..."), OR
2) Suggestions to make it better ("Did you think of doing it this way...?"), OR
3) Comments and quotes usable in the story, OR
4) Ideally, all of the above.

Laying bare your soul like this can be uncomfortable; you've got to be prepared to take some heat; but there's little doubt you'll get a better story.

A caveat: The experts, on whatever side, can have their own agendas, too, and that can influence the feedback they give you. Still, if nothing else, it's expert reaction to which you'll be most vulnerable upon publication...forewarned is forearmed.

Examples:

U.S. News & World Report recently detailed how the nation's nursing homes - despite cries of poverty to Congress as operators pled for more federal money - were actually much more profitable than acknowledged; and that even as patient care remains deeply troubled, operators are frequently engaging in lucrative self-dealing. The findings were based on an analysis of hundreds of thousands of pages of financial reports. Before publication, U .S. News prepared a detailed written report of its findings and methodology, and circulated it to numerous people - policy and health care experts, patient advocates and the industry itself- for comment and review.

The San Jose Mercury News, in a study of the common practice of plea bargaining in criminal court cases, documented a strong racial bias. Well before publication, the Mercury News convened a panel of judges, prosecutors and public defenders to hear a presentation on the findings and to discuss and critique the results. Comments and suggestions emerging from the panel led the newspaper to pursue several new areas of inquiry.

ONCE THE DATA IS SOLID, LOOK TO LEVERAGE IT

5. While analyzing a database can yield good results, the real power - and the particular capability of the computer - is in linking different sets of information, to produce even better, but less-than obvious results. So be alert for opportunities...any "field" in your data is potentially away to link up one file with another. Think of names, unique identifiers, geographic codes...

Example:

When the Mercury News examined the California State Lottery, a significant - but unproven criticism was that the lottery exploits the poor, because they play more often and can least afford to lose. The newspaper obtained lottery sales data broken out by ZIP code, then matched that data against ZIP code-compiled demographic data, to clearly show that lottery sales were far higher in poorer areas than in affluent places.


Back to Journalism Links page