How useful are readability formulas?
Not very. I read about some interesting research last spring and meant to write more about it then. Here’s the bottom line. The researchers evaluated 9 of the most commonly used formulas. Here’s how Willingham summarizes their findings:
All of the readability formulas were more accurate for higher ability than lower ability students. But only one—the Dale-Chall—was consistently above chance.
Only the Dale-Chall predicted reading ability better than flipping a coin. Ouch!
In general, writing experts dismiss the utility of readability formulas as a means of improving the quality of writing produced in the workplace. I mentioned the range of methods for evaluating quality before. The ACM Journal of Computer Documentation dedicated an entire issue to the topic of readability to usher in the 21st millennium century (Volume 24 Issue 3, Aug. 2000). I’ve copied their table of contents with live links and available abstracts for those of you with an interest.
Table of Contents
| Introduction to this classic reprint and commentaries | |
| Bob Waite | |
| Pages: 105-106 | |
| The measurement of readability: useful information for communicators | |
| George R. Klare | |
| Pages: 107-121 | |
| Readability and computer documentation | |
| Gretchen Hargis | |
| Pages: 122-131 | |
| Traditional readability concerns are alive and well, but subsumed within several more recent documentation quality efforts. For example, concerns with interestingness and translatability for global markets, with audience analysis and task sufficiency, and with reader appropriateness of technical text all involve readability, but often in ways not easily measured by any formula. | |
| Readability formulas have even more limitations than Klare discusses | |
| Janice Redish | |
| Pages: 132-137 | |
| A literature review reveals many technical weaknesses of readability formulas (when compared to direct usability testing with typical readers): they were developed for children s school books, not adult technical documentation;they ignore between-reader differences and the effects of content, layout, and retrieval aids on text usefulness; they emphasize countable features at the expense of more subtle contributors to text comprehension. | |
| Readability formulas in the new millennium: what’s the use? | |
| Karen A. Schriver | |
| Pages: 138-140 | |
| While readability formulas were intended as a quick benchmark for indexing readabilty, they are inherently unreliable: they depend on criterion (calibration) passages too short to reflect cohesiveness, too varied to support between-formula comparisons, and too text-oriented to account for the effects of lists, enumerated sequences,and tables on text comprehension. But readability formulas did spark decades of research on what comprehension really involoves. | |
| Klare’s “useful information” is useful for Web designers | |
| Kristin Zibell | |
| Pages: 141-147 | |
| In many ways the writing principles that Klare recommended 37 years ago to promote high readability scores still apply to web-site design. Behind the pursuit of readability lies audience analysis, a concern with the intellectual level, previous experience, motivation, and reading goals of ones intended audience. Suitably adjusted to take account of online interactivity, those same concerns should guide design work on web structure and interfaces today. | |
| Readable computer documentation | |
| George R. Klare | |
| Pages: 148-168 | |
| Arguing that current approaches to understanding and constructing computer documentation are based on the flawed assumption that documentation works as a closed system, the authors present an alternative way of thinking about the texts that make computer technologies usable for people. Using two historical case studies, the authors describe how a genre ecologies framework provides new insights into the complex ways that people use texts to make sense of computer technologies. The framework is designed to help researchers and documentors account for contingency, decentralization, and stability in the multiple texts the people use while working with computers. The authors conclude by proposing three heuristic tools to support the work of technical communicators engaged in developing documentation today: exploratory questions, genre ecology diagrams, and organic engineering. |
Thanks for this post, Kim.
Just to let you know that the ACM links are going via the library authentication system, so they won’t work outside the Uni of Alabama.
Thanks, Iza. I thought I was linking to the ACM site, but I’ll fix it tomorrow.
The links now take you directly to the ACM site. Note that you do have to be a subscriber to access the articles. Of course, you can also pay per article.
The reason existing readability formulas are so poor is twofold.
First, they only measure sentence length and word length.
Second, they use a crude character count to judge word length.
To improve on the existing formulas for the latest StyleWriter, we graded all words in a 200,000-word dictionary by difficulty and frequency. The result was a more accurate formula in StyleWriter than the existing readability formulas.
StyleWriter was 92% accurate within 2 grades and 100% accurate within 3 grades when tested on human-assessed grade level texts. StyleWriter was a maximum of three grades out from the human assessors. All other formulas were had a maximum error of six to ten grades.
You can read about the results at:
http://www.squidoo.com/readability-can-we-design-a-better-readability-formula
We also designed a much more comprehensive measure of writing style for the latest StyleWriter – the Bog Index.
You can read about it at:
http://www.editorsoftware.com/images/StyleWriter/Towards%20a%20better%20readability%20measure.pdf
StyleWriter’s Bog Index knows the words in each sentence and distinguishes between words and abbreviations. It allows that clear, specific and descriptive words, proper nouns and Pep (interest words) words in writing add specific content, which make the document easier to read.The Bog Index also marks the management-speak down for using abbreviations, passive verbs, jargon and other style faults.
Nick Wright
Director
Editor Software
Email: info@editorsoftware.com
Website: http://www.editorsoftware.com