If digital libraries are to live up to their promise as vehicles to achieve widespread educational improvement, the resources in those digital libraries must be of high quality and high relevance to teachers and learners. But in a library which has no acquisition librarians and no pre-defined patron group, how shall we identify those educational resources which are of high quality and utility?

We report here on an effort by the Digital Library for Earth System Education (DLESE) to harness the community-building and information-gathering capabilities of the World Wide Web to find the "best" educational resources, according to seven criteria: scientific accuracy, pedagogical effectiveness, ease of use for teacher and learner, quality of documentation, importance or significance of content, ability to motivate learners, and robustness as a digital resource. The DLESE Community Review System begins by gathering feedback from educators who have used the resource in their classroom, via a web-mediated recommendation engine (viewable at http://dlese.ldeo.columbia.edu). Educators can complete a set of scoring rubrics pertaining to the selection criteria, contribute publicly-viewable teaching tips, and assess the effectiveness of the resource in challenging teaching and learning situations. Results from the web-based reviews are automatically aggregated into reports for the resource creator, the Editorial Review Board, and the public. When a threshold number of educators have tested and reviewed the resource, and the scores on the quantitative rubrics have averaged above a threshold value, the resource moves on to a specialist review step. Experts recruited by the Editorial Review Board review the resource for scientific accuracy and educational quality. Resources which do well on both review steps move from the DLESE Broad Collection to the DLESE Reviewed Collection.

In our first tests of the Community Review System, we have learned that educators are willing to contribute reviews, that resource creators find this form of feedback from classroom educators to be valuable, that many educators take the time to articulate detailed suggestions and comments in addition to simply clicking the rubric buttons, and that the specialist reviews reveal nuances about the resource overlooked by the community reviewers.