This paper describes a new approach for assessing reading comprehension in an accountability setting, called Cognitively Based Assessment of, for, and as Learning (CBAL). The CBAL approach uses evidence centered design to develop a competency model that drives the development of summative, formative, and professional support aspects of the assessment. The assessment is designed to measure the complete range of the reading construct including the pre-requisite reading skills that are necessary to decode and recognize words, to the higher level critical thinking skills needed to ensure success in the 21st century. This paper presents a prototype assessment that illustrates critical features of the CBAL approach. Psychometric and concurrent validity analyses suggest that the CBAL test development approach is a promising method for measuring reading competency.