The Limited Value of "What Works" Research

In the current edition of Education Week, Bo Yan and Mike Slagle write following:

"Ever since educational research became an academic discipline more than a century ago, researchers and educators have been vocal in their dissatisfaction over its impact on practice. For decades, education research has been criticized as confusing, irrelevant, and of little practical use, fueling a pessimistic view that research probably will not lead to better schools.

In response, the federal government and the research community have zeroed in on so-called “what works” research, and, in recent years, studies have mushroomed to answer a broad range of policy questions, such as: Do school vouchers work? Does technology improve student learning? Are private schools better than public schools? At the same time, existing studies on intervention strategies and programs are scrutinized to provide educators, policymakers, and the public with trusted and easy-to-understand scientific evidence of effective programming. The federal What Works Clearinghouse is a premier example of such endeavors.

This is all well and good, but we would argue that it is far from enough. We believe it is time for a research shift, and instead of making determinations about whether programs work or not, attention should turn to identifying the right students for whom a program is effective and the necessary contextual conditions that make a program work. What’s more, local schools should conduct rigorous studies to determine whether programs and initiatives will work for their students..."

Agreed. Rather than the singular focus on "does it work", we need the answers of "how it works". Articles next to never explain in a systematic way what is really unique about the practice, what are the essential elements, what are the critical implementation steps that are never discussed elsewhere, and what were the major mistakes you made on the way to perfecting the practice. That is practical information to improve student outcomes. Being a judge is much easier than being a good teacher.

Asst Secretary Ochoa Promotes Best Practices

Recently Assistant Secretary of Education Ochoa addressed the national leaders of federally-funded TRIO programs concerning priorities of the Secretary of Education's Office. Among topics in his speech were the need to identify, validate, and disseminate (IVD) best practices of TRIO programs. Following is part of a news report of his speech that concerns IVD.

“In TRIO programs, students are empowered to perform and succeed and there are many promising programs that demonstrate that fact,” he said, pointing to the Upward Bound summer program at the University of South Carolina as an example. Through the program, graduating high school seniors can take a college-credit research methodology course taught by one of the university’s professors.

“We know that TRIO programs work. But in this age of accountability, knowing that these programs do well is not enough,” Ochoa said. “We will need to develop a body of evidence that both quantifies the impact and cost effectiveness of these programs and that can demonstrate that effectiveness to legislators, policymakers and the public.” The Education Department plans to enhance existing data collection activities to measure outcomes and impact, help identify best practices and disseminate the results to all its grantees. (emphasis mine).

“To improve student outcomes, we need to spur the field to come up with innovative solutions to address the completion challenge and improve higher education productivity, build evidence of what works through rigorous evaluations and scale up and disseminate those strategies that prove successful,” he said.

Source: Jones, Joyce. (2011, March 8). Ochoa Hails Catalytic Impact of TRIO Programs on K-12 Level. Diverse Issues in Higher Education. Retrieved from http://diverseeducation.com/article/14862/

It is good to see public statements such as this by the Department of Education. I have been advocating for such an  approach for nearly twenty years. Before budget  cuts in the mid 1990s, the Department funded the National Diffusion Network (NDN). It was responsible for identifying promising and best practices, vigorously validating them regarding effectiveness, and disseminating those validated programs with the rest of the education community. I directed the national Supplemental Instruction program which was the only higher education program validated by the NDN. We already have a proven formula for IVD through a similar process used by the old NDN. It is time to get started again with IVD. I am working with a regional education association to conduct a pilot test of IVD with education programs located within a geographical region. Check back with this blog for updates on the progress.

Learning Assistance Often Ignores Impact of Culture on Learning of Students

Too often learning assistance and developmental education conferences and publications treat the issue of cultural and ethnic diversity as only an issue of demographics and not of pedagogy. Decades ago it was believed that sensitivity in this area was observing and honoring cultural events and including people of various cultures in class materials. This was a good start after that the previous focus only on dominant culture examples.The next step is required in learning assistance, teach multiculturally. WHile this has been widely adopted in education, the learning assistance community is far behind. Following is a good reader to illustrate practical ways to meaningfully engage students of different cultures in the classroom, honor their expertise, and make the classroom a richer and more productive environment for students of all cultures and backgrounds.

Higbee, J. L., Lundell, D. B., & Duranczyk, I. M. (Eds.) (2003). Multiculturalism in developmental education. Minneapolis, MN: Center for Research on Developmental Education, General College, University of Minnesota. Retrieved July 4, 2004, from: http://tinyurl.com/2e5wa23

The first three chapters of this monograph provide models for integrating multiculturalism in developmental education. The remaining chapters focus on conversations related to multiculturalism in developmental education, reported by our colleagues in the General College of the University of Minnesota. The work of these authors reflects the General College's efforts to implement its multicultural mission. The following chapters are included in this monograph: The Centrality of Multiculturalism in Developmental Education (Karen L. Miksch, Patrick L. Bruch, Jeanne L. Higbee, Rashné R. Jehangir, and Dana Britt Lundell); Walking the Talk: Using Learning-Centered Strategies to Close Performance Gaps (Donna McKusick and Irving Pressley McPhail); Creating Access Through Universal Instructional Design (Karen S. Kalivoda); Multicultural Legacies for the 21st Century: A Conversation with James A. Banks (Patrick L. Bruch, Jeanne L. Higbee, and Dana Britt Lundell); Is there a Role for Academic Achievement Tests in Multicultural Developmental Education? (Thomas Brothen and Cathrine Wambach); The Triumphs and Tribulations of a Multicultural Concerns Committee (David L. Ghere); MultiCultural Development Center (MCDC): Sharing Diversity (Ghafar A. Lakanwal and Holly Choon Hyang Pettman); Summary Report on the Third National Meeting on Future Directions in Developmental Education: Grants, Research, Diversity, and Multiculturalism (Dana Britt Lundell); Report of the Future Directions Meeting Multicultural Themes Track (Jeanne L. Higbee and Holly Choon Hyang Pettman); and appendices.

Socially contructed learning spaces rather than instructional technology

I have been thinking about the terms "instructional technology" and "learning technology." They are often used interchangeably by many, includinig myself.  Doing so blurs their distinctions. I have implemented a number of Web 2.0 learning tools within my class: wiki web pages, podcasts, self-create music vidoes on a history topic, etc. Yesterday Brian Fredrickson and I facilitated a conference session on "Social media and learning spaces in schools, work sites, and communities." It was at MinneBar with over 1,000 in attendance. We had a great discussion and many within the audience shared how they use Web 2.0 for learning purposes.

Over the past couple of years, I now understand that my role is creating and facilitating "learning spaces" within the classroom so that students are active participants and co-creators of the class experience and learning outcomes. It is really not about which Web 2.0 technology tool or services that is used, it is the engagement and co-creation by students that makes the difference. It reminded me about the classic Barr and Tagg article from the mid 1990s that identified the shift from a teacher-dirven to a student collaborator learning environment within the classroom.

Barr, R. B., & Tagg, J. (1995). From teaching to learning: A new paradigm for undergraduate education. Change Magazine, 27(6), 13-25. Retrieved July 4, 2004, from: http://critical.tamucc.edu/~blalock/readings/tch2learn.htm

This is one of the most often cited articles on this topic and is credited by some as helping to influence higher education significantly since it was published in a journal that is frequently read by college presidents and chief academic and student affair officers. According to the authors, a paradigm shift is occurring in American higher education. Under the traditional, dominant "Instruction Paradigm," colleges are institutions that exist to provide instruction. Subtly but profoundly, however, a "Learning Paradigm" is taking hold, whereby colleges are institutions that exist to produce learning. This shift is both needed and wanted, and it changes everything. The writers provided a detailed matrix to compare the old instruction paradigm with the new learning paradigm in the following dimensions: mission and purposes; criteria for success; teaching/learning structures; learning theory; productivity/funding; and nature of roles.

Mainstreaming best practices of learning assistance and developmental education within first-year courses

Isolated and prerequisite remedial and developmental level courses are on the chopping block. FOr good or bad, the national dialogue argues for their relegation to community colleges and prohibition at four-year institutions. How are needs met for students who still some of the outcomes from such courses? Embedding the best elements into rigorous, first-year courses is a solution. Doing so benefits all students within the courses since all will experience turbulence within the curriculum. Following is an article that describes changes forecast a decade ago.

Damashek, R. (1999). Reflections on the future of developmental education, Part II. Journal of Developmental Education, 23(2), 18-20, 22. Retrieved July 4, 2004 from:  http://www.ced.appstate.edu/centers/ncde/reserve%20reading/V23-2damashek%20 reflections.htm

Interviews were conducted with a number of leaders within developmental education: David Arendale, Hunter Boylan, Kaylene Gebert, Martha Maxwell, Santiago Silva, and Diana Vukovich. The dialogue points to several emerging trends: (a) mainstreaming, (b) removal of developmental education from 4-year institutions, and c) increased professionalism of developmental educators. Mainstreaming developmental education courses into college-level, graduation-credit programs of study fits into the paradigm of learning assistance and enrichment for all students. The participants in the discussion were unanimous in proposing a comprehensive academic support program that would include elements such as a learning center, adjunct or paired courses, Supplemental Instruction, tutoring, student assessment, and program evaluation. Boylan advocates funds for professional development and Gebert proposes faculty, student, and staff recognition whereas Silva includes academic advising, counseling, career services, mentoring, and especially faculty training in his list of important program components. Arendale and Vukovich propose a complete paradigm shift away from the medical model to learning support for all students. By deferring to Maxwell’s (1997) latest book Improving Student Learning, Vukovich gives Maxwell credit for providing insight into best practices based on years of experience and the best research resulting in the recommendation of a comprehensive learning assistance model. the value of such a model is that it is more easily integrated into the academic process because it is understood as service for all students. This model is not burdened by the stigma of serving only the least able students, who, for many academic, administrative, and political leaders, are seen as a drain on the institution’s academic standards.

The Community College and Remedial/Developmental Education

Recently the Gates Foundation announced a gift of over $100 million to support community colleges in identifying best practices to improve remedial and developmental-level courses and other services to support higher student achievement and graduation rates. The following interview forecast many of these recommendations by Robert McCabe a decade ago. Revisiting McCabe and reading his book provides best practices and case studies of success with increasing student success.

Callan, P. M. (2000, Fall). An interview: Robert McCabe. National Crosstalk, Retrieved July 4, 2004, from: http://www.highereducation.org/crosstalk/ct1000/interview1000.shtml

Robert McCabe, senior fellow with the League for Innovation in the Community College and former president of Miami-Dade Community College is the focus of this interview. Much of the interview revolves around McCabe's newest book, No One to Waste, a national study of community college remedial programs. McCabe employs a variety of arguments for the support and expansion of remedial education.

ERIC Database, ED448813, Title: No One To Waste: A Report to Public Decision-Makers and Community College Leaders. Authors: McCabe, Robert H. Abstract: Twenty-five community colleges participated in a study that tracked 71 percent of 592 students who successfully enrolled in a remedial program in 1990. Follow-up interviews of program completers gathered information about further education, employment, family, and facts about post-remedial life. A criminal justice search was also conducted on the entire study cohort. These data were the basis for this first comprehensive national study on community college remedial education students. The study found that most successfully remediated students perform well in standard college work, gravitate to occupational programs or direct employment, and become productively employed. While a majority of the remedial students were white non-Hispanic, ethnic minorities were overrepresented in the cohort and even more so in a seriously deficient student sub-cohort, confirming that remedial education is a significant issue for ethnic minorities. While community college remedial programs are cost effective, most colleges fail to use the substantial research concerning successful remedial education, and do not fund programs at a level necessary for successful results. Recommendations include: (1) giving remedial education higher priority and greater institutional and legislative support; (2) requiring assessment and placement of all entering students; and (3) developing a national guide to assist colleges in developing effective remedial education programs. (Contains 15 figures and tables, 45 references and 63 pages.)/p>

Reinventing learning assistance and developmental education

The recent announcement of $110 million by the Gates Foundation to reinvent remedial and developmental education at community colleges calls for dramatic change. Several years ago a group of past national presidents of two of the largest organizations in that field created a blueprint for not revising, but for reinventing the field. Following is an executive summary to the complete report which is available online [click here for report]. The major findings from this strategic review include the following:

1. Significant and systemic change is required. The external threats to the existing organizations for not engaging in dramatic change and renewal far outweigh the advantages of no change. The organizations have worked incredibly hard with a strong volunteer leadership structure to implement their strategic plans. Progress has been positive but slow and incremental due to limited resources and the volunteer leadership base available through the organizations. The organizations have been working at maximum capacity for a long time. The unmet needs of their association members require a dramatically different structure. An analogy illustrates the current state of the professional associations. A well-known parable cautions against placing “new wine in old wineskins.” When it was common practice to place wine in a leather pouch that was stitched together, fresh leather was always used since it would stretch when the new wine expanded during its fermentation process. Use of a previous old wineskin container that had already been stretched out would rupture if new wine was placed within it. This analogy describes why a new association “container” is necessary to achieve an expanded set of services for members and increased influence within both higher education and the wider society.

2. A strategic review of creating a new professional association identifies a long list of potential strengths for such a new structure. This review also identifies potential external threats and weaknesses. The Working Group identified potential solutions for dealing with each. One of the biggest challenges is the process of creating the new association. A basic law of physics is “where there is movement, there is always friction.” The final section in this report identifies a suggested calendar of events to engage more people in the conversation to encourage buy-in and support for change.

3. The Working Group identifies that in addition to the current profile of association members, there are other groups within postsecondary education that could find affiliation with a new organization desirable. A new association with an expanded mission, vision, and more inclusive language could encourage active membership by these groups. More than 60,000 professionals work in the field served by CRLA and NADE. However, less than10% are members of these associations. A new association would be more attractive to these nonmembers if they could access more services through venues in addition to conferences.

4. The reason for change is not rearranging organizational patterns. It is about increasing the quality and quantity of professional development venues for individual members. CRLA and NADE host excellent chapter and national conferences. However, a careful review of other postsecondary professional associations finds that they offer many more services for their members through rich web sites, multiple publications, on-line conferences, webinars, and research that identifies best practices for their members.

5. A new professional association serves as a catalyst for new language, expanded mission, and reinvented vision for serving a wider community. While the reason for change is about professional development and service to its members, the organizational language and structure have an impact upon the association’s capacity to serve its members. Some examples of these potential changes for conversation among a larger group inside of CRLA and NADE include: More inclusive association purpose statement such as: “The purpose of this organization is to advance the scholarship and practice of professionals dedicated to postsecondary student success through academic support and instruction”.

6. A wider conversation needs to occur concerning the ideas and possibilities presented in this report. The final section of this report identifies a possible scenario for this wider discussion. The inclusion of more voices and ideas will generate an even better vision for a new future.

The pace and scope of change needs to dramatically accelerate. Rather than incremental change, dramatic reinvention and transformation is needed. A new term used to describe this change is “leapfrogging” (Harbison & Pekar, 1998; Nonaka & Nichiguchi, 2001). Instead of “fixing” an old process, it is “leapfrogged” and a new process is introduced. The often-cited classic example is how emerging countries are designating cell phones as the primary communication channel and skipping the wiring of the countryside with traditional telephone wires. The cost of the new technology and ease of implementation is much less than the older technology. With this report, the Working Group strongly supports the creation of a new professional association rather than attempting to retool the older ones. The synergy of a larger organization with more assets and a fresh start presents many opportunities.

This Working Group completed the first draft of this report during its meeting in Austin, TX in June 2007. The conclusion was that a new professional association would be beneficial, but the change process would be difficult and significant. Our group recognizes that the easiest course is to continue to make incremental changes within the current structures. Perhaps the umbrella group for the existing associations, American Council for Developmental Education Associations (ACDEA), could provide more coordination and collaboration for the field to better meet the needs of the members. Those are two possible paths to the future. However, this Working Group recommends a third path: creating a new professional association with a more inclusive language, mission, and vision for the future for all the reasons stated in this report.

A reoccurring question examined by the Working Group has been “why change?” This long report has carefully explored the pros and cons of creating a new association and provided numerous suggestions for its services and organization structures. A better question than “why change?” is “what kind of change is demanded by postsecondary education and needed by these professionals?” Perhaps others can provide a paradigm for us when considering the future. When writing a dialogue among several of his characters in a play discussing the future, Shakespeare penned the expression “The Undiscovered Country” to describe this place. While everyone will visit the future, no one can come back and tell others exactly what it will be like. However, all of us will walk into The Undiscovered Country.

We have the choice regarding how we walk into the future. Change can be proactively managed or simply reacted to. The best of past traditions can be brought into the future and merged with new structures and traditions or all can be left to chance. The most important element that moves forward into the new association are members of the current organizations. They form the core of the new future and bring forward the history and traditions of the previous organizations. Let us encourage new members to join these veterans as we walk together into The Undiscovered Country as colleagues and friends. Let’s build a new future together.

August 2007. CRLA/NADE Working Group: David Arendale, Hilda Barrow, Kathy Carpenter, Russ Hodges, Jane McGrath, Pat Newell, and Jan Norton

References

• Harbison, J. R., & Pekar, P. (1998). Smart alliances: A practical guide to repeatable success. San Francisco: Jossey-Bass.
• Nonaka, I., & Nishiguchi, T. (Eds.). (2001). Knowledge emergence: Social, technical, and evolutionary dimensions of knowledge creation. New York: Oxford University Press.