Data Retreats: A Kick-off to the Continuous Improvement Process

		At this point you have completed the first module and have learned about the importance of using data, specifics about standardized assessment data, and steps for engaging in the process of data analysis. Rather than being an abstract exercise, data analysis should be a purposeful process that contributes to school improvement. For that reason, each of these lessons was couched within the context of the continuous improvement process.

		By now, you may have embraced the concept of continuous improvement, but like many educators, you may wonder how to begin. This is where the process called a "data retreat" comes in. It is a collaborative process that allows educators to step back and use their data in order to move their schools forward.

		Data retreats address common barriers to change in schools by helping educators better understand their own data as well as learn a process for using it. A data retreat provides the time, place, and guidance for educators to come together for a few days, analyze various types of data, prioritize their needs, and develop their own data-based goals, action plans, and self-evaluation requirements. Although schools and agencies have distinct characteristics based on their histories, their cultures, and their communities, the process of collecting, analyzing, reflecting upon, and using data for continuous improvement entails common practices. Data retreats introduce and reinforce these practices so that they can be applied to various school systems and data sources.

		Data retreats are as much about people and the work they can do together as they are about data. They depend on the process of Collaboration, or working together in a group effort towards a common goal. Collaboration is a necessary component for profound levels of change but is not always maximized within schools. For example, some schools make headway in their use of data through isolated events. A mathematics committee may map their curriculum and assess its alignment to the state standards. Language Arts teachers may meet to analyze student writing samples and assign scores according to a writing rubric or a principal may look at trends in student achievement on the school's standardized assessment.

		These are commendable examples of how data can be used within schools but their effects on the overall school improvement process are limited because the events occur in isolation, are not conceptualized within the needs of the entire system, do not involve all staff in a similar process, and do not build knowledge as a system. They are not collaborative. Under these conditions, data-driven decision-making can promote change, but the impact will be more fragmented and limited in the school system as a whole. This can be characterized as "first-order" change (Argyris & Schon, 1978; Cuban, 1988).

		Data retreats aim for "second-order" change. This profound and lasting type of change requires schools to make a cultural shift by going deeper into the how their systems are structured and the ways in which people work together (Cuban, 1988). It touches everyone who participates in the organization by modifying the patterns in which people think, perceive, behave, relate to each other, and collaborate (Stiegelbauer, 1994). In data retreats, the process of second-order change is "jump started" by combining analytical work with teamwork, group reflection, dialogue, problem solving, and goal setting.

		Variations of Data Retreats Although the data retreat process is relatively consistent, the form a data retreat takes can vary according to a variety of factors: Focus. With the multitude of data available in schools, it is important to identify the focus of the data retreat in advance. Most schools focus their first data retreat on academic and attendance data as they relate to the OIEP and local school goals. Once these major trends have been identified, possibilities for follow-up retreats are:

		School & Community Data Retreat. School staff members work side-by-side with parents, community members, and students in an effort to collaboratively enhance school improvement. During this retreat, stakeholders generate and analyze data from a survey related to issues such as School Leadership, Strategic Planning, and School Climate. The process culminates with a School and Community Data Night, during which each stakeholder group presents their data, their observations and their recommendations for improvement.

		Financial Data Retreat. School financial data are analyzed to assess alignment to school reform goals, sources of funding, and budgetary spending. Budget teams identify financial priorities, reallocate resources accordingly, and communicate findings to school stakeholders.

		Special Education Data Retreat. Information specific to special education is analyzed to determine characteristics of this population within the school, service delivery, progress on Individual Education Plans (IEP), and results of school assessments. In addition, special educators identify areas in which professional development is required to help them meet the special needs of these students. Goals for special education are set and are communicated to school stakeholders.

		Each of these data retreats contributes to systemic change by using a variety of data sources that relate to school functioning, involving multiple stakeholders, and creating the opportunity for ample, informed, goal-oriented communication. During this lesson, we will emphasize the data retreat process in general, using examples from the academic data retreat to illustrate the starting point for most schools.

		Composition. Data retreats can differ in terms of the participants involved. To effect second-order change, it is ideal to include as many stakeholders as possible, keeping in mind issues related to confidentiality. For academic data retreats, schools may involve all teaching staff, including instructional aides. This involves all of the key decision-makers in the analysis and goal-setting process. To protect confidentiality, other staff members and parents would not participate in the data retreat when individual student results are analyzed and shared.

		Logistical and/or financial reasons sometimes prohibit the involvement of all stakeholders in a data retreat. When this occurs, schools assemble school improvement teams with representative membership. For example, a six-person team for an academic data retreat at a kindergarten through eighth grade school might include the following members:

		Principal Academic Head Counselor Teacher: lower elementary Teacher: middle elementary Teacher: upper elementary

		School improvement teams should always include an administrator and should have balanced representation of additional staff members. Their challenge after a data retreat is to carry their findings back to the remainder of the staff, communicate the trends, and involve the remainder of the staff in the finalization and implementation of the improvement goals.

		Location. Data retreats can occur onsite, with just one participating school, or in a regional location, with a group of several school improvement teams. When an entire staff is involved in a data retreat, the event is usually hosted locally. School libraries, cafeterias, all-purpose rooms, and even agency training centers serve as good locations for congregating staff for this purpose. There are advantages to having a data retreat onsite. Data that are in paper form do not need to be packed and shipped to another location, staff members are comfortable in their own familiar setting, and any missing data can be quickly located and included in the analyses. Importantly, the school is able to involve most if not all of the staff, encouraging collaboration during the decision-making process and consensus as goals and implementation plans are set.

		When groups come together for a data retreat, they are usually held in a major city in a regional location, often within a hotel conference room. Although school improvement teams have to pack up their data and travel to these events, they benefit from the participation of other schools. In this setting, they can understand how the needs of their own school compare to those of other schools, they can share their experiences, and can gain ideas from the work of their peers. This offers the rare opportunity for interschool collaboration.

		Length. The duration of data retreats varies depending on the quantity of analyses that needs to be conducted, the experience-level of the participants, and the degree that technology is used. The more time that is allotted, the more analyses that can be done. The more experienced the participants are in the area of using their data, the more efficient the data retreat will be. Finally, the degree that participants have access to electronic databases or decision support systems also has an impact on the pace of data analysis. Combining these factors, data retreats tend to range from three to five days. A three-day retreat might involve experienced participants who have access to electronic databases and are focused on the number of analyses they want to accomplish. A five-day retreat might involve inexperienced participants whose data are in paper form and need to accomplish a variety of different analyses.

		The Data Retreat Process All data retreats, regardless of their focus, proceed through a similar process. Handout 1 represents this process by showing three major stages of the data retreat, the corresponding steps, and examples of what the steps would involve during an academic data retreat. Experience has shown that moving through the same stages over time helps participants retain the methodology and become more efficient in its deployment. The three stages include:

		Context Setting. Before launching into data analysis, it is important to establish a rationale for the work. Participants may wonder why they are there and may not have a good grasp of the data they are about to analyze. Developing a common base of knowledge for participants helps the remaining steps of the data retreat make sense and unfold smoothly.

		The first two lessons of Module One exemplify some of the content covered during the context-setting stage of an academic data retreat. That is, participants are introduced to the importance of data-driven decision-making for school improvement, the concept of continuous improvement, and characteristics of standardized assessments. This background information is necessary to ensure that participants understand the meaning of the event and how it will help them grow as professionals. It also attempts to "level the playing field" in terms of knowledge, ensuring everyone present begins the retreat understanding the type of data they will be analyzing. When participants grasp the background information, the rationale for the data retreat becomes clear and they better understand results of their data analyses.

		As the context is being set, system goals must also be discussed. Data retreats provide an opportunity to evaluate existing goals in addition to setting new ones. Reviewing current goals foreshadows the questions that will be asked, the analyses that will take place, and the decisions that will be made throughout the course of the retreat. For example, before an academic data retreat, the five BIA/OIEP goals are reviewed, and goals one (70% of students will be proficient/advanced in Reading/Math) and three (Individual student attendance will be 90% or better) are highlighted in particular because they guide the several of the analyses that occur.

		Data Analysis. The process of data analysis was discussed in detail in Module One, lesson three. In this lesson, the steps will be briefly reviewed to help you understand how they fit into the overall data retreat process. If additional details regarding the data analysis process are desired, please return to lesson three for a thorough explanation. Data analysis always begins by creating guiding questions. These questions reflect what educators are trying to learn through the data analysis process and should align to system and/or local goals. Guiding questions also help educators target the type of data they need to collect and the analyses they should complete. Academic data retreats incorporate three general guiding questions, then reword those questions to align to specific goals. In relation to BIA/OIEP system goals, the following guiding questions are used: General question Reworded and aligned to goals Where are we now? What percent of our students are proficient or advanced in Reading?

		Has our performance changed over time? Are we getting more students into proficient / advanced range this year compared to years past?

		Are we meeting the needs of all students? Are students with disabilities achieving at the same rate as those without disabilities?

		Next, data must be selected to complete the analyses. To do so, educators must consider how the outcomes in question are best measured, the demographic variables that are important to use, and the data necessary to reflect the instructional processes at their school. Educators should aim to use outcome data of the highest quality possible.

		Data are then organized by placing them into a chart. The chart will begin to suggest numeric trends and provide a means for disaggregating data into various categories or and/or demographic variables. After placing data into a chart, the next step is to graph the data. Graphs are powerful because they tend to capture information in an interesting fashion, are easily interpreted, and show trends in the data more readily than charts. Graphs are visual, and many people grasp information better through pictures than from rows of numbers. Bar graphs, line graphs, and scatterplots are each examples of graphing techniques used during a data retreat.

		At various points during the process of data analysis, reflection must take place. Reflection involves stepping back, taking a good look at the results of data analyses, and discussing patterns and trends. To ensure that the process of reflection is constructive and not destructive, participants must resist the "blame game" and avoid pointing fingers at others for results.

		Results from the past must be used to improve the future, not to alienate certain stakeholders or individuals within the school. In the spirit of collaboration, the opinion of all participants should be sought out and respected. Often, one person sees something in the data that is not observed by the others. Finally, participants should pay attention to details in the data, such as which grade levels have the most students proficient in reading, but also the "big picture," such as the general trend for reading growth in all grades over the past three years.

		Planning. Data analysis is not the end point or overall purpose of a data retreat. A data retreat aims to use the information gleaned from data analyses to refine and focus the school improvement process. Typically, participants have identified a number of issues and areas of need as they have delved into their data. Motivated and enthused, they often want to tackle them all, but could easily become overwhelmed if they did. Attempting to make too many changes at once is not an effective strategy. This is why the first step of the planning stage involves prioritization. Participants must make tough decisions about what issues to work on first. After all, it is better to do fewer things and do them well than to try to do a lot of things and do none of them well.

		Next, they must set or modify their school reform goals. Goal setting seems straightforward, but creating "SMART" goals is an acquired skill. SMART is an acronym for Specific, Measurable, Action-oriented, Realistic, and Timely. During the data retreat, participants are taught to use the SMART criteria, as described below, when setting goals.

		Specific: A specific goal has a much greater chance of being accomplished than a general goal. If a goal is too broad, it can be difficult to focus on how it will be achieved. State precisely what needs to be accomplished. Measurable: Concrete criteria must be established to measure progress toward goals so that educators will know if and when the goals have been achieved. Measures can be quantitative or Qualitative, depending on the type of goal being set. For instance, a goal to improve parental involvement by holding regular parent and community meetings may be evidenced by: Quantitative data: Number of parents attending each meeting Qualitative data: Sign-in sheets from meetings, minutes of meetings

		Action-oriented: Action steps, or strategies, must be identified to clarify the process that will be pursued to achieve the goal. In addition, the individual who will be accountable for ensuring the action steps occur must be identified. Realistic: The goal must represent an objective toward which educators are both willing and able to work. A goal can be both high and realistic. Timely: A target date for accomplishing the goals should be identified. Consider setting a deadline that is motivating but achievable. If the deadline is too far in the future, it can be easy to procrastinate. If it is too soon, it may not allow enough time to actually implement the action steps and may dishearten educators in the end.

		Goals should also show alignment to the following criteria: System Goals: Local (school) goals should align to system-wide goals. For example, a school with problematic attendance would want to have a goal that would work towards a minimum of 90 percent attendance rate for all children in keeping with the OIEP goal.

		Data Trends: The need for a goal should be clearly evidenced by trends in the data. For example, if a school decides to focus on vocabulary enrichment as an area for improvement, there should be evidence from various assessment sources that this is an area of weakness in student achievement and/or instruction. Other School Goals: There should be no contradictory strategies or outcomes among various school goals. For example, if one goal aims to begin each day with targeted reading instruction to improve student achievement in this area, another goal should not involve pulling students out of that class every week for a school-wide assembly. In this case, the second goal would interfere in the achievement of the first.

		Periodically, it is necessary to modify existing goals. Data retreats provide the opportunity to study these goals, evaluate progress made toward them and determine if changes are needed. When goals have been achieved, this should be celebrated and shared with school stakeholders. Then, the goals should be updated or removed from the school reform plan. For example, if a school's existing goal is to have 70 percent of the students at the proficient / advanced level in Reading and this goal has been met, then the team has the opportunity to update the goal. Perhaps they can aim for 80 percent of their students to fall into that category over the next two years. On the other hand, if a school's existing goal is to obtain new reading textbooks and this has been achieved, then they may remove this goal and move their focus to other issues, such as how these texts are being used to support instruction.

		The final step of the planning stage, and of the data retreat as a whole, is planning for implementation. It is not uncommon for schools to spend enormous amounts of time setting goals and creating a beautiful binder representing their school reform plan, only to place the binder on a shelf, never to be viewed again until someone asks to see it. That is the risk when schools stop short of planning the implementation process. What do we mean when we use the word "implementation"? Recall for a moment the continuous improvement cycle.

		Depending on where a school is in their continuous improvement journey, a data retreat helps them significantly with the "Plan" or "Study" step. Schools that have never used their data to guide their goal setting and improvement planning can consider a data retreat the initial "Plan" step of their improvement process. More experienced schools that have done some data-driven planning and have implemented their plan, can use the data retreat to "Study" the effects. The middle step is the "Do" step, which is where implementation occurs and data are collected on the process.

		It is not always easy to take a written plan and apply it to a real life situation. After focusing exclusively on data-driven improvement during a data retreat, staff members return to the realities of school life, when distractions and other professional demands may pull them in many different directions. Improvement teams must anticipate and prepare for this phenomenon. After defining their goals, teams should identify potential forces that will drive or prevent the desired change.

		What do we mean by "forces"? Force is the capacity to cause change, often in the face of resistance. In education, there are forces that drive change and forces that prevent, or resist change. All goals have these driving and preventing forces, many of which can be anticipated. When school improvement teams make the effort to predict these, it helps them prioritize the steps they should take in the implementation process. Their first priority should be to remove preventing forces that will get in the way of the desired change.

		One way to structure this type of analysis is by using a tool called a "force field analysis". This is a simple table with the headings "Driving" and "Preventing."

		Under each heading, teams brainstorm ideas, developing a predictive picture that can be used to guide their implementation process. For example, a goal that aims to improve reading achievement by implementing a particular reading method may be prevented if teachers are inexperienced in this method or if they do not have an assessment program in place to evaluate the reading program's impact on student achievement. These two ideas are listed under the word "Preventing."

		Therefore, among the first things the school will need to do to address their reading goal will be to provide professional development to teachers in the reading program so they understand the theory behind the program as well as how to provide instruction within it. In addition, the school will need to identify assessments that can be administered and analyzed to evaluate the program's impact. It is only after these issues have been addressed that the school can capitalize on the other important factors, the driving forces for change.

		On the force field analysis, the forces that will drive change are listed under the word "Driving."

		For example, the same goal to improve reading achievement through the implementation of a new reading program may be driven by forces such as strong leadership and a plan to hold weekly meetings to assess teacher's progress with the reading program and offer additional support. The team should then identify ways to build on these driving forces. For instance, it should specify what school leaders will do to support the achievement of this goal, such as helping to describe the new reading program to parents and community members, helping to facilitate the weekly progress meetings, or providing ongoing classroom observations to offer teachers feedback on their implementation of the reading program. In addition, the team should indicate the objectives of the weekly progress meetings and what supports will be provided to teachers in need.

		Once again, the force field analysis helps identify some of the first steps that should be taken towards attaining specific improvement goals. These steps, together with actions identified within the SMART goal, all comprise the process and path a school will take as they implement their plan or undergo the "Do" step of the continuous improvement process.

		Conclusion A data retreat is a collaborative process that addresses common barriers to change in schools by helping educators better understand their own data as well as learn a process for using it. Data retreats jumpstart profound, "second-order" change by combining analytical work with teamwork, group reflection, dialogue, problem solving, and goal setting. In addition to planning and revising goals, data retreats provide the opportunity to consider the "Do" step of the continuous improvement process. That is, participants predict forces that will drive or prevent the desired changes, then identify a course for implementation with these issues in mind. The focus, composition, location, and length of a data retreat vary based on the particular needs of the participating school or schools.

		All data retreats proceed through a similar process that involves setting the context, analyzing the data and planning next steps.

Using Data for Continuous Improvement