Making the Numbers Come Alive: Basic Data Analysis

		Introduction Once educators have made the commitment to become "data-driven" and enter into the continuous improvement process, they must build competency in data analysis. The act of data analysis involves transforming data into useful information that can help guide decision-making. Having the ability to organize and display data enables educators to effectively plan their work, study their progress, and communicate to stakeholders.

		It is not necessary to become a statistician before data analysis can begin. What is required is an understanding of the questions that will guide the analyses, the available data sources that respond to the questions, and how to represent the data in a way that is meaningful to decision-makers. It is also important that educators understand some of the common strategies for data analysis and how to avoid being misled by statistics.

		The process of data analysis can be represented in four steps. Step one, creating guiding questions, step two, selecting data to analyze, step three organizing the data and step four graphing the data. Each of these steps will be discussed and examples will be provided that align to OIEP's system-wide educational goals.

		Step 1: Creating Guiding Questions Recall that to begin the process of data analysis, schools should first develop guiding questions. Much like researchers, educators must first think through the questions they have at hand. These questions will help educators target the type of data they need to collect and the analyses they should complete. Three guiding questions can be used to help schools cover much of the groundwork necessary to investigate their strengths and needs. Each of these guiding questions can lead to more specific ones that schools create according to the outcomes that are important to them.

		Question 1: Where are we now? This question requires a broad examination of current school outcomes. This is an important starting point because without knowing where a school stands, it is difficult to focus goals and impossible to gauge improvement. Specific versions of this question should be created to respond to school goals. Following are some examples of important school outcomes and specific guiding questions:

		Academic: What are the proficiency levels of our students in each subject area? How do students perform on the state standards in Reading?

		Attendance: What are the attendance rates of our students?

		Participation in extracurricular activities: How many students are participating in athletics and/or clubs?

		Parent involvement: How many parents participated in parent-teacher conferences?

		Language proficiency How many students are fluent in both English and their native language?

		School comparison How does the performance of students in our school compare to the performance of students throughout the agency, within our region, and throughout the state)? For educators to know where their school is, they will want to evaluate whether the school is approaching, meeting, or exceeding these goals.

		Question 2: Has our performance changed over time? To respond to this question, educators will need to evaluate longitudinal outcome data that have been consistently collected over time.

		If a school only analyzes their current status without looking to the past, they will have difficulty putting their performance into context. A series of longitudinal data provides context because it helps illuminate changes and trends in performance. Once again, specific versions of this question should be created to respond to school goals. Following are some examples of how this general question can be tailored into specific guiding questions according to the same outcomes previously presented:

		Academic: Are we getting more students into the proficient/advanced range this year compared to years past?

		Attendance: Are the attendance rates of our students improving?

		Participation in extracurricular activities: Are more students participating in athletics and/or clubs?

		Parent involvement: Are we getting greater parental attendance at parent-teacher conferences?

		Language proficiency Are more students fluent this year in both English and their native language compared to years past?

		School comparison Is our school improving at a rate that is similar, faster, or slower than other schools (throughout the agency, within our region, throughout the state)? When assessing change, educators will want to know whether outcomes are improving, declining, or staying the same. This information provides the key to linking outcome data with demographic and process data. It is when trends in the data can be linked in time to changes in school demographics or processes that the analysis process becomes exciting.

		There are different ways of looking at change. One way to look at change is by tracking changes within a grade level over time. As we know, within a given grade, there are different groups of students from year to year. Therefore, changes in the performance of students in a grade level may indicate issues beyond the educational programs in the school. They may also reflect significant changes in student demographic data. This is called a longitudinal analysis.

		The second way to look at change is by following the same group of students, or cohort, through their educational experience. Changes in the performance of the same group of students may then be linked in time to changes in educational programming or forms of process data. This is called a cohort analysis.

		Question 3: Are we meeting the needs of all students? This question introduces the need to disaggregate, or break down, data. Once schools understand their current status and their degree of improvement, they must assess whether any gaps exist in the achievement of their students. Only by separating the data into different student categories can we assess the issue of equity in our schools. It is here that demographic data become a tool.

		The No Child Left Behind (NCLB) legislation requires that data be disaggregated according to four demographic categories for precisely this reason. The NCLB categories are: Race/Ethnicity Economically Disadvantaged Disability Status Limited English Proficiency (LEP) There may be other demographic categories that are equally or more meaningful to schools, depending on the characteristics of their student populations. These should also be considered when evaluating equity.

		Other important demographic variables may include: Gender Residential/Non-residential Status Length of Enrollment in School Native Language Proficiency

		If we discover that some student groups are consistently higher achievers than others, we have discovered gaps in achievement. As long as achievement gaps exist, some children will be "left behind." Therefore, we must consider any gaps to be "red flags" that require immediate attention.

		Examples Consider OIEP's first improvement goal. It states, "70% of students will be proficient/advanced in Reading/Math." This provides an excellent starting point to create guiding questions for data analysis. Using the three broad questions as a framework, the following guiding questions can be created:

		Where are we now? What percent of our students are proficient or advanced in Reading? What percent of our students are proficient or advanced in Math?

		Has our performance changed over time? Are we getting more students into the proficient/advanced range this year compared to years past?

		Are we meeting the needs of all students? Are students with disabilities achieving at the same rate as those without disabilities? Are students with limited English proficiency (LEP) achieving at the same rate as those that are English-proficient (Non-LEP)? With the three guiding questions defined and applied to each school's unique goals, the next step is to identify data sources that can be used in the analyses.

		Step 2: Selecting Data to Analyze To identify the data we will use in our analyses, we must consider how the outcomes in question are best measured, the demographic variables that are important to us, and the data necessary to reflect the instructional processes at our school. The evidence we rely on to understand our needs can come in the form of quantitative or qualitative data. However, for the purposes of this lesson, we will focus primarily on measures that are quantitative.

		The types of outcome data that can be used to respond to guiding questions will vary according to school goals. Educators must know how each of their goals is being measured in order to choose the appropriate outcome data sources for analysis. Within OIEP, it is likely that schools will want to use data that align to the system's school improvement goals. Consider the following goals and examples of outcome data that align with them:

		Goal 1: 70% of students will be proficient/advanced in Reading/Math Outcome data: Standardized assessments Classroom assessments Report card grades

		Goal 2: All children will read independently at the 3rd grade level by the 3rd grade Outcome data: Classroom reading assessments Reading aptitude assessments

		Goal 3: Individual student attendance will be 90% or better Outcome data: Individual student attendance records

		Goal 4: Students will demonstrate knowledge of their language and/or culture to improve academic achievement Outcome data: Native language proficiency assessments Enrollment in bilingual classes Grades from bilingual classes Attendance at cultural events Number of lessons given using culturally relevant themes

		Goal 5: There will be an improvement in the enrollment, retention, placement, and graduation rates for post-secondary institutions Outcome data: Percent of students graduating high school Percent of students enrolling in post-secondary institutions Percent of students graduating from post-secondary institutions Student survey data related to educational goals

		Educators should aim to use outcome data of the highest quality possible. Computer programmers often use the phrase, "garbage in, garbage out." That is, the functions a computer can do are directly related to the quality of the program, or directions, written for it. If the program is lousy the computer will make errors, and will ultimately frustrate the user. This phrase works for data analysis as well. If the data we use to learn about our schools are inappropriate, the analyses we perform will lose their value, and may even point our decision-making in the wrong direction.

		When we disaggregate our outcome data by demographic variables, we then begin to use these findings to evaluate the processes or methods we use to educate our children. We must conduct this evaluation with a firm grasp of the quality of the data we are using, along with its strengths and limitations. There are two key considerations when evaluating the quality of our data sources; reliability and validity.

		Data that are reliable can be trusted to provide consistent information over time. In order for data to be reliable, we must consider the tools or measures that generate the data. Reliable tools measure the same way each time they are used under the same conditions with the same individuals. The results they yield are repeatable. For example, if we administer a math assessment on a Monday to our class, and, without additional instruction, administer the same assessment again on Tuesday, we would expect similar scores from each student. If the scores differed significantly, we would question the reliability of the test. If we have confidence in the reliability of our measures, we can have confidence in the accuracy of our data.

		Data that are valid measure what we intended to measure. The degree that the data are valid contributes to the strength of our conclusions, inferences or propositions. For example, if an assessment that intended to measure math proficiency had directions that were unfamiliar and difficult for the students, the resulting data from the assessment may actually represent the child's ability to accurately read the directions more than they do the child's proficiency in math. In this case, the directions would need to be reworked to remove this bias from the results and enhance the validity of the assessment.

		If we have confidence in the validity of our measures, we can have confidence that we have measured what we were supposed to measure. After data have been identified for analysis, educators will move through steps three and four: organizing and graphing the data.

		Step 3: Organizing Data Data must be organized in a way that will allow preliminary trends to emerge and will support further analyses down the road. This can be accomplished by putting data into a chart. Charts can be set up in different ways, but a simple way to approach them is as follows:

		Title the chart. The title is an important communication tool. It allows anyone to look at your chart and have a sense of its purpose and message. In addition, it serves to remind educators what they were aiming for when they constructed the chart. Many a person has diligently created charts without labeling them, only to pull the information out at a later time and feel lost. There are several important elements to the title. The title should:

		Identify the source of the data. For example 1, "SAT 9" signifies that the achievement data reflected in the chart were obtained from the Stanford 9 achievement test.

		Indicate the timeframe. In example 1, "2000 - 2002" indicates that three years of data are shown.

		Align to the guiding question. In example 1 that follows, the chart is titled "Percent of students proficient in reading" to align with the first BIA/OIEP improvement goal. Decide if and how the data should be disaggregated. The demographic variables of interest will determine how the data are broken down and organized in the chart. Label the rows within the graph to reflect the demographic variables that are being considered in the analysis.

		In example 1, the data have been disaggregated by grade, a demographic variable in the school. Each row is labeled with a grade level.

		The columns in this example show how the scores change over the course of three years.

		There may be times when data need to be disaggregated based on two demographic variables.

		In this case, the rows can reflect one of the demographic variables, while the columns can reflect another. In example 2, the data have been disaggregated by both grade (rows) and LEP status (columns). This chart allows the educator to make comparisons in reading proficiency within each grade between those students that have been deemed proficient in the English language (Non-LEP) and those that have not (LEP). The "Total" column shows the percent of students that were proficient in each grade as a whole, that is, when data from both LEP and non-LEP students are combined.

		Put data in the cells. Once the chart has been set up, quantitative data can be placed in each cell. These data can represent several concepts, such as: Frequency: the number of students that fall into the category percentage: the percent of students that fall into the category Means: the average score of students within the category In examples 1 and 2, the percentage of students falling into each category has been placed in each cell.

		Step 4: Graphing the Data The next step to data analysis is using the data that are in a chart to create a graph.

		Graphs are powerful because they tend to capture information in an interesting fashion, are easily interpreted, and show trends in the data more readily than charts. Graphs are visual, and many people grasp information better through pictures than from rows of numbers.

		Each graph has the common elements of an "x" and "y" axis. The x-axis is the horizontal line of the graph and often reflects the categories that are being measured or a period of time.

		The y-axis is the vertical line of the graph and reflects the numeric scale for what is being measured. Graphs should be given a title in the same way as charts. There are different kinds of graphs, each more appropriate for certain guiding questions than others. Next, we will link the questions we have with the type of graphic representations that aide our analyses most. Bar graphs are helpful for presenting data from different demographic groups and facilitating comparisons. For that reason, they provide an appropriate tool to display data that respond the guiding question, "Where are we now?" In this type of graph, the data are recorded in bars.

		In the example below, "Where are we now?" is applied to the subject of reading. The data represented in this bar graph have been taken from the 2002 data shown in chart example 1. Each bar represents a different category ("grade 1" or "grade 2") and the heights of the bars indicate a quantity (in this case, the percent of students in each grade that were proficient in reading according to the 2002 SAT 9 achievement test). The scale on the y-axis begins at zero and ends at 100, as that is the range of possible percentage scores.

		In terms of the guiding question, grade 8 has nearly 80 percent of the students in the proficient range, and grades 1 and 4 have over 60 percent. The other grades fall short of these scores. This graph alerts us that there is room to grow in all of the grades.

		It also shows that the percentage of proficient readers in grade 3, at 38 percent, is considerably lower than in the other grades. This should raise a "red flag" to educators.

		Yet before making decisions, they must consider data from this third grade cohort. Return to the chart for these data, and observe that this same cohort was in grade two in 2001 and in grade one in 2000. The percent of proficient students in this cohort in these previous years was 27 and 20, respectively. With this additional information, one can see that in spite of the low percentage of proficient readers, the data from the third graders in 2002 suggest growth and improvement for the cohort. This reminds us to consider all of the available information before we make decisions.

		A variation of the bar graph can be used to respond to the question, "Are we meeting the needs of all students?" In this version, there are two bars for each grade, each in a different color. Each of the bars represents data that have been further disaggregated.

		In the example below, the data for each grade have been broken down into bars representing "LEP" and "Non-LEP" students. The legend at the right tells us which color represents which group. This graph, which was constructed based on the data in chart example 2, quickly shows the disparity in reading proficiency between these demographic groups. The response to the guiding question in this case would have to be "No, we are not meeting the needs of all students." The LEP students in this graph are clearly falling far behind their peers, demonstrating an achievement gap. The school has the responsibility to modify the process of instruction being used for these children to help accelerate their learning.

		Line graphs are used to show change, so they are well suited to respond the guiding question, "Has our performance changed over time?" A line graph is read in the same way as a bar graph. The x-axis usually reflects the time period, and the y-axis shows the numeric scale for what is being measured. A line graph is drawn by dropping points for the data values at each time period, then joining the points with a straight line. When data are disaggregated into demographic groups, lines can be drawn in different colors or the points can be designated by different symbols. In this scenario, a legend for the graph is critical for accurate interpretation.

		Line graphs are particularly useful for visualizing trends in the data. In the example below, one of the clearest trends is that of grade 8. From 2000 through 2002, the percent of proficient readers in this grade has steadily grown. Considering the data representing the 2002 grade 8 cohort in chart example 1, we find that the cohort has indeed made remarkable gains as well. This is a "double" success story within the school. The eighth grade has made consistent gains in the percent of proficient readers since 2000, and the 2002 eighth grade cohort has more proficient readers than ever before.

		The final type of graph that will be discussed is known as the scatterplot. This graph does not align directly with the three guiding questions previously discussed. Instead, it is used as a tool for further exploration when hypotheses begin to be generated. Scatterplots help investigate hypotheses about relationships between school phenomena and therefore respond to a new guiding question, "What is the relationship?" On a scatterplot, one data point, plotted according to numeric scales on the x- and y-axes, represents an individual with scores on two variables. In scatterplot example 1, the relationship between reading proficiency and attendance rate is shown for all students. The x-axis is labeled with a scale for attendance rate and the y-axis represents NCE scores for reading. This scatterplot may have stemmed from a hypothesis that attendance has an impact on reading performance.

		Scatterplots can also show if relationships vary for different demographic groups. In this case, the data points can be color-coded for easy distinction. Scatterplot example 2 shows the same relationship as the first, but disaggregates the students into LEP and Non-LEP students.

		In both cases, the process of analyzing the scatterplot is similar. If there is a strong relationship (also known as correlation) between the variables, then the data points will form a shape, usually a straight line. The stronger the relationship between the variables, the tighter this line should be. A positive relationship occurs when the values of the variables increase or decrease together (for example, as attendance rates increase, so do levels of reading proficiency). Negative relationships can also occur when the variables are inverses (for example, as levels of reading proficiency increase, the number of behavioral incidents decrease). When there is no relationship between the variables, then the points will not form a line or a shape, and will look more like an irregular "cloud."

		Scatterplot example 1 shows a strong, positive relationship between attendance and reading achievement. The students with higher rates of attendance tend to have higher levels of reading proficiency as well. Although the relationship is not perfect, if we draw a line through the middle point of the dots (also known as the "line of best fit"), we see that the students cluster around the line pretty tightly.

		In example 2, the dots fit more tightly around the line of best fit for the LEP students than they do for the Non-LEP students. Both graphs suggest that good attendance is important and may contribute to reading performance. However, the second graph suggests that attendance is particularly important for LEP students.

		Being Data Savvy In this lesson, we have learned how to move through a data analysis process, how to organize our data, and how to construct graphs that help us visualize trends and make decisions. There are four additional concepts to learn to ensure that we do not allow ourselves to be misled by our analyses.

		Pay attention to the number of students represented by the data. The number of students involved in the analysis, sometimes referred to as the "N," is important, particularly when we begin to disaggregate the data. Anytime we break the data down to the point where there are only a few students represented by any given category, we must be extremely cautious in our decision-making. In our previous examples, we have made comparisons between LEP and non-LEP students. We have even disaggregated the results of these students by grade level and made comparisons between the groups. But what if there were only 20 LEP students in the entire school? That would mean that the comparisons in each grade would be based on very few students. When a group statistic, such as "third grade LEP students" is represented by a small number of group members, such as three students, the results of any one student can easily swing the results. For instance, if only one of the three students was proficient, 33 percent of the third grade LEP students would be proficient. But if only one more student made the leap to proficiency, that would mean two out of three were proficient, or 66 percent. In this example, critical decisions related to curriculum and instruction should probably not be based on a group statistic. There are different rules-of-thumb for what "N" is sufficient for decision-making and the correct answer really varies based on many issues. It is often better, with small sized groups, to view individual student results.

		Seek several sources of data when making decisions. Charts and graphs of data can be very compelling and can prompt us to take action. That is the purpose of data analysis. Yet whenever possible, we should attempt to verify trends with additional sources of information. For example, if data from a standardized assessment indicate that our fifth grade students are weak in multiplication facts, we should check other sources before changing our curriculum or switching to a new text. Other sources may include classroom assessments, homework grades, or teacher observations. When multiple sources of data confirm the same trend, we can have additional confidence in the validity of the trend itself. And that leads to more confident decision-making.

		Graph according to an accurate scale. The scale of any graph we draw should ideally represent the entire range of possible scores. In addition, scales from one graph to the next should be consistent. This allows us to view any graph and rapidly assess trends in the data. When a graph only represents part of the possible scale, it can easily be misinterpreted. Below, two of the graphs presented earlier have been modified to show how they look with an incomplete scale.

		On the bar graph, one might assume that nearly every eighth grader is a proficient reader since the bar almost reaches the top of the graph. Yet upon further examination, we can see that the scale ends at 80 percent. This makes it look as if there is a larger percent of proficient readers than there actually is.

		On the line graph, it looks as if the percent of proficient readers in grade 4 has jumped tremendously from 2000 to 2002. Although there has been growth, when looking at the abbreviated scale, one can see that the growth represents only about thirteen percentage points. In both cases, the scale should range from zero to 100 to get an accurate picture of the data.

		Correlation does not mean causation. When a relationship, or correlation, between variables is revealed by a scatterplot, it provides additional insight into the data and can guide our thinking. But a relationship between two variables, such as achievement and attendance, is not sufficient data to imply a pattern of cause and effect. That is, just because they are related does not mean that good attendance causes high achievement, or that high achievement causes good attendance. It is not that simple.

		There are many other variables that can contribute to achievement, such as the quality of the instruction, the curriculum, the materials available in the school, etc. Similarly, various issues, such as student health, parent emphasis on attendance, or school policies, can affect attendance. We must always keep the complexity of issues in mind when we evaluate our data and take care not to jump to hasty conclusions.

		Summary Educators need not be statisticians to analyze their data. Four simple steps form the foundation of the data analysis process and can fit into the larger continuous improvement process. First, guiding questions must be formed to focus the analyses. Then, data must be chosen that respond to the questions and are of adequate quality to allow for confident decisions. Next, these data should be organized into charts and then visually displayed in graphs. Depending on the question at hand, data may be disaggregated to evaluate performance according to various demographic groups. Educators must remember that no matter how compelling their charts or graphs may be, they should attempt to consider multiple sources of information. When multiple sources of information confirm the same trends, they can have more confidence in the validity of their conclusions and in their future decisions.

Introduction to Data-Driven School Improvement