International+Country+Comparisons+-+Implications+for+Practice




 * This blog considers existing international assessment programs that are utilized to assess the relative effectiveness of instruction for each country with respect to other countries. **

International tests are increasingly being used as a yard stick to measure student performance and ascertain the effectiveness of school systems. Countries are using international test rankings as a way to understand how much their students know relative to those from other countries and how such information could be used to make improvements in education and training systems in what is acknowledged to be an ever more competitive global environment. Also, stakeholders in many countries recognize a correlation exists between the knowledge and skills with which young people enter the workforce and long-term economic competitiveness their economies. As such any investments in education are anticipated to produce social and economic benefits for society.

With this increased attention to international test data, comes scrutiny and calls for caution. Some have argued that the interpretations of such data are oversimplified, frequently exaggerated, and misleading (Carnoy & Rothstein, 2013). Specifically, they note that conclusions based on these tests ignore the complexity and contexts within which the tests are taken and it might be imprudent for policymakers to use the results as drivers for education reform.

Below is a brief overview of the most common international tests, concerns raised about the assessments, and implications for practice in developing countries. Program for International Student Achievement (PISA)

= = media type="youtube" key="q1I9tuScLUA?rel=0" width="458" height="258" align="right"




 * A brain child of the Organization for Economic Co-operation and Development (OECD)
 * Launched in 1997 to measure 15-year-old students' reading, mathematics, and science literacy.
 * Tests are given every 3 years – with focus given to one subject in each year of assessment.
 * In 2012, over 70 countries and economies – 34 OECD member countries and 36 non-member partners – participated in PISA testing.
 * Data derived from this test is increasingly being used to assess the impact of education quality on incomes and growth, and for understanding what causes differences in achievement across nations

Sample ranking shown below.



Trends in International Mathematics and Science Study (TIMSS)




 * Launched in 1995, and developed by the International Association for the Evaluation of Educational Achievement (IEA).
 * Lets participating nations to compare students' educational achievement across borders.
 * Measures trends in mathematics and science achievement at the **fourth** and **eighth** grades.
 * Tests is conducted on a regular 4-year cycle.
 * <span style="font-family: 'Baskerville Old Face',serif; line-height: 1.5;">In the latest round of testing done in 2011, nationally representative samples of students in 63 countries and 4 <span style="font-family: 'Baskerville Old Face',serif; font-size: 19px; line-height: 27px;">bench marking <span style="font-family: 'Baskerville Old Face',serif; line-height: 1.5;"> entities (regional jurisdictions of countries, such as states) participated in TIMSS.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Unlike PISA, TIMSS has content or subject matter dimensions students are expected to know by the 4 <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif;">th <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;"> and 8 <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif;">th <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;"> grades.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Additionally, the test covers a cognitive dimension which encompasses a range of cognitive processes involved in solving problems throughout the primary and middle school years

**//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 16pt;">So, how well do PISA and TIMSS compare? //** media type="youtube" key="zCe-7rk4BWU?rel=0" width="560" height="315" align="center"

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 20pt; line-height: 1.5;">Progress in International Reading Literacy Study (PIRLS)






 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Developed IEA, and first assessment was done in 2001.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">PIRLS is conducted on regular 5-year cycles.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Measures **<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">trends **<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;"> in reading comprehension at the **<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">fourth **<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;"> grade level.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Students respond to questions designed to measure their reading comprehension across two overarching categories for reading: reading for literary experience; and reading to acquire and use information
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">In 2011, PIRLS was expanded to include prePIRLS, a stepping stone to participating in PIRLS and a way to assess reading at the end of the primary school cycle for a range of developing countries. Depending on a country’s educational development, prePIRLS can be given at the fourth, <span style="color: #1f497d; font-family: 'Times New Roman',serif; font-size: 14pt;">ﬁ <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">fth, or sixth grade.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Both TIMSS and PIRLS were assessed in 2011, when the cycles of both studies came into alignment.

<span style="color: #1f497d; display: block; font-family: 'Baskerville Old Face',serif; font-size: 20pt; text-align: center;">Regional Assessments for Developing Countries

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Regional testing programs have been used to evaluate performance for many countries in Latin America and Sub-Saharan Africa. <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Some reasons cited for the growth in popularity of regional testing:
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">The need for regional and international benchmarking
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Persistent low scores for some participating countries on OECD and IEA based tests have rendered that data unusable
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">The increasingly popular assumption that students 'have' to be tested and compared to similar cohorts of students in other countries.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">In some cases comparison tests are introduced as a precondition for education funding (loans & other aid / support) by donor agencies.

**<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 16pt;">Major regional testing bodies in developing countries ** <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Latin American Laboratory for Assessment of Quality in Education (LLECE) <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Southern and Eastern African Consortium for the Monitoring of Education Quality (SACMEQ) <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Program for the Analysis of Educational Systems of the CONFEMEN (francophone Africa) countries (PASEC).



<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">These regional assessments have much in common with the international assessments, but there are several important differences, including: <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">The overlap in expertise between the specialists working on the international and regional levels has generally meant that these regional tests are given substantial credibility.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">The relatively greater proximity in content between test and curriculum;
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Normative scales that may or may not be tied to local (normed) skill levels;
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">And attention to local policy concerns (such as the role of the French language in PASEC countries)


 * //<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">For more information on regional testing bodies check out: //**

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">The Latin American Laboratory for Assessment of Quality in Education (LLECE) <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">[|Southern and Eastern African Consortium for the Monitoring of Education Quality (SACMEQ)] <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Program for the Analysis of Educational Systems of the CONFEMEN countries (PASEC). <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">

**<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 18pt;">Use of data from testing organizations ** <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">International assessments are designed to measure learning across multiple countries. By providing comparison rank tables that order the achievement scores on countries by test scores, it has been possible to conduct both cross-national comparisons as well as within country analysis of education systems. Generally, participating countries have used these tests to explore educational issues, including:
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Monitoring system-level achievement trends in a global context,
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Establishing achievement goals and standards for educational improvement,
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;"> Stimulating curriculum reform,
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Improving teaching and learning through research and analysis of the data,
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Conducting related studies (e.g. monitoring equity or assessing students in additional grades), and
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Training researchers and teachers in assessment and evaluation

**<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Unintended consequence of the rankings include ** <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">In the United States, for example, few events in education have had as galvanizing an effect on public debate as the release of international test scores. Much of the public ire has revolved around the relative fall in ranking for US students and the implications such a fall has on the future economic prospects of the country. The inevitable casualties of the public discourse have been public schools that are deemed to be ineffective in educating the youth. <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">As international tests play an ever significant role in ascertaining the quality of a country’s education system, policy makers are seeking out opportunities to learn from high performing countries about education aspects such as teacher quality and effectiveness; curriculum and academic standards; school funding and fiscal equity; governance; and the role and responsibilities of civic institutions in shaping the educational infrastructure. <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Data from surveys indicate that there is a wide disparity in participating countries’ status and caliber of teachers. To improve test score standings, and effectively implement challenging educational reforms, policy makers acknowledge the need to develop high quality teaching professionals who will assume high levels of responsibility. Testing organizations’ recommendations are being taken to heart by countries seeking to improve the recruitment, retention, and evaluation of high quality and effective teachers. <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Data from international tests has been used to drive research into various aspects of countries’ educational systems.
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Public debates on a country’s education quality: **
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Policy makers’ clamor for ‘what works’ in highly ranked countries. **
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">A driver for significant changes in professional development programs. **
 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Research **

**<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 18pt;">Issues of comparability ** <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">Comparability is at the heart of international assessment. In today’s world of cross-national contacts and interdependence, issues related to the quality of education systems products, particularly those of educational achievement are of particular concern to education stakeholders. The ranking of the United States in TIMMS, PIRLS and PISA has been used as a driving force and rationale for the implementation of educational reforms. Given the importance international comparisons play to participating nations’ policy development, it is important that data used be credible and fair.


 * <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 18pt;">Issues Include: **

//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Context //

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Given that education in any country is a system that is situated within a cultural context with different natural, economic and social environments, with differing histories and practices, arriving at a consensus on scientifically credible aspects for comparison can be a challenge.

//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Credibility //

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">The tests’ ability to measure what was intended (reliability) and the relevance of the instrument in capturing the defined target characteristics (validity) significantly affect the trustworthiness of the results. Questions of validity abound on whether or not a test would be correspondingly interpreted by a child in the United States when compared to a child in Jarkata, for example.

//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Sampling of skills and populations // <span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt; line-height: 1.5;">To gauge the relative effectiveness of school systems at educating its youth, representative samples of students have to be tested. These sampling procedures could be done by age, grade level, or a combined stratification strategy that takes into account geographical areas – rural vs. urban – and types of schools – public vs. private (LLECE). If skewed, the results could show substantially different combinations of learning experiences between, and or even, within countries. For example, an oversampling of students from the most-disadvantaged schools in a recent U.S. international assessment is assumed to have led to lower average scores.

//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Also related to sampling. . . //

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">These sampling cases also bring forth the persistent irony that many children most in need of better education face. Often times they are systematically excluded from measurement in international tests. The rationales vary from assessment to assessment, and from one national policy to another, and yet the result is the same – those least likely to succeed on tests, and those who are most disadvantaged, represent the groups most often excluded from the sample population for assessment.

//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Implementation //

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Given that international and regional assessments are carried out on 3 or 5 year cycles, questions inevitably arise over how much data, and of which kind, to collect during the test implementation period. The idea that one collects ‘just enough’ data is easier said than done.

//<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Practical issues //

<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">The practicalities at the school level that go into testing can at times be onerous. International assessment can be lengthy and require careful preparation and substantial control procedures. The program quite easily degenerates into a burdensome project that requires substantial resources from the school, its teachers and students. Continued burden inevitably lead to test weariness, and this might impact the quality of responses students give to studies by international organizations. Such issues might have serious consequences on the analysis and interpretation of the data acquired.

**<span style="color: #1f497d; font-family: 'Baskerville Old Face',serif; font-size: 14pt;">IMPLICATIONS FOR PRACTICE **

<span style="font-family: 'Baskerville Old Face',serif; font-size: 14pt;">There are no magic bullets.

<span style="font-family: 'Baskerville Old Face',serif; font-size: 14pt;">According to PISA, schools and countries where students work in a climate characterized by high expectations and a readiness to invest effort, good teacher-student relations, and high teacher morale tend to achieve better results. Throwing money at education by itself rarely produces results.

<span style="font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Secondly, as McKinsey & Company (2007) aptly noted, “The quality of an education system cannot exceed the quality of its teachers.” Good teachers are essential to high-quality education. Once students fall behind early in their educational careers due to ineffective teachers, it can be very difficult to catch up– particularly if they face this disadvantage early in primary school. Finding and retaining effective teachers is not necessarily a question of high pay. Instead, teachers need to be treated as the valuable professionals they are.

<span style="font-family: 'Baskerville Old Face',serif; font-size: 14pt;">To improve student outcomes, countries will have to ensure that education reform initiatives will no longer receive prefabricated wisdom but on the basis of data and best practice. Whereas countries like the United States operate on a fragmented basis and utilize “hit and miss” policies to implement change, Finland operates a system that fosters strong approaches to leadership and encourages system wide networking and that builds lateral accountability.

<span style="font-family: 'Baskerville Old Face',serif; font-size: 14pt;">Finally, a study by Hanushek and Woessmann (2010) found that when cognitive skills, as measured by PISA scores, are correlated with GDP, then the impact of total years of schooling became irrelevant. In other words, how long it took to learn was less important than that learning had occurred. Whatever the path each individual country takes, there is need to ensure that linkages exist between the school and the community, and the school and the economy, so that education should have a meaning in the context that it is practiced. Additionally, many of today’s job titles, <span style="font-family: 'Baskerville Old Face',serif; font-size: 14pt;">and the skills needed to fill them, simply did not exist 20 years ago. Education systems need to consider what skills today’s students will need in future and teach accordingly.