This article offers a framework for evaluating agroecological principles, practices, and claims. Advocates of agroecology offer principles for sustainable, productive, and efficient agricultural systems. The evaluation question that immediately and inevitably arises in the face of such claims is: Does agroecology work? Thus do 2 fields of inquiry intersect: evaluation and agroecology. But there are many approaches to and models of evaluation. The challenge is to match the evaluation framework to the nature of agroecology. Because agroecology is based on principles, the appropriate evaluation approach is principles-focused evaluation. This article (1) elucidates principles-focused evaluation and demonstrates its relevance to evaluating agroecological practices and claims, (2) offers a principles-focused evaluation framework for monitoring agroecological transition and intensification within a particular context, and (3) concludes with lessons about principles-focused evaluation from agroecology.
Introduction
How can the entirety of a food system, including production, processing, transportation, financial intermediation, marketing, and consumption, be ecologically sustainable, agriculturally productive, and economically efficient? This is the big picture evaluation question. Advocates of agroecology offer a framework for sustainable, productive, and efficient agricultural systems. The evaluation question that immediately and inevitably arises in the face of such claims is: Does agroecology work? Thus, do 2 fields of inquiry intersect: evaluation and agroecology. This article examines that intersection. In so doing, I’ll examine both possibilities and pitfalls. Let me begin with evaluation.
Evaluation
I date the beginning of the evaluation profession to 1975 and publication of the 2-volume Handbook of Evaluation Research (Guttentag and Struening, 1975). That was also the era when the first professional evaluation associations were formed. There are now more than 200 national and regional organizations for professional evaluators around the world involving an estimated 75,000 evaluators. The year 2015 was designated as the International Year of Evaluation by the United Nations as part of the transition from the Millennium Development Goals (2000–2015) to the Sustainable Development Goals (2015–2030). The global evaluation profession has a number of evaluation journals, handbooks, textbooks, training institutes, annual conferences, webinar series, and online libraries.
Everything is subject to evaluation. Government agencies, international organizations, nongovernmental organizations (nonprofits), private sector initiatives, philanthropic foundations, universities, movements, networks, and interventions of all kinds are subject to evaluation even as they engage in evaluating others. Questions of accountability, effectiveness, efficiency, and impact are classic evaluation questions. The formal logic of evaluation involves 4-steps:
define the criteria that will be used to evaluate something,
set standards of performance on those criteria,
measure the actual performance, and
synthesize the results to reach an evaluative judgment.
Standard evaluation requirements include specifying SMART goals (specific, measurable, achievable, relevant, and time-bound) with performance indicators. Programs are expected to have logic models and theories of change. But over the course of 5 decades evaluation models, approaches, framework, methods, and measures have proliferated: impact evaluation, utilization-focused evaluation, theory-driven, cost–benefit, empowerment, participatory, social justice, realist, and culturally responsive, to name but a few examples. Standards have been formulated and adopted specifying what constitutes evaluation quality and checklists are available for what should be included in an evaluation. Principles-focused evaluation (Patton, 2018) is a new direction in evaluation that offers a framework for evaluating principles-based interventions and frameworks like agroecology.
Given the diversity and complexity of the evaluation profession, the challenge is one of matching: what approach to evaluation is appropriate for a particular situation, purpose, and evaluand (the thing being evaluated in the jargon of the field)? That brings us to the alignment between agroecology and principles-focused evaluation.
Agroecological principles
Agroecology is multifaceted: science, practices, and movement. The science of agroecology consists of knowledge about agroecological systems. Practices specify how to apply the science. The movement advocates values of sustainability, diversity, equity, and inclusion that inform the science and practices while seeking to ensure a supportive policy environment. There are diverse, sometimes conflicting, and still emergent perspectives on what agroecology is. Where agroecology coheres is being grounded in and guided by principles. Discussion and debate about which principles are essential illustrates the critical role of principles in defining agroecology. Principles matter, thus the intensity of the engagement about which principles, worded in what ways, are essential. Evaluative thinking and judgment can contribute to both the principles clarification process and assessing the degree of adherence to principles. Therein lies the challenges and opportunities where evaluation and agroecology intersect.
Principles of agroecology
Compared with conventional agricultural systems, agroecological farms seek to support biodiversity, enhance carbon sequestration, and utilize resources more efficiently by farming according to ecological principles (Ong and Liao, 2020, p. 1). In his seminal textbook and influential website, Steve Gliessman of the University of California, Santa Cruz, identified the comprehensive list of overarching agroecological principles presented in Table 1.
Overarching principles of agroecology and sustainability. DOI: https://doi.org/10.1525/elementa.2021.00052.t1
|
|
Source: Adapted from Gliessman (2014, 2015).
Later, Ernesto Méndez, at the University of Vermont, adapted and used these principles to create a tool for analyzing the potential agroecological impact of agricultural interventions.
Analyzing the potential of agriculture interventions to do harm to food production systems and create opportunities to build resilience within them requires looking at the entire food system. This includes the agroecological, biological, farming and market, social and political systems, as well as the interactions between these components. A deep consideration of these sometimes complex interactions is necessary in order to ensure program interventions in one area of a food system do not have unintended negative effects in other dimensions of the food system and inadvertently increase vulnerability of households or the agroecosystem. The overall goal of this screening tool is to support productivity and income generation for smallholder farmers with a long term vision towards resilience. The tool utilizes a food system perspective and agroecological principles to guide those who design and implement agriculture and food security programs through a series of inquiries into how their interventions may affect the resilience of food systems and of those who depend on them. This application of agroecological principles to context analysis and program design is especially useful in minimizing the depletion of crucial natural capital of smallholder farmers. By identifying opportunities to leverage and maximize available resources in resource scarce environments, programs may enhance the resilience of productive, sustainable agroecosystems and the households, markets, and food systems which rely on them. (Méndez, 2010, p. 2)
These agroecological principles include the following:
Preservation and enhancement of agroecosystem diversity
Conservation and enhancement of soil health and nutrient cycling
Supporting ecological pest and disease regulating mechanisms
Maximizing renewable energy potential
Supporting and diversifying livelihoods to manage and mitigate risk exposure
Prioritizing and enhancing local food production and food security
Reducing dependence on external synthetic inputs
Optimizing water use—preserving and regenerating water resources
Integrating local and scientific knowledge
Strengthening local organizations (Scarborough et al., 2014, p. 1)
There is, then, a broad consensus that agroecology is built around a set of principles, derived both from ecological principles, and a strong ethic of valuing farmer and local knowledge. There is also the sense that these principles should provide—not prescriptive advice—but a “design framework” to be optimized and adapted to local contexts (Gemmill-Herren, 2016, p. 5).
Because agroecology supports locally derived solutions, “rather than a set of context-specific technologies, agroecology uses a set of principles that can be locally disseminated and adapted to multiple contexts” (Méndez et al., 2015, p. 3).
Evaluating agroecological principles
As the works of Steve Gliessman, Ernesto Méndez, Barbara Gemmill-Herren, and others illustrate, agroecology has developed as a field, and is still developing, through the identification, articulation, testing, refining, debating, and revising of principles. Undergirding and manifest throughout every aspect of principles development is principles-focused evaluation. The emergent challenges for evaluating agroecological principles have to do with new units of analysis and broader areas of focus for evaluation, what we call the evaluand, that is, the thing evaluated.
Evaluation origins were in evaluating projects. Such tools as logic models and SMART goals work well for project evaluation. They do not work well, in my judgment, for different kinds of evaluands like agroecological principles and systems transformation initiatives. Projects are closed systems, or at least treated as such in most evaluations, in which boundaries can be established and control can presumably be exercised within those boundaries by both program staff and evaluators. In contrast, complex dynamic interventions and principles-driven strategic initiatives are open systems characterized by volatility, uncertainty, and unpredictability, all of which make control problematic. Treating these complicated and complex evaluations like more simple projects is inappropriate, ineffective, and insufficient. Indeed, it can do harm by misunderstanding, misconceptualizing, and misrepresenting the very nature of complex change and thereby generating results that are inaccurate and irrelevant.
The key point here is that evaluating principles is different from evaluating projects. Principles-driven interventions and initiatives are different from goals-driven programs. Principles-focused evaluation can evaluate processes of implementing principles, outcomes associated with principles, longer term and broader impacts that result from principles-driven programming, and innovative approaches to principles adaptation. Principles-focused evaluation can serve a variety of purposes: accountability, formative, summative, developmental, and knowledge generating. In all these applications, principles-focused evaluation should be utilization-focused.
Principles-focused evaluation questions
Three core questions bring the focus to principles-focused evaluation:
To what extent have meaningful and evaluable principles been articulated?
If principles have been articulated, to what extent and in what ways are they being adhered to in practice?
If adhered to, to what extent and in what ways are principles leading to desired results? (Patton, 2018)
These evaluation questions flow from and depend on well-conceptualized, evaluable principles. What constitutes a high-quality principle? The GUIDE framework offers 5 criteria.
GUIDE framework for principles
The GUIDE framework for developing and evaluating principles specifies that principles should provide clear guidance [G], be useful for informing decisions and actions [U], be inspirational [I], be developmental, that is, adaptable to diverse contexts [D], and be evaluable [E]. Table 2 presents the GUIDE criteria.
GUIDE framework and rubric for effectiveness principles. DOI: https://doi.org/10.1525/elementa.2021.00052.t2
| GUIDE . | Each Criterion Defined . | Specific Relevance to Agroecology . |
|---|---|---|
| G = Guiding | A principle is prescriptive. It provides advice and guidance on what to do, how to think, what to value, and how to act to be effective. It offers direction. The wording is imperative: Do this…to be effective. The guidance is sufficiently distinct that it can be distinguished from contrary or alternative guidance. | SCIENCE: Agroecological principles are based on science. This captures the scientific foundation of agroecological guidance. |
| U = Useful | A high-quality principle is useful in making choices and decisions. Its utility resides in being actionable, interpretable, feasible, and pointing the way toward desired results for any relevant situation. | PRACTICE: This criterion is where agroecological principles are converted into practices. |
| I = Inspirational | Principles are values-based, incorporating and expressing ethical premises, which is what makes them meaningful. They articulate what matters, both in how to proceed and the desired result. That should be inspirational. | MOVEMENT: The values and ideals of the agroecological movement are expressed in this criterion. |
| D = Developmental | The developmental nature of a high-quality principle refers to its adaptability and applicability to diverse contexts and over time. A principle is thus both context sensitive and adaptable to real-world dynamics, providing a way to navigate the turbulence of complexity and uncertainty. In being applicable over time, it is enduring (not time-bound), in support of ongoing development and adaptation in an ever-changing world. | CONTEXT: Agroecology is highly context-sensitive. The adaptive (developmental) nature of agroecology is translating principles into practices within a specific farming systems ecological context |
| E = Evaluable | A high-quality principle must be evaluable. This means it is possible to document and judge whether it is actually being followed, and document and judge what results from following the principle. In essence, it is possible to determine if following the principle achieves desired results. | RESULTS: This criterion addresses the results of agroecological transition and intensification. This criterion empirically tests the science, practice, values, and contextual relevance of agroecology. |
| GUIDE . | Each Criterion Defined . | Specific Relevance to Agroecology . |
|---|---|---|
| G = Guiding | A principle is prescriptive. It provides advice and guidance on what to do, how to think, what to value, and how to act to be effective. It offers direction. The wording is imperative: Do this…to be effective. The guidance is sufficiently distinct that it can be distinguished from contrary or alternative guidance. | SCIENCE: Agroecological principles are based on science. This captures the scientific foundation of agroecological guidance. |
| U = Useful | A high-quality principle is useful in making choices and decisions. Its utility resides in being actionable, interpretable, feasible, and pointing the way toward desired results for any relevant situation. | PRACTICE: This criterion is where agroecological principles are converted into practices. |
| I = Inspirational | Principles are values-based, incorporating and expressing ethical premises, which is what makes them meaningful. They articulate what matters, both in how to proceed and the desired result. That should be inspirational. | MOVEMENT: The values and ideals of the agroecological movement are expressed in this criterion. |
| D = Developmental | The developmental nature of a high-quality principle refers to its adaptability and applicability to diverse contexts and over time. A principle is thus both context sensitive and adaptable to real-world dynamics, providing a way to navigate the turbulence of complexity and uncertainty. In being applicable over time, it is enduring (not time-bound), in support of ongoing development and adaptation in an ever-changing world. | CONTEXT: Agroecology is highly context-sensitive. The adaptive (developmental) nature of agroecology is translating principles into practices within a specific farming systems ecological context |
| E = Evaluable | A high-quality principle must be evaluable. This means it is possible to document and judge whether it is actually being followed, and document and judge what results from following the principle. In essence, it is possible to determine if following the principle achieves desired results. | RESULTS: This criterion addresses the results of agroecological transition and intensification. This criterion empirically tests the science, practice, values, and contextual relevance of agroecology. |
Source: Adapted from Patton (2018).
From principles to practices
As the GUIDE framework illustrates, principles provide overall direction but have to be translated into and interpreted for application within a specific context. That involves converting principles to practices. Moreover, not all principles are adopted or adapted at the same time. The transition to agroecology often involves stages of applying and implementing agroecological principles. Given the importance of the transition process, Table 3 presents a principles-focused evaluation framework for monitoring agroecological transition and intensification within a context.
A principles-focused evaluation framework for monitoring agroecological transition and intensification within a particular context. DOI: https://doi.org/10.1525/elementa.2021.00052.t3
| Agroecological Principle . | Actual Practices Informed by This Principle Within a Specific Agroecological/Farming System Context . | Baseline Agroecological Situation Related to the Principle . | Implementation Rubric: Degree of Fidelity to the Principle in Practice . | Changes Resulting From Implementing the Principle: Farming Results . | Lessons Learned About Adapting This Principle to Practice Within This Context . |
|---|---|---|---|---|---|
| PART I. Evaluation of agroecological principles adopted and implemented | |||||
| Enter practices | Baseline data |
| Change data | Lessons |
| Enter practices | Baseline data | Repeat rubric | Change data | Lessons |
| PART 2. Interconnectedness evaluation question: How do the principles and practices adopted and implemented relate to each other within this context? | |||||
| PART 3. Agroecological principles not yet adopted and implemented as practices within this context | |||||
| Specify a potentially relevant agroecological principle relevant to this context | Potential practices informed by this principle within a specific agroecological/farming system context | Baseline agroecological situation related to the principle | Barriers to adopting and implementing this principle | Plan, if any, for transition to this principle | Enabling factors to track |
| 3-a. Specify a potentially relevant agroecological principle relevant to this context | Enter practices | Baseline data | Barriers | Transition plan | Enabling factors to track |
| 3-b. Additional relevant principles | Enter practices | Baseline data | Barriers | Transition plan | Enabling factors to track |
| Agroecological Principle . | Actual Practices Informed by This Principle Within a Specific Agroecological/Farming System Context . | Baseline Agroecological Situation Related to the Principle . | Implementation Rubric: Degree of Fidelity to the Principle in Practice . | Changes Resulting From Implementing the Principle: Farming Results . | Lessons Learned About Adapting This Principle to Practice Within This Context . |
|---|---|---|---|---|---|
| PART I. Evaluation of agroecological principles adopted and implemented | |||||
| Enter practices | Baseline data |
| Change data | Lessons |
| Enter practices | Baseline data | Repeat rubric | Change data | Lessons |
| PART 2. Interconnectedness evaluation question: How do the principles and practices adopted and implemented relate to each other within this context? | |||||
| PART 3. Agroecological principles not yet adopted and implemented as practices within this context | |||||
| Specify a potentially relevant agroecological principle relevant to this context | Potential practices informed by this principle within a specific agroecological/farming system context | Baseline agroecological situation related to the principle | Barriers to adopting and implementing this principle | Plan, if any, for transition to this principle | Enabling factors to track |
| 3-a. Specify a potentially relevant agroecological principle relevant to this context | Enter practices | Baseline data | Barriers | Transition plan | Enabling factors to track |
| 3-b. Additional relevant principles | Enter practices | Baseline data | Barriers | Transition plan | Enabling factors to track |
CONTEXT: Specify the agroecological context (e.g., millet farming in the savannah of Burkina Faso).
Lessons about principles-focused evaluation from agroecology
Here are 6 lessons about principles-focused evaluation related to agroecology.
Generating alternative principles can facilitate dialogue and deliberation about the essential contributions of a new and emergent field of knowledge and practice. In the case of agroecology, diverse sets of principles provide evidence about areas of agreement as well as dimensions of difference and debate. Some sets of principles are quite comprehensive and multifaceted while others are short and focused. These differences help define the boundaries of this innovative and still emergent field of inquiry and knowledge.
Principles, by their very nature, emphasize the importance of context. Agroecological principles encapsulate interdisciplinary knowledge and sustainable farming practices at a general level, but because agroecological systems vary significantly, the specific practices that follow from these principles within a given context will vary. Evaluation includes examining the translation of principles into practices within a context.
Evaluating the principles on shared criteria is how the field advances. Principles-focused evaluation supports evolution and further development of this emergent field of knowledge and practice. Evaluating the principles supports coherence about what is essential as the scientific knowledge base expands and coheres based on research and evaluation.
Principles point to practices. Practices operationalize principles. The guidance in the GUIDE framework is guidance toward practices. This is a critical test for principles, that they can point to practices that, when implemented, lead to desired results.
Evaluating concrete practices is how the overarching principles are evaluated. This is the critical role evaluation plays in linking principles and practices. The GUIDE framework provides a template for articulating high-quality, evaluable principles.
Defining a field of knowledge involves decisions about both what to include and what to exclude (boundary issues in systems approaches). Principles can inform that boundary-setting process.
Summary and conclusion
This article has offered a framework for evaluating agroecological principles, practices, and claims. Advocates of agroecology offer principles for sustainable, productive, and efficient agricultural systems. The evaluation question that immediately and inevitably arises in the face of such claims is: Does agroecology work? But there are many approaches to and models of evaluation. The challenge is to match the evaluation framework to the nature of agroecology. Because agroecology is based on principles, this article has argued that the appropriate evaluation approach is principles-focused evaluation. To support that argument, the article has (1) elucidated principles-focused evaluation and demonstrated its relevance to evaluating agroecological practices and claims, (2) offered a principles-focused evaluation framework for monitoring agroecological transition and intensification within a particular context, and (3) concluded with lessons about principles-focused evaluation from agroecology.
Competing interests
The author has no competing interests to declare.
References
How to cite this article: Patton, MQ. 2021. Principles-focused evaluation for agroecology. Elementa: Science of the Anthropocene 9(1). DOI: https://doi.org/10.1525/elementa.2021.00052
Domain Editor-in-Chief: Alastair Iles, University of California, Berkeley, CA, USA
Associate Editor: V. Ernesto Méndez, University of Vermont, Burlington, VT, USA
Knowledge Domain: Sustainability Transitions
Part of an Elementa Special Feature: Principles-based Approaches in Agroecology
University of California Press