Using Evidence to Frame Problems of Practice

Improvement initiatives crafted based on well-understood problems of practice often stand the greatest chance of leading to sustainable educational improvements. Framing problems of practice using multiple modes of evidence is advisable to fully understand the system of root causes of the problem and its stakeholders. In this study, we used the document analysis method to investigate the types of evidence (e

Decades of reform efforts have yet to produce long-term, sustainable solutions to the issues plaguing the U.S. educational system.Too often, in efforts to create quick change, educational leaders apply interventions that worked in other environments without considering the differences between the two environments and how those differences could negatively impact the intervention's success (Bryk et al., 2016).Such interventions perpetuate the cycle prevalent in education in which initiatives are adopted but quickly abandoned because they are proven ineffective or are not provided enough time to show the desired effect (Rohanna, 2017).Some have argued that most large-scale initiatives fail due to limited understandings of the very problems the initiatives were undertaken to solve or improve (Bryk et al., 2016).The understanding of the problem could be limited by a lack of data or a lack of understanding about the larger systems in which the problem is situated.Take, for instance, the pervasive issue of mathematics achievement that exists at all levels of education in the U.S.-from students' low Algebra passage rates in K-12 to the difficulty community colleges and higher education institutions have moving students beyond developmental mathematics.A K-12 leader would likely first detect the issue from low scores on standardized tests to which the leader might respond by offering tutoring sessions to students.But if the leader investigated further, she would likely find that the problem stemmed from a wide set of factors including teachers' lack of training to effectively teach mathematics, students' language barriers, and curricula focused on rote memorization of facts rather than problem-solving skills.Put simply, low test scores were symptoms of the problem; there was likely a system of root causes.And it is likely the system of root causes that should guide potential improvements to create meaningful and sustainable solutions.

CARNEGIE PROJECT ON THE EDUCATION DOCTORATE
Helping to lead the charge of defining and improving upon educational problems are institutions within the Carnegie Project on the Education Doctorate (CPED) consortium.Currently, there are over 100 CPED member institutions (CPED, n.d.-a) who, as "a network of 'peer-to-peer' faculty members and leaders" (Perry & Imig, 2016, para. 9), strive to better prepare EdD programs to guide and produce leaders in education.According to Perry (2013), CPED is the first "action-oriented effort" aimed at reframing, redefining, and restructuring the EdD (Perry, 2013, p. 115).
Through intellectual exercises and discussions at meetings, CPED members collaborated to define the EdD as a degree that "prepares educators for the application of appropriate and specific practices, the generation of new knowledge, and for the stewardship of the profession" (CPED,para. 6).CPED member institutions created a framework (CPED, n.d.-c) that EdD programs could use to design their programs in ways that honored their particular student constituencies and local educational communities.CPED also created a set of design concepts that member institutions could use in their EdD programs to further the development of scholar practitioners.The design concepts include, among others, a signature pedagogy that is the hallmark pedagogical process particular to that program, similar to "rounds" in medical education or the "Socratic method" in legal education.For the EdD, graduates use "inquiry as practice" (CPED, n.d.-c, para. 1) to identify and improve upon educational issues in their organizations, and the dissertation in practice.

Dissertation in Practice
The cornerstone of CPED-influenced EdDs is the dissertation in practice (DiP), which is similar to a traditional dissertation in some ways but markedly different in others.Both traditional dissertations and dissertations in practice involve investigations of educational phenomena with extensive reviews of literature; however, the main difference between the two is that the dissertation in practice centers on an identified localized problem of practice rather than gaps in the research literature (Storey et al., 2015).And it is the investigation of improvements related to the problem of practice that often defines a DiP.Completed DiPs serve as evidence that EdD graduates possess the abilities, skills, and knowledge to "analyze problems of practice and use multiple frames to develop meaningful solutions" (CPED,para. 6) while in EdD programs, but also in their current and future practice as educational leaders.

Problems of Practice
CPED defines a problem of practice as "a persistent, contextualized, and specific issue embedded in the work of a professional practitioner the addressing of which has the potential to result in improved understanding, experience, and outcomes" (CPED,para. 1).Decomposition of the definition requires consideration of the three phrases within it.A persistent problem is one that happens more than once and likely for more than a short time period, with the timeframe relative based on the nature of the problem.A problem is contextualized when it is situated within a particular setting with particular characteristics.Finally, a problem is considered a "specific issue embedded in the work of a professional practitioner, the addressing of which has the potential to result in improved understanding, experience, and outcomes" (CPED, n.d.-c, para. 1) when it is one that is clearly localized to a practitioner's work and in need of improvement.Mintrop (2016) furthered the CPED PoP definition by stressing the urgency of the solution to the localized issue.
By virtue of the individuals and the types of issues one sees in education, PoPs are often ill-structured or ill-defined, meaning that they are complex and lack clear-cut solutions (Archbald, 2014;Copland, 2000;Jonassen, 2000;Timperley & Robinson, 1998).Educational problems do not often present themselves ready for solution; rather, full understanding of a problem is generated through the collective examination of multiple problematic situations (Schön, 1983).With the assistance of Mintrop (2016), we define what it means to identify, define, and frame a problem of practice below; these steps are necessary to gain a full understanding of a PoP and its system of root causes.Note that while the words identify, define, and frame may have general meanings elsewhere, the meanings related to problems of practice are quite specific to those offered below.
The first step toward a full understanding of a PoP is to identify or select an issue that is an urgent need of an organization (Bryk et al., 2016;Mintrop, 2016).One might think of this as conception of the idea that completion of the next two steps in the process will further explicate.The process at which one comes to identification is likely different for various individuals and organizations but is often guided by organizational strategic plans or goals situated within the individual's particular sphere of influence.
Once an issue has been identified as potential problem of practice, it requires further definition.To define the problem, individuals tap prior knowledge and experience to consider the symptoms or instances that initially made the issue appear problematic (Mintrop, 2016).This has been characterized as the gap between the current state and the ideal (Archbald, 2014;Mintrop, 2016).Gaps represent comparisons such as discrepancies between one's organization and another (e.g., institution A has lower scores than institution B), between an organization over time (e.g., institution A's scores have decreased over the last 5 years), or between an organization and its targets (institution A's scores did not meet the target) (Archbald, 2014).It is the justification of these gaps as problems, contends Archbald (2014), that defines the problem of practice and situates it as an issue for study and potential improvement.
Framing the PoP demands consideration of the broader construction of the issue, including the myriad factors that likely impact it and stakeholders who influence it.Closely related to problem definition, framing of the problem goes one step further in considering the potential causes of the issue as well as individual assumptions and lenses that may influence the development of the frame (Schön & Rein, 1995).
Investigators have empirically examined the nature and content of PoPs (see e.g., Gillham et al.,2019;Ma et al., 2018;Storey et al., 2015), but questions still remain about how to thoroughly frame a PoP.Naturally, scholars often turn to the literature for prior studies and theories that may inform the PoP (Archbald, 2014;Belzer & Ryan, 2013;Mintrop, 2016).But when considering the contextual nature of problems, the literature may only serve to help situate the identified issue within the broader context and may not be enough to provide information on the problem in the local context.Literature can help clarify the ideal state, but the most convincing problem definitions also include practitioner and stakeholder input through multiple modes of evidence, including secondary, primary, and/or anecdotal data in addition to relevant literature (Archbald, 2014).
Mintrop (2016) also advocated framing problems using multiple modes of evidence, but through an iterative design-based process.
The process begins with an initial framing of the PoP and identification of potential improvement initiatives via broad problem statements.The broad statements are further refined as evidence is gathered and assimilated, with a portion of evidence derived as the result of an empirical needs assessment of the organization (Mintrop, 2016).
The improvement science (IS) methodology advances a systems analysis to build a robust understanding of problems and avoid "solutionitis" (Bryk et al, 2016, p. 197) where solutions are enacted before the problem is fully understood.In contrast to experimental science that seeks to minimize variation, IS embraces the idea that "variation in implementation and setting are important sources of information" (Lewis, 2015, p. 55) in solving problems.IS advances a causal system analysis to investigate root causes of problems that might otherwise be obscured by their symptoms.From the causal system analysis develops a solution system that addresses the particular influence of each identified system on the problem and works to improve upon it.The solution is then implemented in iterative plan-do-study-act (PDSA) cycles whereby the intervention is planned and enacted, results are studied, changes are made, and the implementation cycle begins again with the revised plan.Iterative cycles of inquiry provide a mechanism to "[learn] fast to implement well" (Bryk et al., 2016, p. 7) in the aim of continuous improvement.

NEED FOR EVIDENCE IN POST-TRUTH ERA
The need for evidence to substantiate PoPs is arguably more relevant in current post-truth era where "facts are less influential in shaping public opinion than appeals to emotion and personal belief" (Oxford Languages, 2018, para.1).Because they invariably do not uncover the true nature of PoPs, issues framed by personal agenda, emotion, or even anecdotal evidence likely prompt misguided reforms that stand little to no chance of creating long-term sustainable change in education.Evidence is needed to substantiate PoPs in education, and that evidence needs to be of sufficient rigor and quality to withstand the "contempt for facts" (Arendt, 1957, p. 350) that exists in the current post-truth society.

STUDY PURPOSE
Currently, as the dissertation in practice continues to develop in redesigned EdD programs, it is unknown to what extent evidence is used to frame problems of practice within dissertations in practice.Thus, the purpose of this study was to explore the use of evidence in framing PoPs within published dissertations in practice.We sought to understand the nature of PoPs, whether they met the CPED definition of a PoP, what types of evidence and methods were used to frame the identified PoPs, and to what degree particular methods were used to frame PoPs.Note that the study was limited to the framing of problems of practice rather resulting improvements or proposed solutions.

METHOD
We used the document analysis method (Bowen, 2009) to gain a more thorough understanding of the evidence used to frame PoPs as have other authors that studied PoPs (see e.g., Gillham, Williams, Rife, & Parker, 2019;Ma, et al., 2018).According to Bowen (2009), "Document analysis is a systematic procedure for reviewing or evaluating documents -both printed and electronic" (p.27) for which documents are reviewed to extract data.Data are then analyzed and interpreted to make meaning and gain understanding regarding the research question(s) or purpose (Bowen, 2009).
The primary data source comprised dissertations in practice (N=53, from original N=55 with 2 deletions for broken links) that were publicly available through the CPED website (CPED, n.d-b.).Note that the dissertations were likely not a representative sample of dissertations in practice from CPED member institutions, so may not be generalizable to the larger population of dissertations in practice as a result.The dissertations linked from the CPED website comprise the only current centralized source of dissertations in practice known to the authors as of this writing.Limitations notwithstanding, data should still offer interesting results that can contribute to the discussion of how PoPs are and should be framed.
Following the document analysis method, the following were coded for each dissertation: (a) the topic of each PoP, (b) whether the PoP was stated or implied, (c) whether the PoP met the three components of the PoP definition, (d) the evidence provided to frame the PoP, (e) the methodology used to frame the PoP, and (f) the dissertation's institution.All three characteristics of the CPED definition were coded including whether the identified problem of practice was (a) "persistent", (b) "contextualized", and (c) "a specific issue embedded in the work of a professional practitioner" (CPED, n.d.-c, para.1).For the purposes of coding, we considered evidence of the persistent nature of a PoP as at least one reference or sentence regarding the long-lasting nature of the PoP, with "longlasting" being relative based on the nature of the PoP.Evidence of the contextualized nature of the PoP comprised at least one reference or sentence regarding the conditions within which the PoP was situated.Evidence that the work was embedded in the work of a professional practitioner comprised at least one statement that signified whether the topic was localized and practice-based.It was difficult to discern whether a PoP was embedded directly into the work of the author without directly knowing the author and their workplace, so whose work it was embedded within was disregarded.
As a calibration exercise, the researchers reviewed a set of three dissertations together, and codes were discussed until all authors were in agreement.The remaining dissertations were each randomly assigned to two researchers to be coded individually.Any discrepancies were flagged and discussed between each pair until consensus was reached.Finally, the entire research team reviewed the spreadsheet together and tabulated results to gain an understanding of initial results followed by process of categorization and thematic analysis as suggested by Bowen (2009).Interrater agreement between the multiple coders strengthened the credibility of results as well as the fact that coding pairs were randomly assigned and varied for each dissertation in practice.
Much as particular individual's assumptions and lenses can impact the framing of a particular problem of practice, so too could they have impacted our data collection and analysis.The authors are faculty and students in a CPED-influenced EdD program and acknowledge potential biases that could have influenced the study results.

RESULTS
The coded data were analyzed descriptively.Frequencies were run to analyze how each PoP was characterized-specifically, was it explicitly characterized as a problem of practice, was it implied to be a problem of practice without the explicit language, or was it not stated or implied to be a problem of practice?Table 1 presents the results.The majority of dissertations (85%, which comprised 45 of N=53) involved a problem of practice that was stated or implied.But just 27% of the dissertations (14 of N=53) explicitly characterized PoPs as such.Fifty-eight percent (31 of N=53) offered evidence that implied that the identified issues were problems of practice but did not state it and 15% (8 of N=53) were not considered to be PoPs and addressed gaps in the literature or other.
Table 1 also presents the results of descriptive analysis of coding regarding evidence that the PoP met all components of the CPED definition-i.e., that the problem of practice was (a) "persistent", (b) "contextualized", and (c) "a specific issue embedded in the work of a professional practitioner" (CPED, n.d.-c, para.1).The majority of PoPs (43, which comprised 81% of N=53) did not offer evidence to show that all components of the CPED definition of a PoP were met.Interestingly, of the dissertations that addressed a PoP (N=45), most dissertations included evidence of at least one component of the definition, with 78% (35 of N=45) providing evidence of the persistence of the PoP, 87% (39 of N=45) providing evidence that the PoP was contextualized, and 84% (38 of N=45) demonstrating that the issue was specific and within the work of a professional practitioner.However, only 19% of all dissertations (10 of N=53) provided evidence of all three components of the definition.
We coded the evidence and methods used to frame PoPs within the sampled dissertations in practice; results are presented in Figure 1.PoPs overwhelmingly offered references to the literature as evidence of the problems of practice (41, which comprised 77% of N=53 dissertations).This particular result was perhaps not surprising given the emphasis of the literature on the theoretical, empirical, and practical bases on which problems may be framed.Fewer PoPs (23, which comprised 45% of N=53) offered practitioner expertise or anecdotal accounts as evidence of the PoPs.Data were presented in several cases -with more PoPs justified using secondary data (18, which comprised 34% of N=53) than primary data (17, which comprised 32% of N=53).
Particularly compelling, however, is the fact that of all the dissertations in practice-even the ones that incorporated data as evidence of the PoP-just two dissertations (4% of N=53) employed a systematic methodology to frame the PoP.The two dissertations used empirical needs assessments to collect data as has been suggested by Mintrop (2016), however they lacked the systems analysis advanced in improvement science (Bryk et al., 2016).
We examined the types of evidence offered for each of the PoPs by whether each met the CPED definition of a PoP; Figure 2 presents the results.The general pattern of results generally mirrored the results presented in Figure 1.However, the percentage of dissertations using primary (4, which comprised 40% of n=10) or secondary data (4, which comprised 40% of n=10) as evidence was a bit higher in the dissertations where the definition of the PoP was justified over those that did not (primary data -13, which comprised 30% of n=43; secondary data -13, which comprised 30% of n=43).Note.Problems of practice were often framed using multiple sources of evidence, so percentages of the sources of evidence for each category will add to greater than 100%.

DISCUSSION
Results indicated that the sampled dissertations in practice primarily relied on literature to provide evidence to frame PoPs and less frequently cited primary or secondary data as evidence.Due to the varied nature of PoPs and the contexts in which they are situated, the primary reliance on literature to substantiate PoPs is problematic.Whether theoretical or empirical in nature, the literature offers references that could provide a rationale or evidence of PoPs in a general sense, which is needed, but typically not to the particular context in which one selects a PoP, which is also needed.And although secondary data could be considered, primary data from the local context would seem to offer the most representative evidence of the PoP-a certain fit having been designed and collected by the investigator for the primary purpose of framing the PoP.With multifactor problems of practice, it may be best to consult multiple modes of evidence (Archbald, 2014) that include not only anecdotal data and literature, but also primary data that are directly relevant to the identified problem of practice.

Evidence for Claims
As scholars, we are prepared to offer evidence as justification for the claims that we make in our writing.Evidence can many forms, but most often take the form of literature citations, theories, expert accounts, and/or empirical data that is either primary or secondary in nature.It is the presentation of the evidence by which the writer justifies his or her claims and the evaluation of such evidence by which the reader determines its validity (Booth et al., 2016).The credibility of the evidence can depend on various factors, with the circumstances under which the evidence was collected as key, and primary sources are generally considered more credible than secondary.And generally, if found to be valid, more evidence is better than less, particularly when it derives from a variety of sources.
The fact that sources of evidence beyond literature are infrequently used to frame PoPs and systematic methodologies are used even more infrequently is concerning as it potentially limits the credibility of the PoPs.Improvements crafted for problems that are not fully understood stand the chance of ineffectiveness simply because they were developed to solve a problem that does not exist in the form that was originally conceived (Bryk et al., 2016).

Implications and Recommendations for Practice
Systematic methodologies are needed to frame PoPs to generate clear understandings of the root causes of problems to have hope of crafting effective and sustainable improvements.The exploration of a problem of practice would require collection of multiple forms of evidence that would be specific to that particular problem of practice.An inductive methodology that synthesizes the evidence would generate an understanding of the extent, scope, and underlying root causes of the problem of practice.Discussion of the specific methodologies that could be used to complete this goal is beyond the scope of this paper, but further exploration is needed to guide investigators in their methodological efforts to solve important educational problems.
Within the systematic methodology of framing PoPs, clear and agreed-upon definitions of phrases like "local context", "embedded in the work of the practitioner", "persistent problem", etc., are necessary to guide scholar-practitioners and their faculty mentors in the dissertation process.Although the nature of PoPs can vary based on topics and institutional program objectives, a common understanding facilitated clear definitions could provide the standard needed to frame PoPs.And guidance regarding the evidence needed to substantiate these definitions could assist students and their faculty mentors in producing dissertations in practice that address PoPs that represent complex problems in need of improvement.As evidence takes a "backseat" to emotion and personal beliefs in the current post-truth era, methodologically sound systems to frame PoPs are arguably needed now more than ever to drive contextualized educational improvements that can be impactful, long-lasting, and sustainable.
Presented Evidence of All Components of the PoP DefinitionNote.The definition of a problem of practice (PoP) as "a persistent, contextualized, and specific issue embedded in the work of a professional practitioner" (CPED, n.d.-a) was used to classify dissertations into those that met the definition and those that did not.Percentages represent the number divided by the total number of dissertations (N=53) rounded to the nearest whole number; the rounded numbers may not add to 100% as a result.

Figure 1 .
Figure 1.Proportions of Evidence Sources for Problem of Practice (N=53)

Figure 2 .
Figure 2. Proportions of Evidence Sources by the Justification of the Definition of Problems of Practice (N=53)