#language evaluation

Context

It  is vital  to  situate contemporary  Australian  Indigenous language  programs  within the  context  of the  recent  history of  invasion,  colonisation and  genocide.  In the  past  200 years,  successive  colonial and  Australian  governments have  forcibly   removed Aboriginal  people  off their  lands,  stolen generations  of  children from  their  families, and  undermined  traditional lifestyles  and  leadership through  paternalistic  interventions (Stockley,  2010,  p.91;  Simpson  et  al., 2009,  p.23).  

Violent  oppression and  vilification  of  culture  has severed  traditional  ties between  land  and language  and  halted the  intergenerational  transmission of  the  majority of  Indigenous  languages (Stockley,  2010,  p.91). Indigenous  language  revitalisation  projects are  thus  unique amongst  second  language acquisition  programs  in that  participants  are reclaiming  and  reinforcing their  ethnic  identity in  the  face of  colonial  disempowerment  and trauma  (Walsh,  2010, p.23).  

Types  of language  revitalisation  programs include:  school-based  programs, after-school  programs  for children,  adult  language programs,  documentation  development and  home-based  programs (Disbray  et  al., 2018, p.3).  In  examining the  relevant  literature, school-based  programs  appear to  have  commanded the   most  attention. Advocates  for  implementation  of school  language  programs such  as  Anmatyerr educator  April  Campbell and  Warlpiri  educator  Valerie  Patterson believe  such  programs help  keep  traditional languages  strong  (Disbray et  al.,  2018, p.4).  

Depending  on the  local  context, the  program  goals may  focus  on:  revival, for  reawakening  a language  no  longer spoken  by  any community  members;  revitalisation,  for learners  who  have little  to  no passive  knowledge  in a  community  with some  fluent  speakers; or  maintenance,  for learners  who  use the  Indigenous  language as  a  main language,  or  are situated  in  a community  in  which it  is  a main  language  for many  people  (Disbray  et al.,  2018,  p.7;  Northern  Territory Department  of  Education, 2017, p.6).  As  schools are  potentially  an instrument  of  assimilation into  the  dominant culture,  the  implementation of  any  Indigenous language  and  culture program  into  mainstream curriculum  must  remain within  Indigenous  control in  order  to  prevent  further harm  (Walsh,  2010, p.24).  

Issues  in Indigenous  Program  Evaluation

In  Indigenous contexts,  participatory  and collaborative  evaluation  approaches are  commonly  employed, involving  evaluators,  community members  and  stakeholders who  dialogue  together to  produce  knowledge for  program  improvement (Katz,  Newton,  Shona &  Raven,  2016, pp.8-9).  However,  partnerships between  outsiders  and Indigenous  people  are  often  characterised by  an  imbalance of  power  in favour  of  the outsiders,  with  Indigenous participants  taking  on a  consultative  role at  best  (Kelaher,  Luke, &  Ferdinand,  2018, p.ix).  The  lack of  autonomy  afforded to  Indigenous  participants is  often  carried through  to  the publication  of  results and  storage  of data,  in  which the  outsiders  retain ownership  of  data and  evaluation  results are  not  translated into  positive  action in  the  community (Kelaher  et al.,  2018,  p.27; Taylor,  2003,  as cited  in  Katz et  al.,  2016, p.18).  

Furthermore,  operating  within conventional  positivist  frameworks, researchers  have  historically collected  and  classified knowledge  about  Indigenous people  according  to Western  ideas  (Katz et  al.,  2016, p.18).  These  non-Indigenous  researchers and  evaluators  have often  mischaracterised,  omitted or  maligned  Aboriginal people  and  their cultures,  or  used gathered  information  inappropriately  (Board  of  Studies NSW,  2003, p.6).  This  serves to  further  undermine Indigenous  community  autonomy,   exacerbate  tensions  and inflict  harm  on individuals  (Kelaher  et al.,  2018,  p.28).  When a  conventional  evaluation approach  is  adhered to,  following  capitalistic metrics  of  efficiency, performativity  and  productivity, neocolonialist  patterns  are  perpetuated  to the  detriment  of the  community  (Jordan,  Stocek, &  Mark,  2013, p.29).  The  history of  harm  caused to  Indigenous  communities by  Western  research has  resulted  in many  Indigenous  peoples being  hostile  towards or  suspicious  of any  research  attempts (Tuhiwai-Smith,  1999,  as cited  in  Katz et  al.,  2016, p.19).

Additionally, conflicts  may  arise from  the  integration of  Indigenous  and non-Indigenous  approaches  to evaluation,  requiring  ongoing negotiation.  For  example, in  the  New  South  Wales OCHRE  program  evaluation, tensions  were  observed due  to:  different beliefs  about  the rigour  of  certain methodologies,  competing  claims regarding  cultural  authority representatives,  different  views on  what  constitutes success  and  how this  is  measured, and  different  beliefs about  the  time commitments  required  of Aboriginal  participants  (Kelaher  et al.,  2018,  p.34).

Finally,  if  the program  framework  and development  has  not been  oriented  to Indigenous  community  needs from  the  outset, program  evaluation  outcomes are  significantly  less able  to  align with  community  values (Kelaher  et  al., 2018,  pp.39-41).  

Indigenous  Epistemologies

In  the view  of   Katz  and colleagues,  evaluation is  a  kind of  pragmatic  social science  research,  and can  be  framed through  either  a positivist  or  an  interpretive  lens  (2016,  p.3). They  argue  that as  the  focus of  such  enquiry is  the  meaning  and purpose  of   society and  social  actions, there  can  arguably be  no  objective truth  to  uncover, but  rather  a multitude  of  interpretations  according to  the  ideology of  the  stakeholders (Weber,  1949,  as sited  in  Katz et  al.,  2016,  p.3).  Furthermore, critical  interpretive  approaches use  “research  as  a  mechanism  for  critiquing the  social  structures that  underpin  inequality  and  discrimination, with  the  purpose of  challenging  the  current power  relationships  within  society and  promoting  social  justice”  (Katz et  al.,  2016,  p.3).

Decolonising  evaluation is  one  such critical  method,  in which  evaluation  is conducted  with  or  by  Indigenous people  themselves,  in  an  effort to  combat  the ways  that  Indigenous people  have  historically been  silenced  and marginalised  (Katz  et al.,  2016,  p.20). Commonalities  across  Indigenous methodologies  include:  privileging and  legitimising  holistic Indigenous  knowledge  systems; attending  to  the relational  positions  of participants and researchers;  collective  and reciprocal  benefits  to  the  community; and  legitimising  Indigenous ways  of  sharing knowledge,  including  story telling  (Kovach,  2015, as  cited  in Katz  et  al., 2016,  pp.20-21).  As expressed  by  one Native  Alaskan  participant in  an  Indigenous evaluation  capacity  building program,  “sharing  narratives  rather  than numbers  is  the  Indigenous way  to  share  successes”  (Anderson et  al.,  2012, p.580).

Qualitative  data is  often  most conducive  for  these purposes,  as  such data  lends  itself to  emphasising  holistic perspectives  and  narrative interpretations  of  results (Jordan et  al.,  2013,  p.22).  For example,  Blodgett  and colleagues  foregrounded  Indigenous perspectives  by  presenting data  through  portrait vignettes  (2011,  as cited  in  Katz et  al.,  2016, p.29).  Within  this kind  of  data, context  can  be made  explicit,  including: the  characteristics  of the  evaluator  or evaluation  team,  the community  context,  institutional influences  and  program specifics  (Katz  et al.,  2016,  p.14).  

 One  method of   gathering  information in  an  Indigenous program  evaluation  context is  the  culturally specific  practice of  yarning  (Kelaher et  al.,  2018, p.57).  Yarning  is a  valuable  practice within  research  and evaluation,  as  it  “creates a  space  through which  Aboriginal  and  Torres Strait  Islander  peoples  can  voice and  infuse  traditional cultural  knowledge  in  the  creation  and  completion of  research”  (Leeson,  Catrin,  &  Rynne, 2016, as  cited  in Katz  et  al., 2016,  p.32).

Stakeholder Engagement

Indigenous program  development  and evaluation  from  inception must  be  aligned with  community  priorities and  aspirations,  with Indigenous  communities  afforded equal  power,  or leadership  roles  (Kelaher  et al.,  2018,  p.34). As  meaningful  language revitalisation  requires  sustained effort  (Stockley,  2010, p.99),  long-term  partnerships should  be  developed, with  the  intention of  building  evaluation capacity  within  communities (Kelaher et  al.,  2018, p.x).  Stakeholder  engagement should  take  place at  every  stage of  an  evaluation, through  evaluation  planning, design,  data  collection, analysis,  decision-making  and implementation  (Bryson  et al.,  2011,  p.4).  Additionally,  Aboriginal  people should  have  data sovereignty  over  the information  gathered  during an  evaluation  (Kelaher et  al.,  2018, p.43).  

The  evaluation must  be  sensitive to  the  unique local  context  of the  language  program (Katz  et  al., 2016,  p.31).  One  Indigenous  Canadian after  school  program successfully  employed  “participatory  evaluation” methods,  which  may be  particularly  suited to  small  scale language  revitalisation  program evaluations  (Jordan  et  al.,  2013, pp.19-20).  Patton  (2011)  further affirms  that  action research  can  form the  basis  of evaluation  due  to the  cyclical  and localised  nature  of working  within  and improving  a  program (as  cited  in  Gruba, Cárdenas-Claros,  Suvorov,  &  Rick,  p.24). However, where  the  evaluation is  conducted  by a  third  party, the  evaluator  should take  the  role of  facilitator,  collaborator or  participant  in  the  process, rather  than  assigning themselves  a  privileged  position of  “expert”  (Jordan  et  al.,  2013, p.25).  

The  preference expressed  by  various scholars  is  that the  evaluator  should themselves  be  Indigenous, as  “Aboriginal  and  Torres Strait  Islander  researchers  are  more likely  to  convey information  in  a  manner  that is  both  culturally acceptable  and  specific to  those  participating in  the  research and  benefiting  end  users of  the  information” (Kite &  Davey,  2015, as  cited  in Katz  et  al., 2016,  p.30).  

This  is the  stance  also of  the  Maori evaluation  group  Whariki,  who consider  “by  Maori, for  Maori”  evaluations to  be  integral to  their  self-determination  (Anderson et  al.,  2012, p.574).  An  alternative perspective  is  that ethnic  diversity  of evaluators  may  be positive,  as  long as  Indigenous  knowledge and  skills  continue to  form  the basis of  evaluations,  as Indigenous  communities  may benefit  from  strong allies  supporting  their  decolonisation efforts  (Burnette  &  Billiot, 2015, as  cited  in Katz  et  al., 2016,  pp.30-31).

Stakeholders  may be  broadly  defined as  “individuals,  groups,  or  organizations that  can  affect or  are  affected by  an  evaluation process  and/or  its findings”  (Bryson et  al.,  2011, p.1).  In  identifying stakeholders  to  involve in  the  evaluation process,  advice  may be  sought  from Indigenous  advisory  groups or  Aboriginal  cultural advisors  or  mentors (Katz  et  al., 2016,  p.29).  For  Indigenous language  programs  in schools,  key  stakeholders to  engage  may include:  Aboriginal Elders,  local  custodians of  Aboriginal  language  and  culture,  local  Aboriginal community  leaders  and  members,  representatives  of  local Aboriginal  community  organisations, school  principal,  school council,  program  coordinators, assistant  teachers,  Aboriginal language  teachers,  classroom teachers,   and  students’ families  (NSW  Department of  Education,  2018, p.10;  Northern  Territory  Department  of  Education, 2017, pp.10-13;  Simpson  et al.,  2009,  p.13). Engagement  and  trust can  be  built between  stakeholders  through “social  yarning”  (Bessarab  &  Ng’andu,  2010, as  cited  in Katz  et  al., 2016,  p.32).

In  order to  make  decisions about  program  improvement, stakeholders  should  be  engaged to  build  a shared  understanding  of the  program  through the  combination  of their  diverse  needs, expertise  and  perspectives  (Kelaher  et al.,  2018,  p.1).  Consensus building  may  be facilitated  through  the technique  of  a “talking  circle”,  whereby participants  sit  in  a  circle and  sequentially  take turns  to  discuss a  topic  without being  interrupted,  continuing until  all  participants are  satisfied  with their  contribution  (Kholghia  et al.,  2018, p.82).  Furthermore, a  relaxed  and culturally  appropriate  way of  gathering  information is  through  evaluation “topic  yarning”,  while brainstorming  can  take place  through  “collaborative  yarning” (Bessarab  &  Ng’andu,  2010, as  cited  in Katz  et  al., 2016,  p.32).  The evaluation  agenda  evolves over  time  through such  collaboration  (Jordan, Stocek,  &  Mark, 2013,  p.25).  

Conclusion

In  summary, evaluation  of  Indigenous language  revitalisation  programs in  schools  must take  into  account the  historical  and local  context  in order  to  effectively engage  with  stakeholders. Through  privileging  Indigenous epistemologies,  the  evaluation process  can  be an  act  of decolonisation,  benefiting  both the  program  and the  community.

References

Anderson, C., Chase, M., Johnson III, J., Mekiana, D., McIntyre, D., Ruerup, A., & Kerr, S. (2012). It Is Only New Because It Has Been Missing For So Long: Indigenous Evaluation Capacity Building. American Journal of Evaluation 33(4), 566-582.

Board of Studies NSW. (2003). Advice on Programming and Assessment for Stages 4 and 5. Sydney: New South Wales Government.

Bryson, J. M., Patton, M. Q., & Bowman, R. A. (2011). Working with evaluation stakeholders: A rationale, step-wise approach and toolkit. Evaluation and Program Planning 34, 1-12.

Disbray, S., Barker, C., Raghunathan, A., & and Baisden, F. (2018). Global Lessons: Indigenous languages and multilingualism in school programs. Newcastle: First Languages Australia.

Gruba, P., Cárdenas-Claros, M. S., Suvorov, R., & Rick, K. (2016). Essentials of Blended Language Program Evaluation: Towards an Argument-Based Approach. In P. Gruba, M. S. Cárdenas-Claros, R. Suvorov, & K. Rick, Blended Language Program Evaluation (pp. 20-45). London: Palgrave Macmillian.

Jordan, S., Stocek, C., & Mark, R. (2013). Doing participatory evaluation in Indigenous contexts - methodological issues and questions. Action Learning, Action Research Journal 19(1), 12-35.

Katz, I., Newton, B. J., Shona, B., & Raven, M. (2016). Evaluation theories and approaches; relevance for Aboriginal contexts. Sydney: Social Policy Research Centre, UNSW Australia.

Kelaher, M., Luke, J., & Ferdinand, A. (2018). An Evaluation Framework to Improve Aboriginal and Torres Strait Island Health. Melbourne: Centre for Health Policy - Melbourne School of Population and Global Health.

Kholghia, M. K., Bartletta, G., Phillips, M., Salsberga, J., McComberb, A. M., & Macaulaya, A. C. (2018). Evaluating an Indigenous health curriculum for diabetes prevention: engaging the community through talking circles and knowledge translation of results. Family Practice,, 80-87.

Lowe, K., & Howard, P. (2010). So you want to work with the community? Principles and strategies for school leaders affecting the establishment of Aboriginal language programs. In J. Hobson, K. Lowe, S. Poetsch, & M. Walsh, Re-awakening languages: theory and practice in the revitalisation of Australia’s Indigenous languages (pp. 194-209). Sydney: Sydney University Press.

Norris, J. M. (2016). Language Program Evaluation. The Modern Languages Journal 100, 169-189.

Northern Territory Department of Education. (2017). Guidelines for the implementation of Indigenous Languages and Cultures Programs in schools. Darwin: Northern Territory Government.

NSW Department of Education. (2018). Aboriginal Language and Culture Nest 2018 Guidelines. Sydney: NSW Government.

Simpson, J., Caffery, J., & McConvell, P. (2009). Gaps in Australia’s Indigenous Language Policy: Dismantling bilingual education in the Northern Territory. Canberra: Australian Institute of Aboriginal and Torres Strait Islander Studies.

Standing Committee on Aboriginal and Torres Strait Islander Affairs. (2012). Our Land Our Languages: Language Learning in Indigenous Communities. Canberra: House of Representatives.

Stockley, T. (2010). Awakening or awareness: are we being honest about the retrieval and revival of Australia’s Aboriginal languages? In J. Hobson, K. Lowe, S. Poetsch, & M. Walsh, Re-awakening languages: theory and practice in the revitalisation of Australia’s Indigenous languages (pp. 88-100). Sydney: Sydney University Press .