Intrarater Reliability of Accreditation Assessment for Early Childhood Education
Abstract
This study measured intrarater reliability between assessors of Early Childhood Education (PAUD) unit accreditation in the East Nusa Tenggara Province (NTT), Indonesia. In the 2022 PAUD unit accreditation assessment, 50 institutions across 15 districts in the NTT Province were sampled. The intrarater reliability testing technique used was the intraclass correlation coefficient (ICC). The results showed good agreement between Assessor A and Assessor B, with the ICC coefficient in the 2022 PAUD unit accreditation assessment in the NTT Province being relatively high with an estimated ICC of 0.644. However, the ICC coefficient between Assessor A and Validator, as well as between Assessor B and Validator, was in the moderate category with a moderate level of agreement. In addition, the study found a small effect size, indicating that subjective factors such as assessors' backgrounds and accreditation experience had a very small correlation with accreditation scores. Therefore, efforts should be made to improve agreement between Assessor A and Validator as well as between Assessor B and Validator to improve intrarater reliability in the PAUD unit accreditation assessment in the NTT Province.
Penelitian ini mengukur reliabilitas intrarater antara asesor akreditasi satuan Pendidikan Anak Usia Dini (PAUD) di Provinsi Nusa Tenggara Timur (NTT). Dalam penilaian akreditasi satuan PAUD tahun 2022, 50 lembaga yang tersebar di 15 kabupaten di Provinsi NTT dijadikan sebagai sampel. Teknik pengujian reliabilitas intrarater yang digunakan adalah teknik intraclass correlation coefficient (ICC). Hasil penelitian menunjukkan tingkat kesepahaman yang baik antara Asesor A dan Asesor B, dengan koefisien ICC dalam penilaian akreditasi satuan PAUD tahun 2022 di Provinsi Nusa Tenggara Timur tergolong tinggi dengan besaran estimasi ICC 0,644. Namun, koefisien ICC antara Asesor A dengan Validator maupun antara Asesor B dengan Validator berada pada kategori sedang dengan tingkat kesepakatan yang moderat. Selain itu, dalam penelitian ini diperoleh nilai effect size yang kecil. Hal ini menunjukkan bahwa faktor subyektif asesor seperti lingkungan asal maupun pengalaman melakukan akreditasi berkorelasi sangat kecil terhadap skor akreditasi. Oleh karena itu, perlu dilakukan upaya-upaya untuk meningkatkan kesepakatan antara Asesor A dan Validator serta antara Asesor B dan Validator guna meningkatkan reliabilitas intrarater dalam penilaian akreditasi satuan PAUD di Provinsi Nusa Tenggara Timur.
Kata Kunci: Reliabilitas intrarater; Akreditasi; Pendidikan Anak Usia Dini
Full Text:
PDFReferences
Barcikowski, R. S., & Ketrow, S. M. (2014). Assessing the inter-rater reliability of the Wilson Reading System® (WRS) coding system. Journal of Psychoeducational Assessment, 32(1), 78–86.
Chou, Y. C., Lin, H. W., & Chang, Y. T. (2015). Evaluating the inter-rater reliability of an oral presentation assessment instrument using generalizability theory. BMC Medical Education, 15(1), 56.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Routledge.
Eviati, E., & Indrawati, D. (2018). Evaluasi Hasil Belajar Anak Usia Dini dengan Menggunakan Pendekatan CIPP pada Kelompok B di TK Negeri Tegalrejo Yogyakarta. Jurnal Pendidikan Anak Usia Dini, 7(2), 105–118.
Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). Wiley.
Fleiss, J. L. (1986). Reliability Measurement. In the Design and Analysis of Clinical Experiments. John Wiley & Sons.
Grimes, P. W., & Ford, J. K. (2014). Reliability testing: What do we know? Human Resource Management Review, 24(4), 271–282.
Harris, R. B., He, J., & Chen, Q. (2018). Noise and response bias in classroom observations: A randomized controlled trial. Journal of Educational and Behavioral Statistics, 43(5), 569–592.
Heilbronner, R. L., & Strosser, G. L. (2013). (2013). The effect of examiner bias on the forensic assessment of juvenile sex offenders. Journal of Forensic Psychology Practice, 13(5), 362–378.
Hosseini, S. S., Zandieh, M., & Afzali, A. (2020). Accreditation of higher education institutions and quality assurance: A review of the literature. Journal of Education and Practice, 11(19), 84–91.
Hsu, Y. C., Liang, J. C., & Tsai, C. C. (2019). Investigating the inter-rater reliability of the science teacher assessment framework in Taiwan. International Journal of Science Education, 41(4), 481–499.
Katsikas, S. L., Natsis, A. G., & Tsioumis, K. A. (2019). Accreditation of higher education institutions: A review of the literature. Journal of Educational and Social Research, 9(2), 11–18.
Malini, P. S., & Ramya, P. (2019). Factors influencing inter-rater reliability in accreditation assessment of educational institutions. Journal of Engineering Education Transformations, 32(4), 245–250.
Mansoer, R. M., & Haryanto, S. (2015). Sistem Akreditasi Pendidikan Indonesia. Prenada Media.
Mokkink, L. B., van der Vleuten, C. P. M., Bouter, L. M., Sollie, A. W., & Schellevis, F. G. (2010). Inter-rater agreement and reliability of the COSMIN (Consensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Medical Research Methodology, 10(1), 82.
Özçelik, N. S. (2021). Intrarater reliability in assessing the quality of scientific journals. Journal of Academic Librarianship, 47(4).
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. In Psychological bulletin (pp. 420–429).
Sibbald, B., Shen, J., McBride, A., & Cumming, C. (2013). (2013). Factors affecting agreement between physicians’ and nurses’ clinical decisions for newly admitted nursing home residents. Journal of Interprofessional Care, 27(1), 76–83.
Zakiyah, Y., Muslimin, I. A., & Siswoyo, S. (2019). Intrarater reliability of the accreditation assessment process for study programs in Indonesian higher education. Quality in Higher Education, 25(2), 155–169. https://doi.org/10.1080/13538322.2019.1587816
DOI: http://dx.doi.org/10.24014/kjiece.v6i1.22136
Refbacks
- There are currently no refbacks.
Copyright (c) 2023 KINDERGARTEN: Journal of Islamic Early Childhood Education
Diterbitkan oleh:
Program Studi Pendidikan Islam Anak Usia Dini Fakultas Tarbiyah dan Keguruan UIN Suska Riau
Jalan HR. Soebrantas KM 15,5, Simpang Baru, Tampan, Pekanbaru - Riau
email: jcie.piaud@uin-suska.ac.id
This work is licensed under a Creative Commons Attribution 4.0 International License.