Development of Rubric of Higher Order Thinking Skills Assessment on Mathematics Learning

Ega Gradini

doi:10.31764/jtam.v6i1.5342

Development of Rubric of Higher Order Thinking Skills Assessment on Mathematics Learning

Ega Gradini

Abstract

This research aims to develop a rubric of Higher-Order Thinking Skills Assessment on mathematics learning that is valid, reliable, and practical. The Plomp's developmental method, the Generic model for educational design, was deployed to develop the HOTS assessment rubric. The method consisted of 4 phases, namely; (1) preliminary investigation, (2) design, (3) development, and (4) implementation and evaluation. The rubric was designed and developed to measure students' higher-order thinking skills in mathematics learning. The rubric was developed by integrating HOTS characteristics in mathematics: critical thinking, problem-solving, mathematical understanding, mathematical modeling, proof and reasoning, mathematical representation, and mathematical communication. The HOTS assessment rubric's quality was examined using Plomp's product/prototype quality criteria: (1) validity, (2) reliability, and (3) practicality/usability. The rubric of the HOTS assessment was validated by two validators that are experts on HOTS in mathematics. The practicality/usability of the rubrics was examined by five mathematics teachers from different Junior High schools. The validity and reliability were measured using Gregory's Expert Agreement Index (EAI), while the rubric's practicality was analyzed using practicality product criteria. The research shows that the validity of the rubric of HOTS assessment on mathematics learning is 0.82 (valid), and the reliability is 0.79 (reliable). Meanwhile, the practicality of the rubric is 81.02 (Good). In conclusion, the rubric of the HOTS assessment on mathematics learning is valid, reliable, and practical to use to measure students' higher-order thinking skills in mathematics learning.

Keywords

Rubric; Problem-solving; Critical thinking; Higher-Order Thinking; Assessment; Bloom Taxonomy

Full Text:

DOWNLOAD [PDF]

References

Abdullah, A. H., Abidin, N. L. Z., & Ali, M. (2015). Analysis of students’ errors in solving Higher Order Thinking Skills (HOTS) problems for the topic of fraction. Asian Social Science, 11(21), 133–142. https://doi.org/10.5539/ass.v11n21p133

Abosalem, Y. (2016). Assessment Techniques and Students’ Higher-Order Thinking Skills. International Journal of Secondary People, 4(1), 1–11. https://doi.org/10.11648/j.ijsedu.20160401.11

Ahmad, S., Prahmana, R. C. I., Kenedi, A. K., Helsa, Y., Arianil, Y., & Zainil, M. (2018). The instruments of higher order thinking skills. Journal of Physics: Conference Series, 943(1). https://doi.org/10.1088/1742-6596/943/1/012053

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. In Longman (2nd ed). Pearson Education Inc. https://doi.org/10.2307/2281462

Bloom, B. S., Engelheart, M. D., Furst, E. J., Walker H, H., & Krathwohl, D. . (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1. Cognitive Domain. In Longman, Green and Co Ltd (1st ed.). David McKay Company, Inc. https://doi.org/10.1300/J104v03n01_03

Blyman, K., Arney, K., Adams, B., & Hudson, T. (2020). Does Your Course Effectively Promote Creativity? Introducing the Mathematical Problem Solving Creativity Rubric. Journal of Humanistic Mathematics, 10(2), 157–193. https://doi.org/10.5642/jhummath.202002.09

Borko, H., Mayfield, V., Marion, S., Flexer, R., & Cumbo, K. (1997). Teachers’ developing ideas and practices about mathematics performance assessment: Successes, stumbling blocks, and implications for professional development. Teaching and Teacher Education, 13(3), 259–278.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (2005). How people learn: Brain, mind, experience, and school. In R. R. Cocking (Ed.), Social Sciences. National Research Council, National Academy Press.

Brookhart, S. M. (2010). How To Assess Higher-Order Thinking Skills in Your Classroom. In Association for Supervision and Curriculum Development. Association for Supervision and Curriculum Development. https://doi.org/10.1177/002205741808801819

Brookhart, S. M., & Nitko, A. J. (2011). Strategies for Constructing Assessments of Higher-Order Thinking Skills. In G. Schraw & D. H. Robinson (Eds.), Assessment of Higher Order Thinking Skills (pp. 153–168). Information Age and Publishing. https://doi.org/10.4018/978-1-7998-2314-8.ch008

Brown, J. D. (2014). Classical Theory of Validity. In A. J. Kunnan (Ed.), The Companion to Language Asessment: Vol. III (1st ed.). https://doi.org/10.14361/transcript.9783839428566.toc

Collins, R. (2014). Skills for the 21st Century : Teaching Higher-Order Thinking. Curriculum and Leadership Journal: An Electronic Journal for Leaders in Education, 12(14), 4–7. http://www.curriculum.edu.au/leader/teaching_higher_order_thinking,37431.html

Collins, V. (2010). Higher order thinking (HOT) program assessment plan. Fort Worth, Texas: University of North Texas Health Science Center.

Conklin, W. (2012). Higher-Order Thinking Skills to Develop 21st Century Learners. In Shell Education.

Dewey, J., & Bento, J. (2009). Activating children’s thinking skills (ACTS): The effects of an infusion approach to teaching thinking in primary schools. British Journal of Educational Psychology, 79(2), 329–351. https://doi.org/10.1348/000709908X344754

Di Leo, I., Muis, K. R., Singh, C. A., & Psaradellis, C. (2019). Curiosity… Confusion? Frustration! The role and sequencing of emotions during mathematics problem solving. Contemporary Educational Psychology, 58, 121–137.

Doolen, J. (2015). Psychometric properties of the simulation thinking rubric to measure higher order thinking in undergraduate nursing students. Clinical Simulation in Nursing, 11(1), 35–43. https://doi.org/10.1016/j.ecns.2014.10.007

Ercikan, K., & Seixas, P. (2020). Assessment of Higher Order Thinking: the Case of Historical Thinking. In G. Schraw & D. R. Robinson (Eds.), Assessment of Higher Order Thinking Skills (1st ed., pp. 153–168). Infor. https://doi.org/10.4018/978-1-7998-2314-8.ch008

Gallagher, A. M., De Lisi, R., Holst, P. C., McGillicuddy-De Lisi, A. V, Morely, M., & Cahalan, C. (2000). Gender differences in advanced mathematical problem solving. Journal of Experimental Child Psychology, 75(3), 165–190.

Gradini, E. (2021). Penyusunan Soal Tes Bermuatan Lower dan Higher-Order Thinking Skills pada Guru SMP. JIPEMAS: Jurnal Inovasi Hasil Pengabdian Masyarakat, 4(3), 1–2. https://doi.org/http://dx.doi.org/10.33474/jipemas.v4i3.10121

Gradini, E., Firmansyah, B., & Noviani, J. (2018). Menakar Kemampuan Berpikir Tingkat Tinggi Calon Guru Matematika Melalui Level HOTS Marzano. EduMa, 7(2), 41–48.

Gregory, R. J. (2011). Psychological Testing: History, Principles, and Applications (7th editio). Pearson.

Halpern, D. F. (1999). Thought and knowledge an Introduction to critical thinking. In Education (Issue 80). https://doi.org/10.1002/tl.8005

Heong, Y. M., Othman, W. B., Yunos, J. B. M., Kiong, T. T., Hassan, R. Bin, & Mohamad, M. M. B. (2011). The Level of Marzano Higher Order Thinking Skillsamong Technical Education Students. International Journal of Social Science and Humanity, 1(2), 121–125. https://doi.org/10.7763/ijssh.2011.v1.20

Higgins, S., Hall, E., Baumfield, V., & Moseley, D. (2005). A Meta-analysis of the Impact of the Implementation of Thinkings Skills approach on Pupils (Vol. 338, Issue October).

Hyerle, D., & Alper, L. (2013). Thinking Maps for Meetings of the Mind. In A. L. Costa & P. W. O’Leary (Eds.), The power of the Social Brain: Teaching, Learning, and Interdependent Thinking (pp. 102–114). Teacher College Press, Columbia University.

Ichsan, I. Z., Sigit, D. V., Miarsyah, M., Ali, A., Arif, W. P., & Prayitno, T. A. (2019). HOTS-AEP: Higher order thinking skills from elementary to master students in environmental learning. European Journal of Educational Research, 8(4), 935–942. https://doi.org/10.12973/eu-jer.8.4.935

Jelatu, S., Mandur, K., Makur, A. P., Nendi, F., & Gunur, B. (2019). Konstruksi Tes High Order Thinking Skills (HOTS) bagi Guru-Guru Matematika SMP di Manggarai Timur. E-Dimas: Jurnal Pengabdian Kepada Masyarakat, 10(2), 214. https://doi.org/10.26877/e-dimas.v10i2.3070

Kennedy High School. (2006). Critical Thinking and Problem Solving - Mathematics. Kennedy High School. https://www.waterbury.k12.ct.us/2/Content2/460

Kruger, K. (2013). Higher-Order Thinking . Hidden Sparks. New York Ic.

Lane, S., Stone, C. A., Ankenmann, R. D., & Liu, M. (1994). Reliability and validity of a mathematics performance assessment. International Journal of Educational Research, 21(3), 247–266.

Lewis, A., Smith, D., & Lewis, A. (2009). Defining Higher Order Thinking. Theory Into Practice, 32(March 2015), 131–137. https://doi.org/10.1080/00405849309543588

Madhuri, G. V., Kantamreddi, V. S. S. N., & Prakash Goteti, L. N. S. (2012). Promoting higher order thinking skills using inquiry-based learning. European Journal of Engineering Education, 37(2), 117–123. https://doi.org/10.1080/03043797.2012.661701

Malik, A., Ertikanto, C., & Suyatna, A. (2015). Deskripsi Kebutuhan HOTS Assessment Pada Pembelajaran Fisika dengan Metode Inkuiri Terbimbing. Prosiding Seminar Nasional Fisika (E-Journal) SNF2015, IV, 1–4.

Marshall, J. C., & Horton, R. M. (2011). The Relationship of Teacher-Facilitated, Inquiry-Based Instruction to Student Higher-Order Thinking. School Science and Mathematics, 111(3), 93–101. https://doi.org/10.1111/j.1949-8594.2010.00066.x

Marzano, R. J., Brandt, R. S., Jones, B. F., Presseisen, B. Z., & Suhor, C. (1988). Dimensions of Thinking: A Framework for Curricullum and Instruction. Association for Supervision and Curriculum Development.

Marzano, R. J., & Kendall, J. S. (2007). Praise for the Second Edition of The New Taxonomy of Educational Objectives. Corwin Press, i–ii.

Miri, B., David, B., & Uri, Z. (2007). Purposely Teaching for the Promotion of Higher-order Thinking Skills : A Case of Critical Thinking. Research of Science and Education, 37(3), 353–369. https://doi.org/10.1007/s11165-006-9029-2

Newmann, F. M. (1990). Higher order thinking in teaching social studies: A rationale for the assessment of classroom thoughtfulness. Journal of Curriculum Studies, 22(1), 41–56. https://doi.org/10.1080/0022027900220103

Nieveen, N., & Folmer, E. (2013). Formative Evaluation in Educational Design Research. In T Plomp & N. Nieveen (Eds.), Netherlands Institute for Curriculum Development: SLO (pp. 152–169). Netherlands Institute for Curriculum Development: SLO. http://www.eric.ed.gov/ERICWebPortal/recordDetail?accno=EJ815766

Norris, S. P., & Ennis, R. H. (1989). Evaluating Critical Thinking. The Practitioners’ Guide to Teaching Thinking Series. Box 448, (Issue September). Critical Thinking Press and Software.

Peter, E. E. (2012). Critical thinking: Essence for Teaching Mathematics and Mathematics Problem Solving Skills. African Journal of Mathematics and Computer Science Research, 5(3), 39–43. https://doi.org/10.5897/ajmcsr11.161

Plomp, Tjeerd. (2000). Development Research in / on Educational Development. CITE Seminar Series, 04. http://hdl.handle.net/10722/44041

Pogrow, S. (2005). HOTS Revisited: A Thinking Development Approach to Reducing The Learning Gap After Grade 3. Phi Delta Kappan, 87(1), 64–75.

Pratama, G. S., & Retnawati, H. (2018). Urgency of Higher Order Thinking Skills (HOTS) Content Analysis in Mathematics Textbook. Journal of Physics: Conference Series, 1097(1). https://doi.org/10.1088/1742-6596/1097/1/012147

Preus, B. (2012). Authentic Instruction for 21st Century Learning : Higher Order Thinking in an Inclusive. American Secondary Education, 40(3), 59–79.

Rahmawatiningrum, A., Kusmayadi, T. A., & Fitriana, L. (2019). Student’s ability in solving higher order thinking skills (HOTS) mathematics problem based on learning achievement. Journal of Physics: Conference Series, 1318(1). https://doi.org/10.1088/1742-6596/1318/1/012090

Rapih, S., & Sutaryadi, S. (2018). Perpektif Guru Sekolah Dasar Terhadap Higher Order Tinking Skills (HOTS): Pemahaman, Penerapan dan Hambatan. Premiere Educandum : Jurnal Pendidikan Dasar Dan Pembelajaran, 8(1), 78. https://doi.org/10.25273/pe.v8i1.2560

Resnick, L. B. (1987). Education and Learning to Think. National Academy Press. https://bit.ly/3imoWkT

Retnawati, H., Munadi, S., Arlinwibowo, J., Wulandari, N. F., & Sulistyaningsih, E. (2017). Teachers’ difficulties in implementing thematic teaching and learning in elementary schools. New Educational Review, 48(2), 201–212. https://doi.org/10.15804/tner.2017.48.2.16

Rosidin, U., Suyatna, A., & Abdurrahman, A. (2019). A combined HOTS-based assessment/STEM learning model to improve secondary students’ thinking skills: A development and evaluation study. Journal for the Education of Gifted Young Scientists, 7(3), 435–448. https://doi.org/10.17478/jegys.518464

Sa’Dijah, C., Sa’Diyah, M., Sisworo, & Anwar, L. (2020). Students’ mathematical dispositions towards solving HOTS problems based on FI and FD cognitive style. AIP Conference Proceedings, 2215(April). https://doi.org/10.1063/5.0000644

Santoso, T., Cholily, Y. M., & Syaifuddin, M. (2021). An Analysis of Students’ Errors in Completing Essay HOTS Questions Based On Watson’s Criteria Viewed From The Cognitive Style Perspective. JTAM (Jurnal Teori Dan Aplikasi Matematika), 5(1), 121. https://doi.org/10.31764/jtam.v5i1.3776

Saxton, E., Belanger, S., & Becker, W. (2012). The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments. Assessing Writing, 17(4), 251–270.

Staples, M. E., & Truxaw, M. P. (2010). The Mathematics learning discourse project: Fostering higher order thinking and academic language in urban mathematics classrooms. Journal of Urban Mathematics Education, 3(1), 27–56.

Sujadi, I., Kurniawati, I., & Wulandarai, A. N. (2020). Upaya Meningkatkan Pembelajaran Berorientasi HOTS Bagi Guru Matematika SMP Kota Surakarta dengan Pemanfaatan Hasil Program AKSI for. DEDIKASI : Community Service, 2(1), 1–8.

Surya, A., Sularmi, ., Istiyati, S., Wahyuningsih, T., & Sriyanto, . (2020). Analysis of The Elementary School Teacher’s Need in The Implementation of HOTS (Higher Order Thinking Skills) based on Mathematics Learning. ICMIs 2018, 517–521. https://doi.org/10.5220/0008524705170521

Thatcher, R. W. (2010). Validity and Reliability of Quantitative Electroencephalography. Journal of Neurotherapy, 14(2), 122–152. https://doi.org/10.1080/10874201003773500

Tractenberg, R. (2020). The Assessment Evaluation Rubric: Promoting learning and learner -centered teaching in higher education. https://doi.org/10.31235/osf.io/bvwhn

University of South Florida. (2021). Reliability and Validity Classroom Assessment. https://fcit.usf.edu/assessment/basic/basicc.html

Van De Akker, J., Graveimejer, K., & McKenney, S. (2006). Introducing Educational Design Research (1st Editio). Routledge Publishers.

Widana, I. W., Parwata, I. made Y., Parmithi, N. N., Jayantika, I. G. A. T., Sukendra, K., & Sumandya, I. W. (2018). Higher Order Thinking Skills Assessment towards Critical Thinking on Mathematics Lesson. International Journal of Social Sciences and Humanities (IJSSH), 2(1), 24–32. https://doi.org/10.29332/ijssh.v2n1.74

Yee, M. H., Yunos, J. M., Othman, W., Hassan, R., Tee, T. K., & Mohamad, M. M. (2015). Disparity of Learning Styles and Higher Order Thinking Skills among Technical Students. Procedia - Social and Behavioral Sciences, 204(November 2014), 143–152. https://doi.org/10.1016/j.sbspro.2015.08.127

Zohar, A., & Barzilai, S. (2015). Metacognition and Teaching Higher Order Thinking (HOT) in Science Education: Students’ Learning, Teachers’ Knowledge, and Instructional Practice. In R. Wegerif, L. Li, & J. C. Kaufman (Eds.), The Routledge International Handbook of Research on Teaching Thinking (1st ed., p. 14). Routledge: Taylor & Francis Group.

DOI: https://doi.org/10.31764/jtam.v6i1.5342