In this paper, we present an augmented reality learning system that uses the input of a depth camera to interactively teach anatomy to high school students. The objective is to exemplify human anatomy by displaying 3D models over the body of a person in real time, using the Microsoft Kinect depth camera. The users can see how bones, muscles, or organs are distributed in their bodies without the use of targets for tracking.

Introduction

A difficult topic for high school students, human anatomy has been taught in the same way for many years—that is, using static images of the human body. In this type of learning, students are passive and may lose attention easily, and it is for this reason that researchers have investigated new learning alternatives. One such alternative is the use of information technology (IT), which has demonstrated strong potential for improving learning processes (Vivekananthamoorthy et al., 2009; Raghavendra & Rajini, 2012). Augmented reality (AR), a prominent IT tool, has been used since the 1990s in medicine, manufacturing, aeronautics, robotics, entertainment, and more recently, education. This technology has a demonstrated ability to reduce cognitive overload, which helps students utilize knowledge in creative and meaningful ways (Bower et al., 2013).

An AR system can be defined as one that allows real and virtual objects to coexist in the same space in real time. A depth camera consists of a red-green-blue (RGB) camera and a depth camera with an infrared laser. This device calculates the relative distance between any two parts of the human body, and on this basis, it knows where every joint is in relation to the three-dimensional (3D) world. Consequently, it provides a full-body 3D motion capture. The Microsoft Kinect is a depth camera. With recent improvements in computation and the introduction of low-cost depth cameras to the market, a great deal of visual processing can be performed in real time without concerns regarding light quality or slow mathematical processing, which were among the main problems with AR technology in the past. For this reason, AR is now more widely used, and new generations of students are learning how to use it in the context of games, which facilitates their ability to interact with this paradigm. In addition, the cost of AR technology has decreased. In previous years, the required hardware was quite costly—it started at around $10,000—but it now costs as little as $1,000. Taking this into account, we developed a low-cost AR learning system for the study of anatomy in high school. This system enables students to visualize bones, muscles, and organs (digestive, circulatory, urinary, and respiratory systems), images of which are superimposed on their bodies in the correct positions. The system helps students identify the main elements of human anatomy in an easy and interactive way.

Related Research

Innovative methodologies that include the use of IT are being integrated into educational institutions around the world to generate new teaching and learning procedures (Srivastava, 2012). Consequently, education is changing from teacher-centered, passive learning to student-centered, active learning, a development that is improving the relevance and quality of instruction and making it more dynamic and interesting. Newly adopted methodologies include virtual reality (Sampaio & Viana, 2013), e-learning (Kapenieks, 2011), and most recently, AR systems (Sumadio & Rambli, 2010; Phon et al., 2014; Zagoranski & Divjak, 2003).

With respect to the use of IT for the study of anatomy, other researchers (Grimstead et al., 2007) have used virtual reality to enable users to interact with 3D models of the human body in medical contexts in multiple devices, from light-weight computers and other mobile devices, to large-scale stereo displays through a web application. The application developed by Grimstead et al. (2007) is useful for in-depth study of the human body because it provides a great deal of information and displays complex anatomical models for medical users. In a high school context, however, this application has the following drawbacks: the amount and complexity of information presented may be overwhelming for high school students; it is difficult for users to visualize the real-world proportions of the organs; and the application requires an Internet connection.

Other applications (Bichlmeier et al., 2007; Samosky et al., 2012; Kondo et al., 2010) superimpose computer graphics on a physical body-type figure (like a medical mannequin), which helps users visualize the shape, size, and position of anatomical structures. These systems must be installed in a specific space because they require multiple pieces of equipment, such as the body-type figure, a projector, electro-magnetic 3D position sensors, and head-mounted displays; for the same reason, these applications are costly and are not portable. On the other hand, they are useful to visualize a specific part of the body at each moment.

Among more recent applications (Placitelli & Gallo, 2011; Blum et al., 2012; Meng et al., 2013), some use a depth camera to track the user's body in order to create a magic mirror that displays the user's image along with augmented images of the anatomical structures. Placitelli and Gallo's (2011) application creates an in situ visualization that combines a patient's specific data and the patient's 3D data. This enables the doctor to observe a 3D visualization of the patient's computerized tomography/magnetic resonance imaging (CT/MRI) data, which is superimposed on the patient. This system is useful in medical contexts, but it detects and displays only the user's data. Another application (Blum et al., 2012; Meng et al., 2013) tracks the user in real time with a Microsoft Kinect camera, and a 3D model is superimposed on the user, displaying the CT dataset on one side. However, this application lets the user see only the organs and bones located in the thoracic section, and it does not indicate the names of the displayed anatomical elements, which is essential for high school students.

Our learning system uses AR by displaying and tracking in real time the student's image and superimposing the 3D anatomical elements on it. It displays not only the thoracic anatomical elements, but all the bones, muscles, and organs that are taught in high school. It also indicates the names of the anatomical elements in two languages (English and Spanish). The system is easy to install and can perform real-time tracking of the user's movements and height to adapt and translate the anatomical elements as needed.

Development of the Augmented Reality Learning Interactive Tool System

We developed a portable learning system that consists of only three main components: a computer, a screen, and a device with a depth and RGB camera.

The system's depth and RGB functions are both accomplished with a Microsoft Kinect camera. The minimum requirement for running the program is a computer with at least four gigabytes of random-access memory and an NVIDIA GeForce GTX 480 graphics card or the equivalent. The screen used to display the application can be either a monitor or a projector that supports the exact resolution of 1,280 by 920 pixels. These three components can be set up easily in any educational site.

To use the system in the classroom, the teacher asks a student to stand at the front of the classroom, where the depth camera is located (it is connected to the classroom computer and placed on a tripod 85 cm in height), and the rest of the students can look at the screen to see the user interacting with the anatomical elements in real time (see Figure 1). Using the hands and feet, the student can select different options from the system menu to visualize the muscles, organs, and bones with their anatomical names in English or Spanish. The teacher can ask different students to use the system to select different anatomical elements, giving a certain number of minutes to each student. This teaching method helps students remember the anatomical elements' position and size in relation to their own bodies. The system can be moved at any time from one classroom to another; it can even be used in a nontraditional classroom, like a home or other space.

Figure 1.

Visualization of the system and its three main components: depth camera, computer, and screen.

Figure 1.

Visualization of the system and its three main components: depth camera, computer, and screen.

The system is adaptable and automatic; it adjusts to the height and complexion of any person. This functionality was achieved using the depth camera and the implemented algorithms. The depth camera uses an infrared laser to produce a speckle pattern and an infrared sensor to collect the produced data. After gathering the information and applying computer vision techniques, the depth camera is able to determine the relative distance between the anatomical elements. In addition, the use of computational algorithms enables the camera to detect whether a human body is present and, if so, to generate information regarding the position of some established points that represent important joints in the body. This depth information is combined with the color data from the RGB camera to map the user's movements to screen coordinates (see Figure 2) in order to superimpose virtual anatomical elements with a low degree of position error.

Figure 2.

Combination of depth data and color data. The green points represents the joints of the body.

Figure 2.

Combination of depth data and color data. The green points represents the joints of the body.

After matching the depth and color camera, 3D models of the anatomical elements are superimposed on the student's body by assigning their position and rotation according to the different joints of the body (Figure 3). The 3D model sizes are adjusted by defining the initial height of the user, which corresponds to the initial size of each model, and then scaling the models in proportion to the user's height.

Figure 3.

Three-dimensional anatomical elements are superimposed on the student's body.

Figure 3.

Three-dimensional anatomical elements are superimposed on the student's body.

The interaction with the application was analyzed in detail, because it should be easy to use but robust enough to be used in a classroom full of students. For this reason, we implemented a gesture routine for calibration, which limits the system's tracking to one person at any given moment. This was done by identifying the joints of the body on the Kinect and simply tracking the user with hands over the head for five frames. Subsequently, the tracked user can choose between menus that were designed to be operated with the hands. Since the beginning, the menus were not implemented with gestures, because as a result of the user's movement, false positive gestures could be triggered by mistake.

System Implementation

The system was developed in C++ with OpenGL to make the software light and fast. The depth camera used was the Kinect Sensor with Kinect Software Development Kit (SDK) Libraries from Microsoft. The format of the 3D models used was OBJ, which is an open and widely adopted 3D graphic format, and the anatomical models were obtained from the National Bioscience Database Center (N.B.D. Center, 2009). We modified the 3D models to optimize them for the real-time performance of the system (Figures 46). To load the 3D models into the system, we used the OBJ model loader developed by Nate Robins (improved version developed by Tudor Carean; Robins, 2000; Carean, 2008).

Figure 4.

Three-dimensional model of the digestive system.

Figure 4.

Three-dimensional model of the digestive system.

Figure 5.

Three-dimensional model of the skeletal system.

Figure 5.

Three-dimensional model of the skeletal system.

Figure 6.

Three-dimensional model of the respiratory system.

Figure 6.

Three-dimensional model of the respiratory system.

The first step of building the system was adding the video streaming from the Kinect RGB camera to a plane in OpenGL. To do this, we converted each frame coming from the Kinect camera to an OpenGL texture and assigned it to the plane in every frame. Next, we obtained the position of each joint from the depth camera skeleton and transformed it to color-pixel coordinates in order to be able to match the user's digital skeleton with the color image (see Figure 7). Moreover, we applied some OpenGL lighting to visualize the textures of each 3D model.

Figure 7.

System implementation.

Figure 7.

System implementation.

To superimpose the anatomical elements, each model was assigned to the different joints according to the position of the body. The pivot point of each model was adjusted to enable it to translate and rotate according to the joint movement. For the rotation of thoracic elements, we used the rotation matrix from the Kinect SDK, and we calculated the rotation of arms and legs by determining the angle from one joint to another (calculating the slope with the x, y, and z coordinates of both joints).

After some testing, and after taking into account how the depth camera works, we determined that two criteria must be met for the system to work correctly: do not wear dark or bulky clothes, and do not wear a skirt.

A: Interface Design

The main menu is displayed on the left and right sides of the system's window, and the user selects the different options by extending the right or left arm to the chosen option, hovering over it for two seconds to select it. When an option is selected, a submenu appears, and the user selects another option, at which point a third and final menu appears. The menus are organized hierarchically. The main (or first) menu includes the general options bones, muscles, and organs. The second menu includes suboptions such as head, torso, legs, respiratory system, and circulatory system, referring to divisions of the systems to facilitate searching and understanding. The third and final menu includes options for the unit elements of anatomy, such as humerus, biceps, and lungs. This implementation was developed so the user would not select or trigger a menu while moving, and he or she would also be able to see what had been selected. Moreover, when an option is selected from the first menu, the system displays the bones, organs, or muscles over the user's body. Subsequently, after an option in the second menu is selected, the selected 3D models are highlighted in yellow. Finally, when the unit anatomical element is selected in the third menu, this element is highlighted in blue, which allows the student to identify the anatomical element by name.

Although this menu system worked well, after trying it out with multiple students, we observed that the scroll and return options were difficult to select and not very intuitive. To improve the menu, we collected the students' feedback and analyzed their movements (Figure 8). The scroll option was moved from the right side of the screen to the top, and the return option was moved from the right side to the lower left side so students could select it with their left foot.

Figure 8.

Final menu implementation. The selected section is highlighted in yellow, and the selected anatomical element is highlighted in blue.

Figure 8.

Final menu implementation. The selected section is highlighted in yellow, and the selected anatomical element is highlighted in blue.

B: Educational Requirements

Although the system contained all the bones, muscles, and organs, after we presented the system to high school teachers, they informed us that their instruction emphasized bones and addressed fewer muscles and organs (see Table 1), so we removed the muscles and organs that were not taught.

Table 1.
Anatomical elements considered in the system.
Anatomical Systems
MusclesOrgansBones
Anatomical elements Biceps Heart: Left Atrium Carpus 
Branchialis Heart: Left Ventricle Cervical Vertebra 
Deltoid Heart: Right Atrium Clavicle 
Frontal Heart: Right Ventricle Cranium 
Gastrocnemius Appendix Femur 
Intercostals Duodenum Esophagus Fibula 
Nasalis Large Intestine Head 
External Oblique Liver Humerus 
Occipitofrontalis Pancreas Mandible 
Orbicularis Oris Small Intestine Kneecap 
Pectoral Spleen Legs 
Pronator Teres Stomach Lumbar Vertebra 
Rectus Abdominis Sublingual Glands Metacarpals 
Rectus Femoris Submandibular Glands Metatarsus 
Sartorius Adrenal Gland Pelvis 
Sternocleidomastoid Hypothalamus Phalanges 
Temporalis Muscle Lobe of Thymus Radius 
Tibialis Anterior Pancreas Ribs 
Trapezius Pituitary Gland Sacrum 
Triceps Thyroid Gland Scapula 
Vastus Lateralis Brain Tarsus 
Vastus Medialis Spinal Cord Thoracic Vertebra 
 Bronchus Tibia 
 Lungs Torso 
 Pharynx Ulna 
 Trachea  
 Bladder  
 Kidneys  
 Palatopharyngeus Ureters  
Anatomical Systems
MusclesOrgansBones
Anatomical elements Biceps Heart: Left Atrium Carpus 
Branchialis Heart: Left Ventricle Cervical Vertebra 
Deltoid Heart: Right Atrium Clavicle 
Frontal Heart: Right Ventricle Cranium 
Gastrocnemius Appendix Femur 
Intercostals Duodenum Esophagus Fibula 
Nasalis Large Intestine Head 
External Oblique Liver Humerus 
Occipitofrontalis Pancreas Mandible 
Orbicularis Oris Small Intestine Kneecap 
Pectoral Spleen Legs 
Pronator Teres Stomach Lumbar Vertebra 
Rectus Abdominis Sublingual Glands Metacarpals 
Rectus Femoris Submandibular Glands Metatarsus 
Sartorius Adrenal Gland Pelvis 
Sternocleidomastoid Hypothalamus Phalanges 
Temporalis Muscle Lobe of Thymus Radius 
Tibialis Anterior Pancreas Ribs 
Trapezius Pituitary Gland Sacrum 
Triceps Thyroid Gland Scapula 
Vastus Lateralis Brain Tarsus 
Vastus Medialis Spinal Cord Thoracic Vertebra 
 Bronchus Tibia 
 Lungs Torso 
 Pharynx Ulna 
 Trachea  
 Bladder  
 Kidneys  
 Palatopharyngeus Ureters  

Moreover, in the first version, the system worked as a mirror, so the student using it visualized his or her body with the anatomical elements in the screen, as though looking into a mirror. This approach related the real image to the student's augmented image, but some teachers informed us that this could create confusion for students, who are accustomed to seeing anatomical images as a person facing them instead of as seen in a mirror. For this reason, we presented to the students first the mirror version of the system and then a second version that displayed the elements as anatomical images (instead of mirroring them), and asked the students which version they understood better. After several tests, we concluded that the mirror version was easier to understand.

Results

The system was used in five classrooms. In each classroom, we introduced the system and gave students seven instructions on how to use it (see Figure 9), and all the students were able to use it correctly without further explanation.

Figure 9.

Instructions given to the students. With only seven sentences, they were able to use the system easily.

Figure 9.

Instructions given to the students. With only seven sentences, they were able to use the system easily.

After reading the instructions, the user chooses between English and Spanish and then selects either organs, bones, or muscles. At this point, the 3D anatomical elements appear superimposed on the student's body (see Figures 3, 8, and 10, respectively).

Figure 10.

Three-dimensional muscle elements superimposed on the student's body.

Figure 10.

Three-dimensional muscle elements superimposed on the student's body.

The system successfully tracked the users, adjusting the size of the anatomical elements to their height. The students were excited about using the system, and many asked to try it. We explained to the teachers how to set up the system, and they were able to do it by themselves without any problems.

After implementing the system in the classrooms, we administered a usability survey to 68 students. The survey consisted of seven questions to be answered on a scale from 1 (strongly agree) to 5 (strongly disagree). The results of the survey are shown in Figure 11. It is important to note that students found that using the system facilitated their retention of the information.

Figure 11.

Results of the student survey (1 = strongly agree, 5 = strongly disagree).

Figure 11.

Results of the student survey (1 = strongly agree, 5 = strongly disagree).

The results of the survey indicate that interaction with the system is both easy and entertaining, which encourages students to learn and practice the given subject. The results also indicate that most of the students thought that using the system helped them learn the subject faster and better.

This project participated in the 2015 Wharton-QS Stars Reimagine Education Awards, achieving the “K-12 Innovation” Bronze 2015 Discipline Award.

Conclusion and Future Work

We have proposed a new, interactive methodology for learning anatomy that displays organs, muscles, and bones over the body of the user in real time. The system uses AR to superimpose on students' bodies the anatomical elements taught in high school, and can adapt to the height of any student. The system has already been installed in one high school classroom, and its portability allows it to be moved to other classrooms. The system also has a low cost and can be assembled easily.

In future work, we will develop a methodology for devising short anatomy quizzes. We also plan to add more models to the system, with animations and brief descriptions of anatomical elements. Although there are no specific Next Generation Science Standards for teaching anatomy in high school, according to the topics for life science, the students have demonstrated a full understanding of anatomy for their educational level. Using an interactive, real-time simulation, they gained an understanding of the structural organization of biological systems, in this case, systems of human anatomy. This project can be a teaching model for creating a future standard for the anatomy topic.

The authors would like to thank Maria Antonieta Alvarez-Polo and Lourdes Bejarano-Gómez for their support and feedback. The research and development for this project was supported by Fondo de Innovación Educativa NOVUS and the University Tecnológico de Monterrey.

References

References
Bichlmeier, C., Wimmer, F., Heining, S. M., & Navab, N. (
2007
,
November
).
Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality
. In
Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on
,
IEEE
, pp.
129
138
.
Blum, T., Kleeberger, V., Bichlmeier, C., & Navab, N. (
2012
,
March
).
mirracle: An augmented reality magic mirror system for anatomy education
. In
2012 IEEE Virtual Reality Workshops (VRW)
,
IEEE
, pp.
115
116
.
Bower, M., Howe, C., McCredie, N., Robinson, A., & Grover, D. (
2014
).
Augmented Reality in education–cases, places and potentials
.
Educational Media International
,
51
(
1
),
1
15
.
Carean, T. (
2008
).
Nate Robins .obj loader with texture support
. Retrieved from https://github.com/royshil/HeadPosePnP/blob/master/glm.h and https://github.com/royshil/HeadPosePnP/blob/master/glm.cpp
Grimstead, I., Walker, D., Avis, N., Kleinermann, F., & McClure, J. (
2007
).
3D anatomical model visualization within a grid-enabled environment
.
Computing in Science Engineering
,
9
(
5
),
32
38
.
Kapenieks, J. (
2011
,
April
).
Knowledge creation: action research in e-learning teams
. In
2011 IEEE Global Engineering Education Conference (EDUCON)
,
IEEE
, pp.
859
864
).
Kondo, D., Suzuki, H., Kijima, R., & Maeda, T. (
2010
,
October
).
Remote education system using virtual anatomical model
. In
Virtual Systems and Multimedia (VSMM), 2010 16th International Conference on
,
IEEE
, pp.
375
377
.
Meng, M., Fallavollita, P., Blum, T., Eck, U., Sandor, C., Weidert, S.,…& Navab, N. (
2013
,
October
).
Kinect for interactive AR anatomy learning
. In
Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on
,
IEEE
, pp.
277
278
.
N.B.D. Center
. (
2009
).
Bodyparts3d, the database center for life science licensed under cc attribution-share alike 2.1 Japan
. Retrieved from http://lifesciencedb.jp/bp3d
Phon, D. N. E., Ali, M. B., & Halim, N. D. A. (
2014
,
April
).
Collaborative Augmented Reality in Education: A Review
. In
Teaching and Learning in Computing and Engineering (LaTiCE), 2014 International Conference on
,
IEEE
, pp.
78
83
.
Placitelli, A. P., & Gallo, L. (
2011
,
November
).
3D point cloud sensors for low-cost medical in-situ visualization
. In
Bioinformatics and Biomedicine Workshops (BIBMW), 2011 IEEE International Conference on
,
IEEE
, pp.
596
597
.
Raghavendra, N., & Rajini, R. (
2012
,
July
).
A qualified analysis of traditional and technology assisted learning: An IT industry outlook
. In
Engineering Education: Innovative Practices and Future Trends (AICERA), 2012 IEEE International Conference on
,
IEEE
, pp.
1
6
.
Robins, N.
2000
.
.obj loader
. Retrieved from http://devernay.free.fr/hacks/glm/
Samosky, J. T., Nelson, D. A., Wang, B., Bregman, R., Hosmer, A., Mikulis, B., & Weaver, R. (
2012
,
February
).
BodyExplorerAR: Enhancing a mannequin medical simulator with sensing and projective augmented reality for exploring dynamic anatomy and physiology
. In
Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction
,
ACM
, pp.
263
270
.
Sampaio, A. Z., & Viana, L. (
2013
,
June
).
Virtual Reality used as a learning technology: Visual simulation of the construction of a bridge deck
. In
2013 8th Iberian Conference on Information Systems and Technologies (CISTI)
,
IEEE
, pp.
1
5
.
Srivastava, P. (
2012
,
January
).
Educational informatics: An era in education
. In
Technology Enhanced Education (ICTEE), 2012 IEEE International Conference on
,
IEEE
, pp.
1
10
.
Sumadio, D. D., & Rambli, D. R. A. (
2010
,
March
).
Preliminary evaluation on user acceptance of the augmented reality use for education
. In
Computer Engineering and Applications (ICCEA), 2010 Second International Conference on
,
IEEE
, Vol.
2
, pp.
461
465
.
Vivekananthamoorthy, N., Sankar, S., Siva, R., & Sharmila, S. (
2009
,
December
).
New paradigms for innovation in teaching and learning process
. In
2009 7th International Conference on ICT and Knowledge Engineering
,
IEEE
, pp.
94
99
.
Zagoranski, S., & Divjak, S. (
2003
,
September
).
Use of augmented reality in education
. In
2003 The IEEE Region 8 (EUROCON) on
,
IEEE
, Vol.
2
, pp.
339
342
.