Development and Evaluation of the Utility of a Respiratory Monitoring and Visual Feedback System for Radiotherapy Using Machine Vision Technology
Article information
Abstract
Background
We developed a machine vision technology program that tracks patients’ real-time breathing and automatically analyzes their breathing patterns.
Materials and Methods
To evaluate its potential for clinical application, the image tracking performance and accuracy of the program were analyzed using a respiratory motion phantom. Changes in the stability and regularity of breathing were observed in healthy adult volunteers according to whether the breathing pattern mirrored the breathing guidance.
Results and Discussion
Displacement within a few millimeters was observed in real-time with a clear resolution, and the image tracking ability was excellent. This result was consistent even in the sections where breathing patterns changed rapidly. In addition, the respiratory gating method that reflected the individual breathing patterns improved breathing stability and regularity in all volunteers.
Conclusion
The findings of this study suggest that this technology can be used to set the appropriate window and the range of internal target volume by reflecting the patient’s breathing pattern during radiotherapy planning. However, further studies in clinical populations are required to validate this technology.
Introduction
When administering radiotherapy, changes in the position of tumors in the chest and abdomen are particularly affected by the respiratory cycle [1–5]. Respiratory-gated radiotherapy (RGRT), which irradiates in relation to a specific breathing cycle [6], is noninvasive and can reduce unnecessary radiation to normal tissues. Many previous studies have shown that RGRT is useful for lung and liver cancers, where tumor movement is closely related to respiration [7–12]. With RGRT, irradiation occurs only in association with a specific breathing cycle set in the treatment planning phase, so a stable and regular breathing pattern is required; if a patient cannot cooperate, the duration of the treatment may be prolonged.
Therefore, as a solution to this, respiratory monitoring and a respiratory gating device can be used to closely observe a patient’s breathing pattern and to induce regular breathing. The types of respiratory monitoring devices commercially available in clinic these days are: a respiratory gating system (Varian Medical Systems, Palo Alto, CA, USA) [13]; a spirometer (Active Breathing Coordinator; Elekta, Stockholm, Sweden) that measures the rate of change in the airflow through a mouthpiece [14]; and a system incorporating a belt-type sensor (AZ-733VI; Siemens Healthineers, Erlangen, Germany) that measures the change of strain gauge [15]. Another described method [16, 17] incorporates a thermocouple sensor into a nasal mask to indicate the subtle temperature difference during inhalation and exhalation.
However, the aforementioned respiratory gating methods are sensitive to changes in the surrounding environment and may be limited in clinical application to certain patients, if a sensor needs to be attached to the patient’s body. In addition, when inducing the patient’s breathing according to a standardized respiratory waveform that does not reflect the patient’s breathing patterns, adequate training and considerable effort is required from a patient in advance.
In this study, machine vision technology, which is inexpensive, easy to use, and less restrictive, was used to acquire breathing signals in real-time and to analyze the breathing pattern acquired for a specific time period. Prior to clinical application, phantoms were used in healthy adult volunteers to evaluate the performance and accuracy of the image tracking, and its clinical utility was assessed by monitoring the stability and regularity of the breathing, depending on the presence of the respiratory monitoring and guiding system.
Materials and Methods
In this study, machine vision technology, which is widely used in various fields such as control, inspection, measurement, and recognition, was applied to track the movement of markers that synchronize with breathing in a visible light environment. The system is divided into an image acquisition unit, for observing the motion, and an image processing unit to process the collected images and represent the region of interest as a real-time breathing signal. The image sensor used in the image acquisition unit (acA1300; Basler AG, Ahrensburg, Germany) has a resolution of 5.3 μm with 1,280×1,024 pixels, and a data transmission rate of up to 60 frames/s is possible. Its wide-angle lens with a 12 mm focal length (Computar Lens, Basler AG) allows for a wide angle of view and detailed observation of the movement. For data collection, a frame grabber (PXIe-8234; National Instruments Inc., Austin, TX, USA) with a transmission rate of up to 1,000 Mb/s per port was used.
The program for image processing and analysis of the upper and lower abdominal motions was developed using LabVIEW 2019 (National Instruments Inc.) and was run in a general desktop PC environment (Intel Xeon CPU E5-2620 2.4 GHz, 32.0 GB). This program was developed to track an object of interest in real-time using the geometric matching method [18] based on the pixel intensity of the template image. With the program, the movement of markers attached to the patient’s abdomen can be tracked, allowing for the extraction of the breathing signal. The acquired breathing signal is analyzed to ascertain the amplitude, period, and pattern of breathing, which generates a reference respiratory waveform to regularly induce the patient’s breathing. The reference waveform was generated as the average value of the phases after overlapping each breathing signal. During treatment, when the patient’s breathing was induced, the data were shown in the format of a bar graph so that the patient could easily follow the breathing pattern according to the signal (Fig. 1).
Configuration of the in-house software developed to analyze respiratory signals and guide patient’s respiration during the radiation treatment. (A) Respiratory signals obtained from the anterior-posterior motion of the marker placed on the abdomen of a volunteer. (B) Analysis screen of the measured breathing signals. (C) Average respiratory wave form. (D) Visual guidance screen for patient breathing.
Fig. 2 shows the RGRT procedure using the machine vision system proposed in this study. In this study, as part of a preliminary study on the clinical applicability of the machine vision system, the mechanical accuracy of the machine vision system was verified while excluding radiotherapy.
Flowchart of the respiratory gating radiotherapy procedure. The work carried out in this study is expressed in boldface. CT, computed tomography
1. Evaluation of the Mechanical Performance of the Machine Vision System
The respiratory motion phantom (QUASAR; Modus Medical Device Inc., London, ON, Canada) used in our study is capable of modifying waveforms, amplitudes, and periods through a commercialized program, and collecting information on the number and direction of revolutions through the encoder in the driving motor. Therefore, accuracy of the image tracking was evaluated through mutual comparison between the data collected through the encoder in the motor of the respiratory motion phantom and the location information observed through the camera of the machine vision system under various conditions.
In this study, the respiratory waveforms of an irregular breathing pattern, similar to the actual pattern, and of a sine wave shape with various periods and amplitudes were implemented using a respiratory motion phantom. In addition, the geometric matching method of the machine vision system was applied to generate the breathing signal of the respiratory motion phantom. The observed value was verified through location information stored in the encoder of the respiratory motion phantom, and was compared with the value obtained via the respiratory gating system (RPM, Varian Medical Systems) that is used in clinical settings.
2. Evaluation of the Utility of the Respiratory Gating Method in Reflecting Breathing Patterns
Ten adult healthy volunteers in a supine position were given a marker to be attached to their abdomen and were asked to breathe freely for about 3 minutes. Based on the respiratory information observed by the machine vision system, the amplitude, period, and waveform of the respiratory signal were analyzed, and the reference respiratory signal was generated. Then, the ease and stability of breathing that a patient would feel when the reference respiratory waveform, which takes into consideration the individual’s breathing pattern, or the standardized respiratory waveform was applied to the patient. First, an individualized reference breathing waveform was shown to the volunteer using a visual device, and the volunteer breathed for 3 minutes according to the shown waveform. After a sufficient resting period, a sine waveform with a period of 4 seconds was shown to the volunteer, who then breathed accordingly. After the experiment, the regularity and stability of the measured amplitude and periodicity were evaluated depending on whether one’s individual breathing pattern was reflected in respiratory gating.
Results and Discussion
1. Evaluation of the Mechanical Performance of the Machine Vision System
The position of the camera and the distance to the respiratory motion phantom were accounted for so that they did not interfere with the radiation treatment and the reproducibility of the measured values was not impaired. Fig. 3 shows the scatter plot of the marker positions per frame according to the distance between the marker attached to the respiratory motion phantom and the camera. From these data, the observation distance between the respiratory motion phantom and the camera was determined to be 80 cm.
Standard deviation of the observed marker position along the distance from the respiratory phantom.
The motion of the QUASAR phantom observed in real-time through the machine vision system, and the RPM system when the movements of irregular and regular respiratory signals were stimulated using the phantom, is shown in Fig. 4. The signals observed with both systems showed almost identical patterns to the motion information obtained from the respiratory motion phantom. The data collection rates of the machine vision system and the RPM system were 12 frames/s and 30 frames/s, respectively, and the time resolutions for each system were 0.08 seconds and 0.03 seconds, respectively.
Comparison of acquired the respiratory signal and the QUASAR respiratory motion phantom encoder signals for the participants with (A) regulated breathing and (B) irregular breathing.
Fig. 5 compares the movement observed by the machine vision system and the RPM system over one cycle period, when the motion of the phantom was in the form of a sine wave. When the phases of the three signals were aligned and the amplitudes were normalized to 1, the root mean square error (RMSE) of the signals obtained by the machine vision system and the RPM system relative to the actual movement of the phantom was 0.016 and 0.037, respectively.
Comparison of the QUASAR phantom movement recorded in the encoder over a period of a cycle, with the measurements observed by the RPM and the machine vision system.
Fig. 6 shows the difference between the movement of the phantom and the value observed by the machine vision system according to the slope of the breathing signals, after observing the movement of the phantom with the machine vision system for a specific period of time. The slope of the breathing signal was determined as the ratio of the distance the phantom moved per unit time in the region of observation. The difference in values observed by the motion vision system by the movement of the phantom increased in proportion to the slope of the respiratory signal at the observed point, and was independent of the direction of the phantom’s motion. Therefore, greater amplitude and shorter cycle of the breathing signal led to a greater RMSE of the values observed with the machine vision system, as shown in Table 1.
Residuals of the observations from the phantom movement with the machine vision system according to the slope of the breathing signals. The slope is the ratio of the pixel value displacement to the time increment at the observation point.
Comparison of the Result Obtained by the Vision Machine System with the Movement of the Motion Phantom at Various Amplitudes and Cycle Lengths
2. Regularity of Breathing depending on Whether the Breathing Pattern was Reflected during Respiratory Gating
Fig. 7 shows the standard deviation of the amplitude and period of breathing, depending on whether the breathing patterns of the 10 volunteers were reflected during respiratory gating. Guided breathing using the standardized sinewave improved the amplitude and periodicity of breathing by 60% and 70%, respectively, compared to free breathing; however, in the remaining 40% and 30% of the volunteers, respectively, the regularity of breathing decreased. This was because they felt discomfort to following the sinewave. However, the guided breathing method, which reflected the volunteers’ personal breathing patterns, improved the regularity of breathing in terms of both amplitude and cycle in all volunteers.
Variance in (A) the periods and (B) amplitudes of the volunteer’s breathing under the different guidance conditions and free breathing among the 10 volunteers. (A) shows that the amplitude was expressed in arbitrary units without any distance conversion corresponding to the pixel values, because the distance between the camera and the marker attached to the applicant’s abdomen was not always constant.
3. Advantages and Limitations
The duration of treatment with RGRT can be decided according to the patient’s breathing pattern. It is therefore important to keep the patient’s breathing stable as well as constant, and many studies have been conducted to develop techniques for the close monitoring of the patient’s breathing pattern and the induction of regular breathing [19–25]. Yoshitake et al. [19] developed a machine vision system which helps to tolerate the breath-hold maneuver. They concluded the machine vision visual feedback could increase the reproducibility. George et al. [22] developed an audio-visual bio-feedback system during free-breathing. The efficiency and reproducibility of the gating was improved.
In these previous studies, patients had difficulty breathing in synchrony with a standardized breathing signal that does not take into consideration the patients’ own breathing pattern. However, in our study, we evaluated a technique that extracts the patient’s breathing pattern based on their respiratory signal measured through a machine vision system and guides regular breathing that reflects the patient’s own breathing patterns during the treatment.
The biggest advantage of our system is that there is no psychological pressure or rejection associated with our system, since it is not necessary to wear a sensor for acquiring a breathing signal. In addition, the respiratory signal measurement using the geometry matching technology of the machine vision system uses the pattern of the image to evaluate whether the template matches the geometric information. It thus demonstrated excellent tracking ability regardless of changes in lighting, blurring, noise, object engagement, geometric position, rotation, and even slight changes in size. As a result, the machine vision system showed excellent image tracking ability even when the breathing pattern rapidly changed, and the RMSE was smaller compared to the existing image tracking technology that incorporates infrared reflectors. In addition, the high-definition image acquired by the camera is suitable for measuring breathing signals because the real-time observation of a few millimeters of displacement is possible with a clear image quality. Moreover, since the therapist can closely observe the patient’s condition through an external monitor during radiotherapy, any accident or emergency that may occur during the treatment can be prevented in advance.
However, there were some limitations. Owing to the physical limitations of the system for processing high-quality images, a significantly lower number of frames per second is obtained compared to the existing system. This makes it difficult to interpret the breathing pattern in detail when acquiring the breathing signal of a patient with irregular breathing, and it is thus necessary to improve the image processing speed of the system for future clinical application. Second, this study involved only 10 healthy adult volunteers, so the number of subjects was insufficient, and the psychological discomfort or changes in breathing patterns felt by actual patients with cancer could not be directly confirmed.
However, we found that the regularity of breathing significantly improved with guided breathing in healthy adult volunteers. Some volunteers felt uncomfortable and dizzy after breathing to a standardized signal; in particular, volunteers whose normal breathing cycle and amplitude differed from that of the standardized breathing showed a more irregular breathing than before the guided breathing was introduced. On the other hand, when guided breathing reflecting the individual breathing pattern was conducted, the regularity of the breathing greatly improved in all volunteers, and the discomfort felt by the volunteers was relatively lower.
Conclusion
The motion tracking performance with the machine vision system described in this study shows that the system can be used with RGRT, regardless of the surrounding environment or the presence of irregular breathing patterns. Therefore, we believe that RGRT using machine vision technology could be useful in the future for radiotherapy targeting lesions where movement is caused by breathing, such as tumors in the lung and liver. However, further studies in clinical populations are required to validate this technology.
Acknowledgements
This research was supported by a grant from the Institute of Health Sciences (No. IHS GNU-2018-02).
Notes
Conflict of Interest
No potential conflict of interest relevant to this article was reported.
Author Contribution
Conceptualization: Jeong H, Song JH. Data curation: Kim CH, HS Choi, Jeong BK. Formal analysis: Kim CH, Choi HS, Song JH. Funding acquisition: Song JH. Methodology: Kim CH, Kang KM. Project administration: Kang KM, Jeong BK, Ha IB. Writing - original draft: Kim CH, Choi HS. Writing - review & editing: Kang KM, BK Jeong, Ha IB, Song JH. Investigation: Kim CH, Kang KM, Jeong H, Ha IB. Resources: Jeong H, Song JH. Software: Jeong H. Supervision: Kang KM, Jeong BK. Validation: Kang KM.