1.

Introduction

An assessment of the mechanical properties of biological tissues is often performed to understand the constitutive relationship of natural tissues and can be helpful for improving clinical treatments. Concerning the need for strain measurements in this pursuit, recent developments of digital image correlation (DIC) have made it appealing for biomechanical studies. Indeed, there are many distinct advantages such as the simple optical arrangement, high accuracy, and large range of measurement.

DIC is an optical method that employs image registration techniques for achieving accurate two-dimensional (2-D) and three-dimensional (3-D) measurements from changes in images. This technique is most often used to measure engineering deformation including displacement and strain.^1,2 With use of the Newton–Raphson algorithm, DIC can achieve 0.01 pixel precision when a bicubic interpolation is adopted.³ The strain components are calculated from the displacement by a discrete difference of derivatives. As an optical noncontact method, DIC has been applied for determining the mechanical properties of soft and hard biological tissues.^4–7 This technique has also been used for characterizing the integrity of implants and joint replacements. For instance, loosening of the femoral component in total hip replacements as a result of cyclic loading has been evaluated using DIC.^8,9 In cemented total knee arthroplasty, the micromotion and strains at the cement-trabeculae interface were quantified to demonstrate the loss of interlock between cement and trabeculae at the tibial interface. This problem could potentially lead to increased circulation of interstitial fluid, which in turn causes fluid-induced resorption of the trabeculae.¹⁰

Metal and ceramic implants and heterotopic ossification have been reported in surgical treatments of total hip, knee, cervical, and lumbar arthroplasty. The primary goal of these treatments is to recover the basic function of the human body, which often is focused on restoring the range of motion, and reducing pain and suffering. In these treatments, a 3-D kinematic analysis is needed for evaluating the range of movement before and after the treatment.^11–14 It is customary to use typical motion tracking systems that utilize an array of active or passive markers attached to the human body to transmit infrared light to a multiple camera system. One attractive quality of this approach is that the 3-D coordinates of the markers can be accurately determined at a rate of 200 Hz. However, the system is expensive. In addition, since the markers must be attached onto the target, the existing 3-D kinematic analysis system is a contact measurement method, which is a drawback. As the existing 3-D kinematic analysis system depends strictly on the reflective markers, it is susceptible to interference of other heat sources.

This paper presents the development of a noncontact video camera system for 3-D motion tracking based on DIC. The artificial or natural surface texture can be used as markers and the corresponding 3-D coordinates are reduced based on feature point correlation matching and photogrammetry. A real-time 3-D-DIC technique, which is capable of tracking multiple measurement points simultaneously, has been developed. The experimental results show that the proposed 3-D-DIC is a robust and efficient method for measuring 3-D motion in real time.

2.

Material and Methods

2.1.

Real-Time Three-Dimensional Digital Image Correlation

DIC is a noncontact optical method that utilizes surface speckle patterns as an information carrier and cross-correlates images of the structure in “reference” and “deformed” states to quantify the surface deformation of an object. Using a pair of cameras, the 3-D motion can be measured.¹⁵ A schematic diagram for the basic principles of photogrammetric theory is shown in Fig. 1. In this approach, two digital cameras are placed at different viewing angles to capture images of the same object. In order to achieve precise measurements, each camera requires a calibration to determine the intrinsic parameters, which include the focal length, principal point, skew coefficient, distortions, and the extrinsic parameters, which include the orientation and position of the optical center.¹⁶ Generally, an object with a checkerboard pattern and well-defined geometry is used in the calibration process, which includes rotating and tilting in different angles. The image coordinates corresponding to the intersections of the checkerboard pattern are compared with the corresponding world coordinates based on Zhang’s principle.¹⁶ The relationship between the image coordinates $P_i(u_i,v_i,i=\mathrm{1,2})$ and the true 3-D coordinates of point $P(x_w,y_w,z_w)$ can be expressed using the following equation:

Eq. (1)

$$\left[\begin{array}{ccc}{a}_{11}^i-u_i{a}_{31}^i& a_{12}^i-u_i{a}_{32}^i& a_{13}^i-u_i{a}_{33}^i\\ a_{21}^i-v_i{a}_{31}^i& a_{22}^i-v_i{a}_{32}^i& a_{23}^i-v_i{a}_{33}^i\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ z{}_w\end{array}\right]=\left[\begin{array}{c}{u}_i{a}_{34}^i-a_{14}^i\\ v_i{a}_{34}^i-a_{24}^i\end{array}\right](i=1,2),$$

where ($u_1,v_1$) and ($u_2,v_2$) are the 2-D image coordinates in cameras 1 and 2, respectively. The $a_{mn}^i$ represents the projection matrix that involves the intrinsic and extrinsic parameters of the imaging system that were obtained from calibration. In Eq. (1), the world coordinates ($x_w,y_w,z_w$) are unknowns and can be calculated from four equations by means of the least square method.

Fig. 1

A schematic diagram of the photogrammetric theory for 3-D-DIC involving two cameras.

To achieve image correlation with an acceptable rate, a modified 3-D DIC algorithm has been developed. In image correlation, a subset was defined in the reference image and the corresponding coordinates are searched in the image sequence with subpixel precision. This proposed algorithm is composed of matching in the integer domain and subpixel registration. The searching process is described in this section.

The search process is initially employed in the integer domain using an improved particle swarm optimization (PSO) algorithm^17,18 to find an approximation of the initial guess, which would be used for subpixel registration. Since the radius of convergence of the inverse compositional Gauss–Newton algorithm (IC-GN) is within three pixels,¹⁹ a swarm of particles is defined that are equally spaced within three pixels. The center of the particles is located at the predefined position in the reference image. This is also called the first generation of the PSO. Upon the flying experience of the individual particles, the swarm of particles moves toward the positions with highest correlation coefficients, which is called the next generation. This iterative process continues until either the correlation coefficient is $>0.75$ or after five iterations are completed. In most cases, the correlation coefficient can reach 0.75 or higher within two generations of PSO. The image is discarded if the correlation coefficient is less than this threshold at the fifth generation of PSO. In order to further refine the initial guess, the BBDGS algorithm is performed if the correlation coefficient is $<0.9$.²⁰ The BBDGS starts with a $3\times 3\text{pixel}$ block, which includes points surrounding the result obtained from the PSO. The optimal position with the maximum correlation coefficient is identified by calculating the correlation coefficient at each point. If the optimal position is at the center, the BBDGS stops. Otherwise, the checking block is reset and the calculations are repeated until the optimal position is found at the center. The resulting coordinates in the integer domain are used as the initial guess for the following subpixel registration.

The efficient IC-GN algorithm with bicubic interpolation is adopted for fast subpixel registration.¹⁹ Parallel computing is also employed to further increase the rate of processing. As reported previously, this algorithm has been applied in fatigue testing for real-time characterization of strain at a rate of 60 Hz for a camera with sampling rate of 62.5 fps.²¹

In 3-D measurements, the motion of the object is recorded in sequences with two synchronized cameras. To achieve real-time measurement, the aforementioned correlation algorithm is conducted between image pairs and two sequential images, respectively. At the beginning, the correlation is conducted between the reference images captured from cameras 1 and 2, as shown in Fig. 1. In practice, the points of interest are manually selected in the reference image acquired via camera 1 by user interaction. A search for the image coordinates of the corresponding points in the reference image, concurrently acquired via camera 2, is conducted with the proposed correlation algorithm. Based on the stereo vision principle [Eq. (1)], the world coordinates of the POIs ($x_0,y_0,z_0$) are determined. Then image correlation is conducted on the two image sequences, which are acquired through cameras 1 and 2, respectively. A schematic diagram for the scheme of 3-D matching is shown in Fig. 2.

Fig. 2

A schematic diagram for real-time 3-D-DIC matching procedures. Images 1 and 2 are obtained from cameras 1 and 2. Additional images are continuously captured at a chosen acquisition rate as indicated by the deformed frames of increment “$i$.” First, POIs are selected in the reference image obtained from camera 1 and the corresponding positions are determined in the reference image obtained in camera 2. The initial world positions of the POIs are constructed based on the photogrammetric theory. Then the motion of the POIs is traced by determining the position of POIs in the two image sequences captured with cameras 1 and 2, respectively.

Since the image acquisition process for the two cameras is synchronized, the world coordinates of the target points ($x_i,y_i,z_i$) at any moment during movement can be estimated based on Eq. (1) after the image coordinates are determined in the two concurrently captured image pairs. The displacement of the target points ($u_i,v_i,w_i$) during loading can be calculated by subtracting the original world coordinates ($x_0,y_0,z_0$) from the current world coordinates ($x_i,y_i,z_i$). This process continues until the calculations in the two image sequences are completed.

2.3.

Experiments for Biomechanical Evaluation

Human lower legs used in this study were obtained from participating hospitals in Shanghai, China, according to approved protocols issued by the Institutional Review Board of Shanghai University. The samples were frozen at $-18°\mathrm{C}$ right after extraction, and at 5°C 24 h before the experiments.

2.3.1.

Experiments involving foot arch

There is a functional relationship between the structure of the arch of the foot and the biomechanics of the lower leg. Since the arch provides an elastic, springy connection between the forefoot and the hind foot, a majority of the forces incurred during weight bearing of the foot can be dissipated before the force reaches the long bones of the leg and thigh. Fallen arch is a postural deformity in which the arches of the foot collapse, which causes the entire sole of the foot to have contact with the ground. In this study, the movement of the arches was evaluated when a vertical load was applied to the foot before and after subtalar joint arthrodesis was conducted. A schematic description of the experimental arrangement is shown in Fig. 3(a). The near end of the tibia was fixed in the upper grip and the crosshead of the test machine was carefully adjusted so that the foot was placed on an aluminum plate. A vertical load was applied to the tibia at a rate of $10\mathrm{N}/\mathrm{s}$ until reaching up to 700 N.¹¹

Fig. 3

Schematic diagrams for (a) the experimental setup used for the foot arch measurement and (b) the user-defined coordinate system.

The stereo camera system was used to monitor movement of the arches while loading. In order to measure the invisible talus and metatarsals, surgical pins were firmly inserted into the bones to ensure the white heads of the pins move with the bones. Prior to the experiments, the intrinsic parameters and extrinsic parameters of the stereo camera system were calibrated with a planar checkerboard based on Zhang’s calibration method.¹⁶ The world coordinate system was defined with the $XOY$-plane parallel to the aluminum plate underneath the foot, as indicated in Fig. 3(b). The $z$-axis was defined as the normal to the calibration plate and directed toward the cameras. Images were captured in pairs simultaneously with the stereo imaging system. The 3-D coordinates of the markers relative to talus and metatarsals were determined while loading.

2.3.2.

Experiments involving distal tibiofibular syndesmosis

The syndesmosis locates between the triangular fibular notch of the lateral surface of the distal tibia and the medial convex surface of the distal fibula. Damage to normal kinematics of the tibiofibular joint may cause recurrent ankle sprains and even chronic ankle instability.²² Therefore, the distal tibiofibular syndesmosis rupture must be recognized and treated to prevent late complications. In general, operative fixation usually involves the insertion of one or two metallic diastasis screws. The evaluation of the relative movement between tibia and fibulacan reflect the effect of the treatment of distal tibiofibular syndesmosis.^23,24 In this experiment, the relative movement between the tibia and fibula were evaluated with the 3-D motion system while the cadaver foot sample was stressed before and after surgery. The experimental arrangement was shown in Fig. 4. The imaging system was calibrated before the test and the corresponding right-hand world coordinate system was defined such that the $x-y$-plane coincided with the sagittal plane and the $y-z$-plane coincided with the coronal plane, as shown in Fig. 4. Two surgical pins were inserted into the tibia and fibula, respectively. During the experiments, the foot was subjected to a monotonic rotational torque up to 5 Nm at a speed of 5 deg per min.²⁴ The movements of the four pins were tracked in real time by the proposed DIC technique.

Fig. 4

Experimental arrangement used for measuring the relative motion between the tibia and fibula.

3.

Results

In conducting the validation experiments, the accuracy of the real-time 3-D-DIC system was evaluated by performing the translation tests. The displacements of the four POIs were averaged in the $x$- and $z$-directions and the corresponding trajectories were displayed in Fig. 5. The displacement trajectories of in-plane and out-of-plane both show good agreement with the preset values. In order to give a quantitative error analysis, the displacements in Fig. 5 at each step were averaged and compared with preset values as listed in Table 1. It was found that the maximum relative error of the in-plane ($x$-axis) displacement was $-0.25\%$. Similarly, the maximum relative error of the out-of-plane ($z$-axis) was 1.17%. During the measurements, the displacement was output at a rate of 32 Hz from the imaging system.

Fig. 5

Average motion of the selected markers on the translation stage. (a) in-plane ($x$-direction) and (b) out-of-plane ($z$-direction).

Table 1.

Error estimation of the proposed real-time 3-D-DIC system.

In the foot arch experiments, the movement of the five metatarsal bones and the talus were measured as a result of loading before and after treatment. The 3-D displacements were recorded in real time using the computer. Results for the displacement trajectories with loading are shown in Fig. 6. Specifically, the displacement trajectories for the flatfoot deformation in the $x$-, $y$-, and $z$-directions are shown in Figs. 6(a), 6(c), and 6(e), respectively. Similarly, the corresponding displacements in the three directions resulting from loading after the treatment are shown in Figs. 6(b), 6(d), and 6(f), respectively. As ligaments are sheets or bundles of collagen fibers that form a connection between two bones, ligaments act as protective backups for the bones. As subtalar joint arthrodesis involves the fusion of two bones to restore the position of the calcaneus under the talus, which recreates the arch, this fusion can effectively reduce the characteristic “splaying” of the foot in pes planus in the $xy$-plane. In the loading situation, the displacements of the foot arches were decreased 5 to 20 mm in all directions after treatment, as shown in Fig. 6. The degree of decrease observed in the $x$- and $y$-directions exceeds that in the $z$-direction.

Fig. 6

The 3-D displacement trajectories for the foot arches in the individual metatarsals and the talus. Shown individually are the (a) displacement in $x$-direction before treatment, (b) displacement in $x$-direction after treatment, (c) displacement in $y$-direction before treatment, (d) displacement in $y$-direction after treatment, (e) displacement in $z$-direction before treatment, and (f) displacement in $z$-direction after treatment.

In the distal tibiofibular syndesmosis experiments, four markers were monitored and the relative displacement components in the three directions between the tibia and fibula were calculated. The variation of the average relative displacement with increase of torque was evaluated before and after treatment. As shown in Figs. 7(a) and 7(b), after the treatment for the broken distal tibiofibular syndesmosis, 3-D displacements for the measurement were decreased. Specially, the relative displacement decreased significantly in the $z$-direction. This is consistent with the function of the distal tibiofibular syndesmosis, which limits tibia and fibula movement in the coronal plane.

Fig. 7

The 3-D relative displacement between the tibia and fibula that occurred while loading. (a) Before treatment and (b) after treatment.

4.

Discussion

The development and application of noncontact methods for precise displacement measurement technique is of substantial interest for engineers.^25–28 For biomedical applications, there are often additional technical challenges to overcome that must be considered. This study has illustrated the use of DIC as an optical approach for accurately quantifying the movement of bones in the treatment of pes planus and distal tibiofibular syndesmosis injury. To enable real-time measurement, a fast DIC matching algorithm for 3-D measurement was employed. As shown in Figs. 6 and 7, the measured displacement trajectories of the foot bones in the three directions were presented as a function of loading. Using the displacement data, it is possible to extract key mechanical parameters that help to describe the effectiveness of treatment. For example, the vertical displacements, the relative displacements between bone/bone and bone/joint, the active degree of movement, and so on can be easily accessed. These measures can facilitate greater understanding of the interactions between foot bones and the success of surgical treatment.^29–31 In order to keep the specimens intact during the experiments, the soft tissue on top of the bones was not removed. As such, surgical pins were inserted into the bones and were used as the markers for the imaging system. However, as DIC can use the surface texture as the information carrier, it can be applied without markers if the bone can be observed. From this point of view, DIC is still regarded as a noncontact measurement technique.

In the traditional approach to DIC, the motion information is extracted by postprocessing the sequential images captured during testing. Clearly that approach was time-consuming, not automated, and also inapplicable for dynamic measurement. This is also a driving force for the development of real-time measurements. In these previous studies, bilinear³² or biquadratic interpolation³³ was used in the correlation process to save the computational time and a 10-Hz measurement rate was achieved.³² This could potentially reduce precision in the measurement without the use of bicubic interpolation.³ In contrast, the proposed DIC approach can achieve accurate strain measurement at 60 Hz in a 2-D application.²¹ In this study, a robust and automatic strategy for real-time measurement of 3-D motion has been presented. As shown in Table 1, a measurement accuracy of $-0.25\%$ in the $x$-direction (in-plane) and 1.17% in the $z$-direction (out-of-plane) was achieved. In the biomechanical experiments, the displacements of six bones in the foot were tracked in real time with a rate of 16 fps. For the proposed method, the tracking speed mainly depends on the camera frame rate, computer performance, and the measured number of points. It is possible to further improve the acquisition rate by upgrading the hardware system. For example, the displacement/strain measurement could be instantly achieved if high-speed cameras were used with greater framing rates.

When compared with conventional 3-D-DIC, the proposed method has three distinct advantages. First, the proposed method for 3-D-DIC combined the PSO algorithm with the BBGDS algorithm, which enables rapid convergence to the global optimum integer pixel position. Second, the IC-GN subpixel iteration with bicubic interpolation was introduced, and all correlation matching was performed with the reference frames. That strategy requires that the Hessian matrix is calculated only once in each sequence, which significantly reduces the computational time required without any loss of accuracy in comparison to conventional DIC. Third, the proposed method employed multicore parallel computing technology, which enabled correlation of multiple points to be achieved simultaneously without noninterference. The experiments verified that the proposed 3-D-DIC method is precise and effective for real-time displacement/stain measurement in engineering and biomedical applications.

It is important to comment on the potential use of the proposed technique with respect to existing technology. The infrared tracer technique has been widely applied for biomechanical research as a method for noninvasive motion analysis.^34–37 This technique employs multiple high-speed or ultra high-speed infrared cameras to monitor reflective markers placed at specific locations for measurement of 3-D trajectories. Although this technique possesses the advantage of nearly instantaneous monitoring speed, it relies on the frame rate of the infrared cameras, which is quite costly. Furthermore, the tracer technique has measurement accuracy that is limited to the millimeter range. An X-ray imaging technique is also an option for determining 3-D motion in biomedical applications,³⁸ whereas the protection for radiation makes it difficult in common laboratory studies. In comparison, the proposed real-time 3-D-DIC technique could be performed using affordable CCD cameras for real-time motion measurement and has an accuracy within the submicrometer range. The proposed technique can also achieve a high-tracking speed if high-speed CCD cameras and a multicore computer are employed.

5.

Conclusion

A real-time 3-D-DIC method for dynamic displacement/stain measurement has been described and applied in this paper. The proposed system is capable of measuring the displacement of six points simultaneously with a monitoring speed of about 16 fps. The approach utilizes a combination of a fast 2-D-DIC search strategy with an immediate 3-D matching scheme. The accuracy and effectiveness of the presented technique were proven through a translation test using a motorized precision translation stage and biomechanics applications involving the motion of foot bones. From the experimental results obtained, it can be concluded that the established system can offer a solution for dynamic motion measurement in 3-D space. It could facilitate performing fatigue/cyclic tests on biological materials and facilitate achieving a better understanding of the mechanical behavior of biological materials. The system has potential for improvements in displacement/strain measuring speed if the CCD cameras and the computer used possess higher speeds than the current ones.