3.1

Patient’s limb motion

In the case of limbs, it is suggested to map the movement on the image using a time series representing time-variable coordinates of the areas of interest or the value of their functions. In order to simplify this task, markers of the same color were fixed to the patient’s limb during the recording, so that it was possible to apply segmentation through color image binarization in the primary color channels, and cluster analysis. The values of the T_R, T_G and T_B thresholds in each of the I_R, I_G and I_B channels were adapted to the color of the used marker. Therefore, the I_BIN output image after binarization for the red marker used in the finger tapping test [9] was determined as follows:

The aforementioned binarization shall ensure obtaining points within a plane, the location of which corresponds to the fingertips and shall enable determining the coordinates for their centers using the cluster method. However, the binarization result shown in Fig. 1A indicates the presence of numerous artefacts associated with the occurrence of colors similar to the marker color within the image.

Figure 1.

Idea of image segmentation for the purposes of separating fingertip areas: A – binarization result, B – erroneous detection of fingertips using the k-mean method based on image A, C – median filtration of a binary image, D – correct detection of finger tips using the k-mean method based on image C.

Grouping algorithm for k-means for a two-class problem (2 fingers) provides in such a case false coordinate values for detected clusters – Fig. 1B. The nature of the occurring artefacts is similar to the “salt & pepper” type noise, which is why it can be easily eliminated using median filtration. Fig. 1C shows the result of such filtration using a size-20 square mask filter, along with the result of k-mean clusterization in the form of two-cluster centers. The cluster centers projected on the photo in Fig. 1D correspond to fingertips.

Fig. 2 shows subsequent video stream frames with the effect of applied aforementioned segmentation for tracking patient fingertips for the purposes of the finger tapping test (point III.4 of UPDRS). Using x₁ and x₂ to mark the coordinates of both cluster centers, it is possible to determine the Euclidean distances between them based on the sequence of successive video recording frames:

Figure 2.

Limb motion tracking illustration based on finger tapping test example (circles in green are coordinates of points returned by the developed algorithm).

and thus, obtain a time series representing relative limb motion.

Fig. 3 shows time waveforms – obtained this way – for the right and left hand of the patient diagnosed with the classic form of Parkinson’s disease with left-sided symptoms, evaluated in terms of the motor component at 24 UPDRS points.

Figure 3.

Illustration of time waveforms and signal spectra of distances between the fingers in point III.4 of a clinical study for the right hand (above) and left hand (below) in a PD patient.

The illustration of time waveforms enables comparing the range of motion for both hands and evaluate the motion regularity disturbances, included in point III.4 of UPDRS. At the same time, simple processing based on, e.g., spectral analysis (Fig. 3) enables to quantitatively express the mean number of finger taps. In the case discussed in Fig. 3 this number was ca. 2.5 per second for the right hand and ca. 1.5 per second for the left. For comparison, Fig. 4 shows a time waveform and the spectrum of a signal obtained in the case of a healthy person. We can notice better regularity and a greater motion speed, which can be assessed at ca. 3.5 taps per second.

Figure 4.

Time waveform and signal spectrum for a distance between the right-hand fingers in a healthy person.

3.2

Drawing created by a patient

The motivation behind commencing research in the field of analyzing artwork created by a patient is a symptom often brought up by the patients themselves, which results from the slowdown, tremor, as well as muscle stiffness in the hand, namely, micrographics [10]. In the course of the implemented pilot research, the patients were first asked to write the sentence “Dzisiaj jest ładna pogoda” (eng. The weather is nice today) five times, one by one. There are naturally ongoing changes during writing, the quantitative assessment of which can find a diagnostic application. These include pressure of the pen on the piece of paper, reducing letters, shifting of the successive sentences relative to the previous ones, appearance of larger spacing between the words and time intervals between subsequent words or sentences resulting from the need to longer think about the following activity, which might indicate the deterioration of cognitive abilities. The second task involved drawing an Archimedean spiral, successively using the right and then the left hand, and then following a contour of a circle, once again, with the right and then the left hand. Both these activities can reveal not only differences associated with right- or left-handedness, but also with the impact of the disease on one side of the body only.

The properties of the used graphic tablet provide a potential to quantitatively describe the performed activities and the changes therein. The completed recording contain such information as the sample number, recording time in [ms], position of the pen on the X and Y axes of the tablet, pressure in arbitrary units, pen inclination angle relative to the tablet surface in [°] and the angle included between the Y axis of the table and the pen, the so-called azimuth in [°]. Table 1 shows a fragment of collected sample data.

Table 1.

Examples of a data set recorded by the used graphic tablet.

The resulting data are in tabulated form, which requires preliminary conversion to a simpler form facilitating their interpretation.

3.2.1

Drawing reproduction

Using a computer screen to recreate an image created by a patient on a piece of paper placed on the surface of the tablet requires, first of all, to remove all redundant samples collected when the pen was raised above the tablet surface. As a result, the drawing will only contain the exact trace, which has been drawn on the piece of paper in reality. A finished image is created as a graph of relationships between the value of coordinates X and Y, using points of appropriate thickness. The ultimate effect is a black image on a white background. Thus reproduced, e.g., Archimedean spiral images drawn with the right and left hands are shown in Fig. 5.

Figure 5.

Reproduced spiral drawings created by the same patient, successively, with the right and left hands.

The analysis of the obtained images provides certain interpretation-related possibilities, but does not utilize information about their creation. Such additional opportunities in terms of interpretation appear thanks to an analysis of the time waveforms corresponding to the image created by the patient.

3.2.2

Time series analysis

The data obtained from the tablet, such as stylus position, pressure, angle of inclination or azimuth can be presented in form of time series illustrating the kinematics and dynamics of image creation. Fig. 6 shows the time waveforms based on data from the tablet, which correspond to the Archimedean spirals created by one person (Fig. 5). The graphs in Fig. 6A showing changes of the stylus position in the course of drawing an Archimedean spiral using the right hand can be used to determine its creation time (ca. 4s). The presented signals have a characteristic waveform, similar with its shape to the sine and cosine function with increasing amplitude, which is an effect of drawing a circle of increasing radius. Additional information on the manner of spiral drawing by a patient can be obtained by analyzing the recording dynamics, which is a change of the pen pressure on the piece of paper. In order to illustrate the difference in the pressure during the same task conducted with the right and left hands, their corresponding pressure time waveforms are projected on each other in Fig. 6B.

Figure 6.

Illustration of the kinematics of creating the Archimedean spiral using the right hand (A) and a comparison of the dynamics of both hands (B).

When analyzing the presented waveform, it can be observed that the time for drawing the spiral with the left hand was longer than in the case of the right one. Despite the fact that the nature of both formed curves is similar, the averaged pen pressure when drawing with the right hand is higher than for the left hand, whereas in both cases, in grows along with creating successive spiral laps. Quantitative evaluation of the obtained waveforms is possible using numerical parameters, such as maximum pressure, average pressure and pressure standard deviation. Table 2 lists the values obtained for the case of signals shows in Fig. 6B.

Table 2.

Selected time waveform parameters of the pressure when drawing the Archimedean spirals.

	right hand	left hand
maximum pressure [arb. unit]	28918	27730
average pressure [arb. unit]	26390	22215
std. deviation [arb. unit]	2229	5858

Apart from pressure, the difference between drawing with the right and left hands can also be observed when analyzing the time series of the pen inclination angle relative to the tablet surface, which is shown in Fig. 7.

Figure 7.

Change of the pen inclination angle relative to the tablet surface when drawing the spiral with the right and left hands.

Pen holding manner, expressed by an angle included between the pen and the tablet surface, is almost constant in the case of the right hand. The difference between the maximum and minimum angle is only 8°, and the standard deviation is 1.39°. In the case of the left hand, the difference between the maximum and minimum inclination angles is 22°, while the standard deviation is 5.71°. The average inclination angle when hold the pen with the right hand was equal to 48.9°, and lower than in the case of the left hand, with that value amounting to 71.4°. These results show that writing with the left hand was more difficult for this patient, and the manner of holding the pen was more chaotic than for the right hand.

The time series analysis can also be conducted for data representing handwriting samples. In the case of the handwriting sample shown in Fig. 8, by a patient with genetic Parkinson’s disease, already the very analysis of the change of stylus position over time, provides quantitative information regarding the characteristic tendency ongoing in the course of writing.

Figure 8.

Handwriting sample for a patient with genetic Parkinson’s disease.

The graph in Fig. 9 indicates, that the coordinate signals for the stylus position when writing sentences are in the form of a sawtooth. The characteristic vertices determine locations where sentences end. It can be seen that in the case of coordinate X these vertices are lower, which means that the sentences were shorter.

Figure 9.

Stylus position change when writing 5 sentences visible in Figure 8.

The length of written sentences can be of diagnostic significance. Lengths of each of the 5 sentences created by all patients were determined based on the set of data recorded within the pilot research. The list of the basic descriptive statistics for each of them is contained in Table 3. It can be noticed that, with the exception of a single case, the standard deviations of the length of sentences written by sick persons were larger than the standard deviations of the length of sentences written by healthy persons.

Table 3.

List of mean values and standard deviations of a parameter described by sentence length.

	PD	healthy
L̅ [cm]	14.03	13.81	18.33	11.74	10.00	15.25	13.62	9.77	12.56	13.21	10.50	16.33	12.21	10.22
σL [cm]	1.02	1.09	1.05	0.73	0.54	0.50	0.55	0.59	0.19	0.36	0.07	0.41	0.42	0.12
	0.70	0.27

3.3

Patient’s facial expressions

Masked face, otherwise called hypomimia or mimic diminution is a symptom of the Parkinson’s disease, which involves the limited ability of a patient to react spontaneously (e.g. eyelid blinking) and express facial emotions due to a deteriorating of the activity speed and coordination of face muscles. The techniques of processing facial images for the purposes of biometrics developed over the recent years provide potential possibilities for assessing hypomimia, however due to the specificity of the subject and the conditions of the conducted tests, it is not a simple task. First of all, due to the possible movements of a patient’s head in front of a camera, tools suitable for this purpose should be sought after among processing methods, which are independent of subject movements and rotations. The very methodology is also important, and it should minimize the strong impact of individual features on the recorded material. Therefore, this research paper suggests using the detection of characteristic points, between which constant relationships under a fixed facial expression will undergo, regardless of the rotation or movement. With the changing facial expression, these relationships should be disturbed, according to the type of the emotion and the ability of a patient to express them. Therefore potentially, such points are the tip of the nose, eye corners, points on eyebrow and eye arches, mouth contour points, etc.

Contemporary techniques in the field of searching for characteristic facial features are based on supervised machine learning methods, in which a large learning set coming from manually entered points is a base to create models for detecting points within a tested image [13-14]. The algorithm utilized for the purposes of this paper is based on deep learning techniques (convolutional neural network), the development of which was based on 10 000 learning patterns [15]. The motivation behind its application was the fact that the data for learning included facial images with highly diverse pose, partially covered and representing variable expressions. The algorithm detects 68 points but requires prior indication of an area of interest in the form of a framed facial image, which is relatively easy to obtain using the classic Viola-Jones algorithm. Fig. 10 shows the results of combining both of the aforementioned approaches to processing an image recorded during the pilot studies.

Figure 10.

Illustration of face detection within an image using the Viola-Jones algorithm and the detection of characteristic point.

3.3.1

Quantitative emotion expression evaluation

In order to study the quantitative evaluation of hypomimia, the data recording procedure involved the application of a certain form of coercion, which involved asking a patient to express predetermined emotions using his/her face, namely, asking to show how he/she looks when happy, sad, surprised, etc. Fig. 11 shows the results of the preliminary image processing in the form of the distribution of determined characteristic points for a PD and healthy person, whereas for both cases, image A is the reference image, corresponding to a neutral facial expression.

Figure 11.

Distributions of characteristic points on the face of a patient with Parkinson’s disease (top row) and a healthy patient (bottom row) when attempting to express the following facial emotions: A – neutral facial expression, B – joy, C – sadness, D – surprise, E – anger.

A quantitative method for determining the facial expression changes entailing various emotions can be calculating the distance D between appropriate points i and j of a given X distribution as per relationship (2) and relating them, as a percentage, to the distances occurring between the points of a phase with a neutral expression (on the A distribution in Fig. 11). Therefore, the value of the numerical parameter p will be determined from the relationship:

where D_X(i, j) is the distance between points numbered i and j, and X adopts letter values depending on the expressed emotion, as shown in Fig. 11. The adopted numeration of characteristic points within a face image is shown in Fig. 12.

Figure 12.

Numeration of detected characteristic spots.

Table 4 shows a list of the p parameter values for 10 exemplary pairs of points describing an emotion of surprise expressed by a face of a sick and healthy person. It should be noted that in each case of the selected points, the change of a distance due to a presented emotion was higher for a healthy person.

Table 4.

Percentage changes in the facial expression when expressing surprise (emotion X=D)

No.	points in Fig. 12	parameter p
i	j	PD	healthy
1	31	20	-1.6	-7.0
2	31	25	-1.4	-3.7
3	31	58	0.2	3.6
4	31	59	0.4	3.3
5	38	42	3.0	-20.9
6	44	48	3.0	-20.3
7	49	55	0.9	25.7
8	54	59	0.8	40.8
9	24	48	-2.1	-10.7
10	21	41	-2.7	-12.6

3.3.2

Eyelid blinking frequency

The eyelid blinking frequency evaluation included in point III.2 of UPDRS can be determined within the same image processing methodology. For this purpose, it is sufficient to monitor, by calculation, the distances between the points, which detect eye edges and to count the number of events of such distances declining due to their closure during the recording. Fig. 13 shows a graph illustrating the determined distances for subsequent video stream frames in the case of the studied persons blinking the eyes.

Figure 13.

Eyelid blinking detection: photographs below – subsequent video stream frames – corresponding to the frames of the distance between points 38 and 42 (right eye) and 44 and 48 (left eye).