Advancements in science and technology has led to development of complex, safetycritical, and capital-intensive systems and processes. To ensure optimal and safe operations of these technical systems and processes requires the development of quality monitoring and supervision approaches. Data obtained during monitoring and supervision can be evaluated for state-of-health diagnosis. The selection of an evaluation approach for an application may require performance assessment. In this thesis a new evaluation approach based on the Probability of Detection (POD) reliability measure is developed. The POD is a probabilistic method to quantify the reliability of a procedure taking into account statistical variability of sensor and measurements properties. The a90/95-criteria representing probability of 90 % detection at a confidence level of 95 % is examined to the measurements and related outcomes. The generalizable capability of the developed approach is demonstrated in multidisciplinary fields: vibration-based Structural Health Monitoring (SHM), Machine Learning (ML), and Computer Vision (CV) classification approaches.

Structural Health Monitoring systems are based on suitable sensor techniques allowing online and offline supervision of technical systems. The quantification of sensors/measurement devices is a key issue for qualifying their effectiveness and efficiency. For vibration-based supervision approaches the POD approach can not be applied similarly. This results mainly from the complexity of the dynamical behavior of systems monitored in relation to faults, sensors position (observability), and the related feature extraction or monitoring task. In this research, POD evaluation of vibration-based fault detection of elastic mechanical structures to be monitored is developed. Beside a principal discussion of the problem serving as introduction, an example using different sensor types in combination with mechanical modifications of an elastic beam is presented. Based on the analysis of a suitably chosen feature and dependent on the mechanical modes considered, the efficiency and deficiency of the different combinations are shown. Based on the proposed approach, a new insight to the usefulness of different sensor types and fault position combination becomes possible. To improve the detection quality, suitable assumptions in combination with sensor/information fusion are applied to feature-based analysis as detection task using vibration measurements. Dependent on noise analysis, a trade-off between flaw size detection and probability of falsely characterizing a fault with a90/95 reliability level can be attained.

Machine learning approaches provide the ability to automatically learn and improve from experiences. These ML approaches are functional tools that allow classification abilities. In this sense also the machine learning-based recognition of complex driving situations for automated vehicles as well as for human drivers is of increasing interest. In this field based on technical sensors a complex scenario and related dynamical changes have to be interpreted and classified. Evaluation of these classifiers plays a crucial role in their selection for a specific task. In this thesis a new approach is established permitting a new evaluation measure to ML approaches used as classifiers. Current methods utilize measures like the receiver operating characteristic (ROC) providing visually and graphically the performance of classifiers. Beside the ratio of detection rate and false alarm rate (combined as ROC), other properties related to process parameters are not integrated in the evaluation process. In this research, a new evaluation method based on the Probability of detection (POD) reliability measure is developed integrating and discussing the effect of further process parameters on the classification results. As an example for illustration the comparison of classifiers applied to driver prediction behavior is used. The temporal distance of decisions moments to the instant of decision itself is used as process parameter for the classification process. The proposed approach is used to compare the suitability of Artificial Neural Networks, Hidden Markov Models, Random Forest, Support Vector Machines, and improved versions of these classifiers with respect to the reliability of related classification of upcoming events prior to the real ones, here: human decisions based on the visual impression of changes within the environment. Consequently, based on the POD-related evaluation the different classifiers can be clearly distinguished with respect to their ability to predict the correct behavior as a function of time prior to the event itself.

In Computer Vision systems, image classification and detection is an important task. Deep Convolutional Neural Network (CNN) is a suitable classifier for large-scale image classification. It is well-known that image classification performance strongly depends on problem characteristics and vary with process parameters. This work focuses on explicit and accurate evaluation of CNN classifiers in image detection. The POD approach is implemented allowing the evaluation of further image parameters affecting the classification results and subsequently the comparison of classifiers related to the same parameter depending on classification task. The proposed evaluation method is implemented on a number of classifiers and comparisons made on parameters with the best detection capabilities.