Engineering Trustworthy Intelligent Software Systems

Mots-clefs :

• intelligent systems, artificial intelligence, trustworthy AI, privacy, security, explainable AI, edge AI, legislation, fairness, bias, ethics, transparency and traceability, machine learning, software engineering, system engineering, systems of systems, DevOps, continuous deployment, smart devices, Machine Learning, AI Systems, Trust, Compliance, Ethics, Explainability, Audibility, Reliability, Privacy, Security, Software Engineering / ML-Ops / AI-Ops, intelligent systems, artificial intelligence, trustworthy AI, privacy, security, explainable AI, edge AI, legislation, fairness, bias, ethics, transparency and traceability, machine learning, software engineering, system engineering, systems of systems, DevOps, continuous deployment, smart devices.
• (ajout 22/07) CTI (Cyber Threat Intelligence) – Renseignement – espionnage – prédiction – Résilience des systèmes

Organisations pertinentes :

• SEMLA, Mila, Polytechnique Montréal, Université de Montréal, Concordia University, University Ottawa, Queen’s University, York University, University Alberta, NII, Propulsion Québec, ENCQOR, FRQNT, Prompt, InnovÉÉ, CRSNG, BMW Group, Bombardier, SAP, Ericsson, IBM, Banque Nationale, Thales, CISCO, TDBank, BMO, Airudi, Radio-Canada, Airudi, MoovAI, HAXIO Machine Vision and Robotics, Banque Nationale (BNC), IBM, Thales, Bombardier, SAP, Mitacs, CRSNG.
• (Ajout 22/07) Les organismes étatiques et les associations d’entreprises.

Personnes pertinentes suggérées durant la consultation :

Les noms suivants ont été proposés par la communauté et les personnes mentionnées ci-dessous ont accepté d’afficher publiquement leur nom. Notez cependant que tous les noms des professeur.e.s (qu’ils soient affichés publiquement ou non sur notre site web) seront transmis au comité conseil pour l’étape d’identification et de sélection des thèmes stratégiques. Notez également que les personnes identifiées durant l’étape de consultation n’ont pas la garantie de recevoir une partie du financement. Cette étape sert avant tout à présenter un panorama du domaine, incluant les personnes pertinentes et non à monter des équipes pour les programmes-cadres.

• Giuliano Antoniol
• Foutse Khomh
• Mohammad Hamdaqa
• Michalis Famelis
• Marios Fokaefs
• Ettore Merlo

To achieve the goal of engineering trustworthy intelligent software systems, we divide our work into four work packages (WPs) based on the research questions listed in the previous section.

Define quality attributes for ISS and design quantitative metrics for evaluation of the quality attributes, with special focus on these new system attributes that emerge from integrating ML systems in the software design process (e.g., software ethicality, privacy, fairness). We will start from the following list of attributes:

• Correctness: The main factor for trustworthiness of an intelligent software system is its correctness. For ML-powered intelligent software systems, correctness measures the probability that the ML system under test ‘gets things right’.  That directly depends on the quality of the training data, the features extracted from the data for training, the characteristics of the ML models trained, and the correctness of the implementation, and the interactions between the ML components and other software components.
• Robustness and security: ML models can be attacked and manipulated for malicious purposes (e.g., unauthorized access to resources) or trigger accidents (e.g., image segmentation error leading to incorrect diagnosis). Sensors and measurement devices may also be subject to noise and ML models must be robust to noisy inputs. In addition, the input data and the operations environment of ISS tend to be continuously evolving, thus a trustworthy ISS needs to preserve its performance under such evolution.
• Explainability: Decisions made by ML-powered systems should be explainable, otherwise the systems can not be certified neither will it gain the trust of its stakeholders. Unfortunately, the most advanced and accurate ML techniques (such as Deep Learning (DL)) are also the least interpretable ones.
• Fairness: ML models should not be biased towards certain subgroups; for example, they should not discriminate on the grounds of sex, gender, age, ethnic or racial status. Testing for fairness is essential and challenging. Research shows that even when fairness is a design goal, developers can easily introduce discrimination in software.
• Privacy: The confidentiality and privacy of user’s data are among the most important challenges in applying machine learning in critical domains such as healthcare, e.g., in data acquisition, reporting, and potential re-identification of patient data.
• Ethicality: How can we translate high-level principles like “trustworthy,” “fairness” and “explainable” into clear ethical engineering requirements? How can we ensure appropriate levels of trust in an ML-powered system? It is critical to develop techniques and tools allowing for the audit of these systems.
• Transparency: To gain trustworthiness, the decision making process of an ISS should be transparent to its stakeholders. The transparency includes traceability of AI decisions to the data source, and model, and the business logic built on them. Such transparency also requires that the ISS be monitorable.
• Efficiency (speed, size, energy consumption): For cloud-based ISS, efficiency is usually not a major concern as it can be mitigated by increasing the capacity of the hardware running the ISS. However, as the computing power and storage capacity of edge devices is limited, ISS deployed on edge devices ISS needs to be efficient in terms of speed, energy consumption and size.
• Availability: For cloud-based ISS, the services are usually provided by a number of instances (with duplications) at the same time. Failure of a single instance usually does not cause the failure of the provided services. However, in an edge environment, the services are usually deployed on single devices, thus ensuring the availability of such services becomes particularly challenging.

Design architectures for trustworthy ISS based on the quality attributes defined in WP1.

We will define a set of design patterns and best practices that focus on the newly identified attributes of trustworthiness.

We will define components, their interactions and constraints to support the trustworthiness quality attributes and then use the defined metrics from WP1 to quantitatively evaluate different architectures and optimize them.

We will define a reference architecture (composing components and their interactions) for trustworthy ISS based on the results of evaluating different architectures.

Devise new testing and verification techniques for trustworthiness.

We will devise software testing mechanisms that can verify correctness, robustness, security, privacy, and fairness/bias of ISS. Key underlying techniques will include evolutionary computation and ML to optimize search toward specific cases (data inputs) where the prediction results violate requirements.

We will provide explainability mechanisms (based on model inference, program comprehension, and debugging techniques) that can shed light on how and why predictions are made, under which conditions they can be trusted, both in the learning phase as well as in production.

We will provide ML model adaptation approaches that can diagnose the root-causes of poor predictions and will provide automated techniques to repair models, for example by synthesising new data to retrain models.

Build an automated pipeline to deploy, operate, monitor, and continuously improve ISS running in different deployment environments (e.g., centralized servers and edge devices).

We will develop reliable methodologies and techniques that deploy ISS in different platforms, including optimizing/compressing ML systems for running on edge devices and ensuring the trustworthiness of the ML systems are not impaired when they are optimized/compressed.

We will develop approaches for profiling and scheduling the digital twin services (i.e., services provided by both the centralized servers and edge devices), to plan the workload allocation that takes into consideration all the quality attributes listed above (e.g., data security and privacy) and achieves the highest level of system performance.

We will develop monitoring techniques that ensure high observability and transparency of ISS while imposing minimum overhead on the monitored systems (in particular, those deployed on edge devices) and minimum data communication between edge devices and centralized servers. Such monitoring would ensure the healthiness of the code, data, and model, and provide feedback/alerts in cases of failures, anomalies, or data/model drifts.

We will develop methods and techniques to continuously repair, maintain, and improve the ISS based on the feedback obtained from monitoring, to improve its reliability and robustness. In particular, when data/model shifts are detected, the techniques will automatically and continuously update the model with minimum footprint on the running systems. When failures or anomalies are detected, the techniques will automatically repair or mitigate such failures/anomalies or provide instant alerts or suggestions to the operators.

This project can have wide range industrial impact. Other than all the organizations and institutes mentioned in the proposal, all my current collaborators will be interested in this project.

• AI and Cloud Provides: IBM, Microsoft, Google, Facebook, etc.
• Autonomous Systems Providers: (e.g., Self Driving Cars Providers, Robotics, etc.)
• Intelligent Decision Making Providers for services such as insurance technology, banking, and health informatics, etc.
• Governments (Government of Canada)
• All the software development community

ML systems are and will be pervasive. We need ways to ensure the robustness, resiliency and trustworthiness of AI enabled systems. Hurles are many just consider the 2015 NIPS Google seminal paper.