Authored by Terry Leslie, Vice President of External Research & Development
Interpretability of machine learning models is one of the hottest topics in computer science today, with over 20,000 articles published in the last five years. Why is this such a hot topic and what’s going on?
First, let’s understand what interpretability of a model means. In the machine learning field, researchers are defining interpretability as the ability to explain the model to a human or to present the results or decisions being made by the model in understandable terms.
With the rapid growth of machine learning in our society, there is a reality and fear that humans cannot fully understand machine-made decisions that affect them or their families in areas such as healthcare, finance or personal safety. Machine learning models are considered black boxes that people cannot see inside but have real, tangible effects on their lives. For example, paying more for auto insurance due to an autonomous vehicle accident, or worse, being denied access to a drug or admittance to a hospital drive major motivations for knowing why that decision was made. That answer may not be attainable with the “opaque” machine learning models being employed today.
The more disconnected from human understanding these systems become, the more challenging it becomes for companies that rely on these systems for doing business. Customer health and safety, maintaining customer satisfaction, complying with government regulations, and meeting legal obligations are some of the challenges companies face when making broad usage of black box machine learning models. At the extreme, AI systems making unexplainable decisions will prompt fears of SkyNet and the Terminator, whether founded or unfounded.
Another reason for building interpretable models is the need to continuously improve the model. How can a model be improved without understanding the decisions being made by the model? How can a model be fully tested without understanding the relationship between the inputs and the outputs of the model? The most advanced and complex models in use today are neural network models. These models are optimized by performing backpropagation and error minimization, where coefficients are tuned in the various network layers to minimize error in testing datasets. For most deep-learning models today the connection of input variation to output decisions does not exist in understandable human terms.
There are many varieties of machine learning neural network based models ranging from convolutional to recurrent and beyond. These networks are complex, contain many hyperparameters, rely on multiple layers of mathematical computation and transformation, need huge annotated training and testing datasets and require highly trained personnel for design, setup, and optimization. Even the experts creating the model cannot fully understand the rationale behind the decision being made by the model. The optimization process has sometimes been described as seen in the photo on right.
Are there ways to make models more easily understood by humans? As noted, there is a lot of research underway on developing models that are both interpretable and accurate. In most cases a tradeoff must be made between accuracy and intelligibility: the most accurate models usually are not very intelligible (e.g., random forests, boosted trees, and neural nets), and the most intelligible models usually are less accurate (e.g., linear or logistic regression).
The future development of new processing architectures enabling models to be built that more closely mimic the ways that humans learn and think may provide the answer. At Natural Intelligence Semiconductor (NIS) we are developing the Natural Neural Processor (NNP). The NNP is a novel, brain-inspired architecture that is enabling research and development into new symbolic and graphically based neural networks where new solutions to human-interpretable machine learning and AI models will soon emerge. This architecture allows for models to change and adapt and for decision traceability through the model. Check out the Natural Intelligence Semiconductor website for more information on the NNP.
Recently, research has begun on one variation of symbolic neural networks called Deep Symbolic Networks (DSN). In a recent publication, the authors’ abstract states, “The DSN model provides a simple, universal yet powerful structure, similar to DNN (Deep Neural Networks), to represent any knowledge of the world, which is transparent to humans. The conjecture behind the DSN model is that any type of real-world objects sharing enough common features is mapped into human brains as a symbol. Those symbols are connected by links, representing the composition, correlation, causality, or other relationships between them, forming a deep, hierarchical symbolic network structure. Powered by such a structure, the DSN model is expected to learn like humans, because of its unique characteristics.” The authors’ premise is that creating models from symbolic representation starting at micro levels that aggregate to macro levels can reveal a totally transparent, intelligible machine learning model for AI.
A simple example developed in the paper examines how to model the MNIST database of handwritten digits. Starting with the basic building blocks of numeric digits; straight lines, open curves, closed curves and others, one can model the components of all digits and build digits from the components. In this manner, a complex system can be represented by its underlying component parts and be very understandable. This may be much like how we learned as children to recognize and write numerals and the alphabet.
Whether improved human understanding of AI models is the evolutionary path of existing model types or the development of new machine learning symbolic models executed on new architectures such as the NNP, the need for interpretability is one of the most important challenges facing machine learning. This need will drive significant research and rapid innovation, and provide great opportunities for companies that are well positioned in this exciting field.