The probabilistic programming system makes it easier to represent and learn about uncertainties in AI models
Uncertainty in deep learning makes it tricky to deploy artificial intelligence in several high-risk areas. AI and its applications are among the fastest-growing sectors. And, many companies in healthcare, automotive, and manufacturing reportedly have increased their investments in AI this year. While AI can offer many benefits, there are obstacles to building trustworthy systems. AI research currently has been focused on understanding how an environment is influenced when AI predictions are matched with confidence or uncertainty metrics.
Recently, Meta (formerly Facebook) released a probabilistic programming system that makes it easier to represent and learn about uncertainties in AI models.
What Is A Bean Machine?
Built on top of Meta’s PyTorch machine learning framework and Bean Machine Graph (BMG) with a custom-designed C++ backend, Bean Machine allows data experts to write math during model creation, directly in Python, and inculcate BMG to create a probabilistic model inferring the possible distributions for predictions based on the declaration of the created model.
“Bean Machine is inspired from a physical device for visualising probability distributions, a pre-computing example of a probabilistic system. We on the Bean Machine development team believe that the usability of a system forms the bedrock for its success, and we’ve taken care to centre Bean Machine’s design around a declarative philosophy within the PyTorch ecosystem.” said the Meta researchers behind Bean Machine in a blog.
Currently available in early beta, Bean Machine can be used to help discover unobserved properties of a model automatically, using “uncertainty-aware” learning algorithms. Using Bean Machine, predictions can be quantified with reliable measures of uncertainty in the form of probability distributions.
An analyst can not only understand the system’s current prediction, but also the relative likelihood of other possible predictions. One can also query the model’s intermediate learned properties. With such a system, users can interpret why a particular prediction was made, which in turn, can aid in the model development process.
Decoding Model Uncertainty
A deep learning model can seem to be overconfident about making decisions at times, even if it makes mistakes, due to inaccurate data being fed to understand from. Uncertainties within the model might arise in such cases which result in faulty output. Several uncertainties such as epistemic uncertainty, which describes what a model does not know, due to inappropriate training data being fed, and aleatoric uncertainty, the kind of uncertainty arising from certain natural randomness of observations can be addressed. Given enough training samples, epistemic uncertainty decreases, but aleatoric uncertainty cannot be further reduced, even when more data is provided.
The AI technique that Bean Machine adopts is known as probabilistic modeling, which can measure several kinds of uncertainty by taking into account the impact of random events in predicting the occurrence of future outcomes. When compared with other present machine learning approaches, probabilistic modeling offers certain benefits that can provide an edge over others, with many aware features such as uncertainty estimation, expressivity, and interpretability.
Uncertainty when measured by Bean Machine can help analyse a created model’s limits and potential failure points. As a real-world business case example, making use of uncertainty measures can reveal the margin of error for a house price prediction model or the confidence of a model designed to predict whether a new app feature will perform better than an old feature.
Importance of Addressing Uncertainty
A recent Harvard study, that illustrated the importance to the concept of uncertainty, found that showing uncertainty metrics to both individuals with a background in machine learning and non-experts had an equalising effect on their certain resilience to AI predictions.
While trusting an AI might never be as simple as providing metrics, addressing and encouraging awareness of the pitfalls could be a step towards protecting people from machine learning’s limitations.
Learning of this kind can prove to be useful when creating reinforcement learning agents, where inculcating such knowledge can help agents handle uncertainty by using the methods of probability and decision theory, while they learn their probabilistic theories of the world from a series of episodes or repetitive experiences.
Introducing probability to such learning agents provides the foundation for their iterative training in many machine learning models, known as maximum likelihood estimation, the math behind models such as linear regression, logistic regression, artificial neural networks, and many more.
Bean Machine can quantify predictions with reliable measures of uncertainty in the form of probability distributions, which makes it easier to encode a rich model directly into the source code. The Meta research team hopes that such properties make using Bean Machine simple and intuitive, “whether it be authoring a model or advanced tinkering with its learning strategies.”
With such advancements in AI, we might see systems in the future that are more aware of their decisions and possess a clear view during their decision-making process.
Similar developments will not only help implement better automated systems but also guide AI developers towards creating unbiased systems that can boost the work environment it is implemented in.
If you liked reading this, you might like our other stories
Real-time Data Analytics Predictions for Businesses
Budget 2021 – Are CMOs Prioritizing Investments in Marketing Technology?
