Uber today released a framework for designing experiments within Pyro, its open source tool for deep probabilistic modeling. The framework leverages machine learning to enable optimal experimental design (OED), a principle based on information theory that enables the automatic selection of designs for complex experiments. With the framework, experimenters can apply OED to a large class of experimental models, from DNA assays to website and app A/B tests.
Experiments play an important role in, for example, the product development process, and designing them often requires sufficient domain expertise. But even experts sometimes struggle to contend with hundreds of design parameters, noisy data, and real-time adaptation. Uber’s solution is a new class of algorithms for computing experiments that’s faster and more scalable than previous methods. In practice, it could help neuroscientists map microcircuits in the brain, psychologists compare models of human memory, and statisticians identify election polling strategies, among other things.
Uber’s framework begins with experimental design: It scores each possible design using a function and picks the highest-scoring of the bunch. Then, it ingests observations recorded during the experiment and performs inference to model the probability of various outcomes.
To score the experimental designs, the framework makes use of expected information gain (EIG), a measure of the information an experimenter can expect to learn from an experiment. Basically, if uncertainty regarding the target — the thing the experimenter is attempting to learn about — decreases as a result of new information, it’s implied that the design is superior.
Uber’s framework specifically scores designs on expected information gain, or the expectation of the information gain over all possible observations that might result if an experiment was run. It simulates a number of possible experimental outcomes given its current knowledge of the world, and it computes the information gain for each outcome before aggregating the gains to yield EIG.
“By developing and open-sourcing [this framework] in Pyro, we hope others in the community can benefit from this framework and apply it to their own research areas, wrote Uber research scientist Martin Jankowiak and intern Adam Foster in a blog post. ” [The framework] has allowed us to learn more about the world using the same experimental budget. While this may not be a big deal for a simple experiment … it can be very difficult to find good design heuristics for more complex experiments and almost impossible to find good heuristics that are adaptive. This is why OED is so attractive: it automates experimental design and makes experiments more efficient.”
The release of the new framework follows Uber’s open-sourcing of Manifold, a visual tool for debugging AI models, and Plato, a platform for building, training, and deploying conversational AI and machine learning. Early last year, the company debuted Ludwig, an open source toolbox built on top of Google’s TensorFlow framework that allows users to train and test AI models without having to write code. And in February 2019, it launched the Autonomous Visualization System (AVS), a standalone web-based technology for understanding and sharing autonomous systems data.
Source: Read Full Article