Exploratory microscopy is simultaneously driven by observations and hypotheses. Consequently, we need to develop robust interfaces that connect massive, complex, noisy, and incomplete data with foundational scientific knowledge.
Our group is interested in combining machine learning with scientific and instrument priors to create computational lenses that help make sense of the chaotic and nearly invisible dynamics that occur at the nanometer-scale. These lenses combine statistical learning with priors (i.e., probe-sample interaction Physics, Optics, and instrumentation).
We first developed these computational lenses for single-particle diffractive imaging with X-ray Free-electron Lasers, where unsupervised statistical learning can discover transient intermediate states and spontaneous order formation in highly heterogeneous and dynamic systems.
Our group then extended these noise-robust methods to electron-based imaging sciences to develop core ideas and technologies towards far-reaching computational lenses that are too challenging for hardware-based electron microscopy alone.
Most recently, we have been learning spatiotemporal motifs in complex systems using semi-supervised statistical learning. These motifs allow us a unique window to understand complexity as a language comprising tokens (motifs), grammar (motif sequences), and context (emergence).
Human visual perception is part-eye and part-brain. In the context of imaging, imaging algorithms form the brain of our optical instruments. In doing so, these algorithms can help us measure instead of merely see.
High-throughput detectors and efficient computation have enabled us to create computational versions of physical lenses. These computational lenses are capable of imaging very small objects in noisy and challenging situations, often beyond the designed capabilities of their physical counterparts. For example, we are able to augment the functions of computational lenses by inserting known prior knowledge about the capabilities of the underlying optics and/or characteristics of the specimen under study. Our group pushes and helps define the capabilities of such computational lenses.
Midjourney thinks this is a Computational Lenses with transmission electron microscope… 😂
Computational Lenses with X-ray Diffraction Imaging. Ewald sphere sections (Source: Duane@Mathematica).
Living cells are built from countless tiny, fleeting interactions between molecules. Each interaction is weak and often noisy, and almost random on its own. But when many of them happen together, in the right places and at the right times, they produce reliable and meaningful outcomes. Cells divide, move, respond to stress, and organize themselves in ways that are not clockwork precise, but statistically predictable. Today, we can observe these dynamics in extraordinary detail using advanced imaging. Yet we still lack a clear framework to describe how all these small, transient interactions combine to shape the larger behavior of life.
Our goal is to uncover the hidden “patterns” within this complexity. Instead of treating biological data as overwhelming or chaotic, we aim to identify recurring spatiotemporal motifs (i.e., meaningful patterns in space and time) that act as the building blocks of cellular behavior. By combining cutting-edge imaging with interpretable artificial intelligence, we seek to translate these patterns into a kind of biological codebook. This approach allows us not only to predict what cells will do next, but also to connect those predictions back to established physical principles. In doing so, we hope to build a bridge between data-driven discovery and mechanistic understanding, revealing how fragile molecular forces collectively give rise to the organized, robust dynamics of living systems.
We coin this method of discovering spatiotemporal motifs in large collections of interacting objects, COSMOS (COllective Spatiotemporal MOtif Sensing).
A dramatic enactment of how we infer the orientations of diffraction patterns (source: Duane@Blender).
From weak, transient molecular interactions to reliable cellular behavior: fleeting and noisy interactions give rise to recurring spatiotemporal motifs, which are distilled into an interpretable “codebook” of tokens. These tokens capture the key states and transition rules that enable mesoscale predictability in cellular and multicellular dynamics.
Computational Lenses for X-ray Imaging.
Computational Lenses for Electron Microscopy.
Order-disorder Transitions.
Entomology
Biophysics
Epidemiology
Data Science and Machine Learning.
Light Microscopy (Bio-Imaging)
AI for Science Masters Program at the Faculty of Science, National University of Singapore.
We created the world’s first (only?)