Timezone: »

 
Poster
Sparsity in Continuous-Depth Neural Networks
Hananeh Aliee · Till Richter · Mikhail Solonin · Ignacio Ibarra · Fabian Theis · Niki Kilbertus

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #121
in Poster Session 5 »

Neural Ordinary Differential Equations (NODEs) have proven successful in learning dynamical systems in terms of accurately recovering the observed trajectories. While different types of sparsity have been proposed to improve robustness, the generalization properties of NODEs for dynamical systems beyond the observed data are underexplored. We systematically study the influence of weight and feature sparsity on forecasting as well as on identifying the underlying dynamical laws. Besides assessing existing methods, we propose a regularization technique to sparsify ``input-output connections'' and extract relevant features during training. Moreover, we curate real-world datasets including human motion capture and human hematopoiesis single-cell RNA-seq data to realistically analyze different levels of out-of-distribution (OOD) generalization in forecasting and dynamics identification respectively. Our extensive empirical evaluation on these challenging benchmarks suggests that weight sparsity improves generalization in the presence of noise or irregular sampling. However, it does not prevent learning spurious feature dependencies in the inferred dynamics, rendering them impractical for predictions under interventions, or for inferring the true underlying dynamics. Instead, feature sparsity can indeed help with recovering sparse ground-truth dynamics compared to unregularized NODEs.

Keywords: Dynamical Systems · interpretability · Neural ODEs · Sparsity
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Hananeh Aliee (Helmholtz AI)

I am a postdoctoral researcher in computational biology working in the ngroup of Prof. Fabian Theis. I did my phd in Computer Science.

Till Richter (Helmholtz Munich)
Till Richter

I'm a doctoral candidate in the Helmholtz International Lab, consisting of the Machine Learning group of Prof. Dr. Dr. Fabian Theis at ICB, the Reliable Machine Learning group of Prof. Dr. Niki Kilbertus at Helmholtz AI, and Prof. Dr. Yoshua Bengio at MILA. I'm a member of the graduate school Munich School for Data Science (MUDS) and HELENA. Fields: Causal Machine Learning, Reliable Machine Learning

Mikhail Solonin (Technische Universität München)
Ignacio Ibarra
Fabian Theis (Helmholtz Munich)
Niki Kilbertus (TUM & Helmholtz AI)

More from the Same Authors