Many areas of science make extensive use of computer simulators that implicitly encode likelihood functions of complex systems. Traditional statistical methods are poorly suited for these so-called likelihood-free inference (LFI) settings outside the asymptotic and low-dimensional regimes. Although new machine learning methods, such as neural density estimators and generative models, have revolutionized the sample efficiency and capacity of LFI methods, it remains an open question whether they produce confidence sets with correct instance-wise coverage in general settings. In this talk, I will describe a modular inference framework that bridges classical statistics and modern machine learning to provide (i) confidence sets with finite-sample validity at any value of the unknown parameters, and (ii) interpretable diagnostics for estimating empirical coverage across the entire parameter space. We refer to this framework as likelihood-free frequentist inference (LF2I). Any method that defines a test statistic can leverage LF2I to create locally valid confidence sets and diagnostics without costly Monte Carlo or bootstrap samples at fixed parameter settings on a grid. In my talk, I will discuss where we stand with LF2I and challenges that still remain. (A version of this work can be found on arXiv:2107.03920)