Online learning is a framework that makes minimal assumptions about the sequence of instances provided to a learner. This makes online learning an excellent framework for dealing with sequences of instances that vary with time. In this talk, we will look at inherent connections between online learning, certain Probabilistic Inequalities and the so called Burkholder Method. We will see how one can derive new, optimal, adaptive online learning algorithms using the Burkholder Method via the connection with Probabilistic Inequalities. We will use this insight to help us get a step closer to what I shall term Plug-&-Play ML. That is, help us move a step towards building machine learning systems automatically.