Timezone: »
Poster
An Efficient PessimisticOptimistic Algorithm for Stochastic Linear Bandits with General Constraints
Xin Liu · Bin Li · Pengyi Shi · Lei Ying
This paper considers stochastic linear bandits with general nonlinear constraints. The objective is to maximize the expected cumulative reward over horizon $T$ subject to a set of constraints in each round $\tau\leq T$. We propose a pessimisticoptimistic algorithm for this problem, which is efficient in two aspects. First, the algorithm yields $\tilde{\cal O}\left(\left(\frac{K^{0.75}}{\delta}+d\right)\sqrt{\tau}\right)$ (pseudo) regret in round $\tau\leq T,$ where $K$ is the number of constraints, $d$ is the dimension of the reward feature space, and $\delta$ is a Slater's constant; and {\em zero} constraint violation in any round $\tau>\tau',$ where $\tau'$ is {\em independent} of horizon $T.$ Second, the algorithm is computationally efficient. Our algorithm is based on the primaldual approach in optimization and includes two components. The primal component is similar to unconstrained stochastic linear bandits (our algorithm uses the linear upper confidence bound algorithm (LinUCB)). The computational complexity of the dual component depends on the number of constraints, but is independent of the sizes of the contextual space, the action space, and the feature space. Thus, the computational complexity of our algorithm is similar to LinUCB for unconstrained stochastic linear bandits.
Author Information
Xin Liu (ShanghaiTech University)
Bin Li (Pennsylvania State University)
Pengyi Shi (Purdue University)
Lei Ying (University of Michigan, Ann Arbor)
More from the Same Authors

2019 Poster: FiniteTime Performance Bounds and Adaptive Learning Rate Selection for Two TimeScale Reinforcement Learning »
Harsh Gupta · R. Srikant · Lei Ying