Alpha entropy search for new information-based Bayesian optimization

D. Fernández-Sánchez, E.C. Garrido-Merchán, D. Hernández-Lobato

Knowledge-Based Systems Vol. 322, pp. 113612

Summary:

Bayesian optimization (BO) methods based on information theory have obtained state-of-the-art results in several tasks. These techniques rely on the Kullback–Leibler (KL) divergence to compute the acquisition function. We introduce a novel information-based class of acquisition functions for BO called Alpha Entropy Search (AES). AES is based on the alpha-divergence, which generalizes the KL-divergence. Iteratively, AES selects the next evaluation point as the one whose associated target value has the highest level of dependency with respect to the location and associated value of the global maximum of the optimization problem. Dependency is measured in terms of the alpha-divergence, as an alternative to the KL-divergence. Intuitively, this favors evaluating the objective function at the most informative points about the global maximum. The alpha-divergence has a free parameter α, which determines the behavior of the divergence, balancing local and global differences. Therefore, different values of α result in different acquisition functions. AES acquisition lacks a closed-form expression. However, we propose an efficient and accurate approximation using a truncated Gaussian distribution. In practice, the value of α can be chosen by the practitioner, but here we suggest using a combination of acquisition functions obtained by simultaneously considering a range of α values. We provide an implementation of AES in BOTorch and we evaluate its performance in synthetic, benchmark, and real-world experiments involving the tuning of the hyper-parameters of a deep neural network. These experiments show that AES performance is competitive with other information-based acquisition functions such as JES, MES, or PES.

Spanish layman's summary:

Proponemos Alpha Entropy Search (AES), un método de optimización Bayesiana basado en la divergencia alfa, que equilibra exploración local-global y compite eficazmente en tareas como el ajuste de redes neuronales.

English layman's summary:

We propose Alpha Entropy Search (AES), a Bayesian optimization method using alpha-divergence. AES balances local-global exploration efficiently, showing competitive performance in optimization tasks, including neural network tuning.

Keywords: Bayesian optimization; Information theory; Entropy search; Alpha-divergence

JCR-JIF Impact Factor and WoS quartile: 8,000 - Q1 (2025)

DOI reference: https://doi.org/10.1016/j.knosys.2025.113612

Published on paper: July 2025.

Published on-line: May 2025.

Citation:
D. Fernández-Sánchez, E.C. Garrido-Merchán, D. Hernández-Lobato, "Alpha entropy search for new information-based Bayesian optimization", Knowledge-Based Systems, Vol. 322, pp. 113612, July 2025. [Online: May 2025] doi: 10.1016/j.knosys.2025.113612