A Lyapunov theoretic approach to stability and robustness of optimal control is well-established for deterministic systems. Considering the rise of learning in control, the close links between optimal control and reinforcement learning (RL), as well as the fact that RL is usually formulated for stochastic systems, the question arises whether similar stability and robustness guarantees can be achieved in a stochastic setting. After exploring the relations between stability, optimal control, RL and dynamic programming in a general setting, and how these concepts work in the stochastic case, the main result, which guarantees (robust) semiglobal practical recurrence under stabilizability and detectability assumptions, will be presented. This work received the Young Author Award at the 13th IFAC Symposium on Nonlinear Control Systems (NOLCOS), 2025.