diff --git a/doc/source/rllib-algorithms.rst b/doc/source/rllib-algorithms.rst
index aa3637acf..9cbfff9fb 100644
--- a/doc/source/rllib-algorithms.rst
+++ b/doc/source/rllib-algorithms.rst
@@ -700,7 +700,7 @@ Tuned examples: `CartPole-v0 <https://github.com/ray-project/ray/blob/master/rll
 
 
 Multi-Agent Methods
--------------------
+~~~~~~~~~~~~~~~~~~~
 
 .. _qmix: