Uninstalled all plugins: I only have 3 plugins installed
Installed YouTube Music - Tested and forward and back button works and library stays
Installed Music Services Shield - Tested and forward and back button works and library stays
Looks like the culprit is the Fusion DSP