This was bothering me.
I'm using an wireless headset (HyperX Cloud Flight Wireless Gaming Headset) that has an USB receptor and some old Logitech speakers on my desktop, currently running Ubuntu 20.04 (dual booting with Windows 10).
Every time I reboot or turn on my computer and boot Ubuntu the default output audio device is set to my headset, which most of the time is off (I only use it when gaming).
Going to Settings - Sound and choosing the correct output device (Line Out) only changes it for the current session.
You can solve this by doing the following:
Edit the file default.pa:
$ sudo vim /etc/pulse/default.pa
And comment the following lines:
Reboot. Now your computer should leave the output device you select in settings alone. My guess is that every time you turn on the computer it finds the USB receptor and change to it.