to move control over sound card in case you have applied some dirty hacks you have to refer to root powers.

First, remove all access to the sound card from current user (user1):

setfacl -x u:user1     /dev/*dsp /dev/audio /dev/mixer* /dev/snd/*(^/)

Then add it back to for another user (user2):

setfacl -m u:user2:rw- /dev/*dsp /dev/audio /dev/mixer* /dev/snd/*(^/)

pulseaudio -k will reload the resp. pulseaudio and will not find any card for user1, but find the cards for user2. I only did this with no client connected, maybe needs more investigation.