Is the DAC working? I.e. is the audio output approx same as with your Dragonfly? Is it undistorted? You need to do some diagnosis
What is the nature of the USB noise? Broadband hiss or high frequency tones or low frequency mains hum?
Is the DAC OK when fed with Toslink signal from TV? This checks whether the DAC is OK.
Does the DAC (running Toslink) get noisy when connected to the USB of the RPI? This would suggest that the DAC is extremely susceptible to USB noise from the RPI.
Using the DAC on other USB ports on the RPI - do they all have the same issue?
Hum would suggest a ground loop problem or faulty RCA leads from the DAC to amp