Why my preview becomes pixelated in OBS Studio when I downscale the preview?
The problem is the inevitable rounding of the pixel count that occurs given to the original size and reduced size. Ideally, when reducing size you should select resize values that divide to whole numbers and things should improve. Since different software has different rescaling proportions, you will see the pixelated when the image gets smaller.
You can try downscaling the resolution of the preview as below:
If the preview screen is deliberately reduced, 1080p is more likely to cause pixel stacking than the 480p, causing the pixel phenomenon in the left picture.