Do you use your Pi on a WiFi or ethernet connection? Also, which Pi version do you use? We do not encounter particularly slow performance in Spotify Connect Plugin.
Are the delays still as large as in your original post? We will set up a new image with Pi 3B and see if we can recreate it. Very odd that there are not more people reporting this issue if it is a general one. Could you paste the debug info from your Spotify Connect plugin?