I couldn't figure out how to cancel this download either. Fortunately, I found the Garageband preferences plist file, which contained a list of active downloads. To make Garageband 10.3.4 stop downloading on Catalina (10.15.3), I did the following:
1. Quit Garageband 10.x
2. Open Terminal (find it in the Applications => Utilities folder, or you can use Spotlight and search for "Terminal" by name)
3. Run this command in Terminal to remove the download list from the preferences file and make Garageband forget that it was downloading anything (type the following and press Enter):
defaults delete ~/Library/Containers/com.apple.garageband10/Data/Library/Preferences/com.apple.garageband10.plist MAPrefs_ActiveDownloads_2
4. Relaunch Garageband.
Whew! Download stopped!
(To restart the download, select Garageband >> Sound Library >> Download All Available Sounds)
Note: I could not find where it downloaded the files to recover any space. Garageband had already downloaded maybe 1GB of the sounds when I stopped it. The "MAPrefs_ActiveDownloads_2" list consisted of many packages; it might have even been installing them as each package downloaded. I suspect some of that first 1GB went into "/Library/Audio/Apple Loops/Apple Loops for GarageBand", as its modified date was set to today.
I hope this helps others recover from Apple's failure to put a (x) cancel button in Garageband's very slippery download window! (Even though I realize the OP's problem was resolved by completing the download over faster internet!)
-Bill
ps. "defaults" has been a useful command to read, write, and delete properties in Mac OS X property list (plist) files. To learn more, try running "defaults --help" or "man defaults" in Terminal.