How to scrape many files of the website without getting blocked
I am working on a project that consists of a neural network trying to recreate music. To do this, I need a lot of data. I wrote a web scraper to download about 200,000 solo violin files from MuseScore. After downloading about 200, I got blocked from the site and only get a http 403 access denied error. My problems are these: how do I get unblocked? (Or more likely, how long until I get unblocked). Second, how can I download 200,000 music files without getting blocked from the website?
Comments
Seems you're at the wrong site here, musescore.org, with a question/problem about musescore.com. Maybe talking to support@mzusescore.com, or asking that question in https://musescore.com/groups/improving-musescore-com/ is a better approach
I suggest that you use the contact link at the bottom of this page and explain what you want to do. They can advise you and possibly even give you a special temporary permission for doing this. Keep in mind that today is Sunday, even in Russia and they may need to do some internal talking before they give you an answer.
I am trying to do something similar to this. Did you ever resolve this issue?
In reply to I am trying to do something… by dudebandnerd
You too are asking this at the wrong website.
In reply to You too are asking this at… by Jojo-Schmitz
If the OP had made a duplicate post on the correct site, I would have tried to communicate there. Since they didn't I was forced to reply here.
In reply to If the OP had made a… by dudebandnerd
No, you could have asked there, just like the OP should have.
Nobody here on this site can help you with this.
In reply to No, you could have asked… by Jojo-Schmitz
I will create a post there, but since this is a fairly uncommon request, it has a high probability that it will go unanswered. This was my best option to get in contact with Kevin.
In reply to I will create a post there,… by dudebandnerd
True
In reply to True by Jojo-Schmitz
[EDIT]
Never mind
How do you download so many files? Script?