all_tickers.py is stuck

[dropcunt]

I think there is a problem with the urlib.request. Might need to add a header

[tmhieu99]

Hi @dropcunt, I got the same problem. Have you found a way to fix it?

[dkubanyi]

Hi @tmhieu99 and @dropcunt, I stumbled upon the same problem and the solution is actually very easy. The exchange where the script is fetching the data from changed its routing, which is why it gets stuck. In all_tickers.py, you need to replace the part where you fetch the exchange data to this:

for exchange in ["NASDAQ", "NYSE", "AMEX"]:
    # this is the changed URL
    url = "https://api.nasdaq.com/api/screener/stocks?offset=0&exchange={}&download=true"

    repeat_times = 10
    
    for _ in range(repeat_times):
        response = urlopen(url.format(exchange))

I was then able to successfully download the tickers

[tmhieu99]

Hi @dkubanyi, tt worked now. Thanks for your solution.

[vedantk281007]

@dkubanyi could you send the full code block?
When I do this, it still does not work.

[glitchawy]

can someone explain because it still doesn't work after changing the code

[dkubanyi]

@glitchawy took a look at it again, and even though I can't test the rest of the process, I can give you a head start - this implementation fetched the data from nasdaq's json api and wrote it into a csv file:

def get_tickers(percent):
    """Keep the top percent market-cap companies."""
    assert isinstance(percent, int)
    for exchange in ["nasdaq", "nyse", "amex"]:
        repeat_times = 10 # repeat downloading in case of http error

        for _ in range(repeat_times):
            try:
                url = "https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=3296&exchange={}".format(exchange)
                headers = {
                    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0",
                }

                print("Downloading tickers from {}: {}...".format(exchange, url))

                response = requests.get(url, headers=headers)
                j = response.json()

                table = j['data']['table']
                table_headers = table['headers']

                with open('input/tickerList.csv', 'w', newline='') as f_output:
                    csv_output = csv.DictWriter(f_output, fieldnames=table_headers.values(), extrasaction='ignore')
                    csv_output.writeheader()

                    for table_row in table['rows']:
                        csv_row = {table_headers.get(key, None): value for key, value in table_row.items()}
                        csv_output.writerow(csv_row)
            except:
                continue

It's just quick and dirty fix, feel free to adjust it to your needs and preferences.