Using Selenium on PythonAnywhere

NOTE These instructions are for the system image fishnchips or later. If you have an older system image, you will need to update your system image to use these instructions. Also note that selenium will only work in tasks, web apps and consoles. It will not work in notebooks and over SSH.

If you're on the fishnchips system image, the latest version of selenium confirmed working (as of 2022-06-13) is 4.1.3, and we recommend upgrading it for your account/virtualenv to this version. For example, if you're using Python 3.7, run this in a Bash console:

pip3.7 install --user selenium==4.1.3

You can run Selenium with Chrome using code like this:

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
browser = webdriver.Chrome(options=chrome_options)
try:
    browser.get("https://www.google.com")
    print("Page title was '{}'".format(browser.title))
finally:
    browser.quit()

Cleaning up manually

If you fail to clean up properly in your code, you may find that you cannot start new Chrome processes (you will get error messages saying Chrome crashed). In that case, you can use the "process listing" buttons on the Consoles page and the Tasks page to manually find and kill any leftover Chrome processes.

Free users are currently restricted to a whitelist of sites. If you want to use Selenium to talk to a site that's not on the list, you'll need to upgrade to a paid account.

Don't use selenium from your web app

Selenium and Firefox are too slow to start up, and the request/response cycle is meant to be fast. Instead, build some sort of queue of jobs, and use a scheduled task to process those jobs. More info here

Page crash errors

Sometimes your code may fail with an error like this:

Message: unknown error: session deleted because of page crash

If you get this, try adding the option

chrome_options.add_argument('--disable-dev-shm-usage')