Universal

A universal job, which scrapes any URL.

Usage

Universal()

The model takes no domain, locale, start_page, pages or limit, because no docs page names them for universal. extra passes them. What a live test shows about universal and the instruction parameters records the run that checked it.

Attributes

url: str: The page to scrape. A page with an empty body faults the job, whatever its status.
geo_location: str | None: A country name or an ISO 3166-1 alpha-2 code, such as Germany or DE. The API accepts any string, and a value it does not know, such as de, has no effect and still bills.
user_agent_type: _Device | None: The device of the job’s user agent. A desktop_* value raises, because it draws from the same agents as desktop.
force_headers: bool | None: Sends headers to the site.
force_cookies: bool | None: Sends cookies to the site.
successful_status_codes: list[int] | None: More status codes that end the job done, such as 503. The API rejects a 3xx code for free.
follow_redirects: bool | None: False faults a job whose page redirects. A chain of more than 10 redirects faults the job either way.
cookies: list[_Cookie] | None: The cookies that the site receives. They raise without force_cookies, because the site then receives none and the job still bills.
headers: dict[str, str] | None: The headers that the site receives. They raise without force_headers, because the site then receives none and the job still bills. A User-Agent header never replaces Oxylabs’ own.
session_id: str | None: Jobs that share an ID share an exit IP, for 100 jobs or 25 minutes after the first.
http_method: Literal["get", "post", "options"] | None: post sends content as the request body.
content: str | None: The request body in Base64, which the API rejects for free in any other encoding.
store_id: str | None: A Home Depot store ID.

oxyscraper is not affiliated with or endorsed by Oxylabs. Oxylabs and Oxy are trademarks of Oxylabs.