User interface is accessible at: http://localhost:8890.
When you open Scrapoxy User interface, Scrapoxy prompts you to log in.
Scrapoxy offers various login methods depending on your configuration:
- Local User/Password: Allows only one user.
- Social Logins: Supports authentication through platforms like Google, Github, etc.
The administration interface includes the following components:
- Left Menu: Facilitates navigation between different project sections.
- Header: Displays the name of the connected user and provides a menu for accessing user settings and logging out.
- Central Area: Reserved for displaying pages and relevant content.
During the initial connection, Scrapoxy will guide you through the creation of a new project with the following settings:
- Name: Unique identifier for the project;
- Username: Authentication username used for proxy authentication in requests (click on the
clipboardicon to copy the username);
- Password: Authentication password used for proxy authentication in requests (click on the
clipboardicon to copy the password);
- Renew token: Click on this button to renew username and password;
- Minimum proxies: The minimum number of proxies online when the project status is CALM;
- Auto Rotate Proxies: If enabled, proxies are automatically rotated at random intervals within the specified Delay range;
- Auto Scale Up: When enabled, the project status switches to
HOTupon receiving a request, and all proxies are started;
- Auto Scale Down: When enabled, the project status switches to
CALMif no requests are received after a specified delay, and all proxies are stopped;
- Intercept HTTPS requests with MITM: If enabled, Scrapoxy intercepts and modifies HTTPS requests and responses.
- Certificate: Install this CA certificate to avoid security warnings in Browsers or Scrapers;
- Keep the same proxy with cookie injection: If enabled, Scrapoxy injects a cookie to maintain the same proxy for a browser session (sticky cookie);
- Override User-Agent: If enabled, Scrapoxy overrides the User-Agent header with the value assigned to a proxy instance. All requests made with this instance will have the same User-Agent header;
After saving these settings, Scrapoxy will prompt you to create the first credential for the project.
Settings can also be accessed through the left menu.
List of projects
If you come back to the
Projects section, Scrapoxy will display a list of all accessible projects.
Upon the first connection:
- Scrapoxy will redirect you to the
- Search for a provider by name or type;
- Providers are categorized by type (e.g., datacenter provider, proxies services, etc.);
- Click on
Createto commence the configuration the chosen provider.
The settings for the connector's credential include:
- Type: The provider of the credential;
- Name: A unique name for the credential within the project;
- Options: Configuration options for the credential, dependent on the type of provider.
When you save the settings, Scrapoxy performs a validation test to ensure that the entered credentials are valid.
List of credentials
After saving, Scrapoxy will display the list of credentials.
If you open a credential, Scrapoxy displays the settings:
You have the following options when managing a credential:
- Update: Modify the credential with new settings;
- Delete: Remove the credential, provided there are no connectors currently using it.
Upon the first connection, Scrapoxy will guide you through the creation of a new connector with the following settings:
- Credential: Selection from the list of available credentials from the previous section;
- Name: Unique identifier for the connector within the project;
- # of proxies: Maximum number of proxies that the connector can provide and that you intend to use;
- Proxies Timeout: Maximum duration for connecting to a proxy before considering it as offline;
- Proxies Kick: If enabled, maximum duration for a proxy to be offline before being removed from the pool;
When you save the settings, Scrapoxy performs a validation test to ensure that the entered configuration is valid.
Proxies Kick must be greater than Proxies Timeout.
List of connectors
After saving, Scrapoxy will display the list of connectors.
If you open the connector, Scrapoxy will present the associated settings:
Upon opening the connector, Scrapoxy provides the following options:
- Update: Modify the connector with new settings;
- Validate: Confirm the validity of the connector configuration. Particularly useful for datacenter providers, as Scrapoxy checks for the existence of the template image;
- Delete: Remove the connector only if the project's status is OFF.
The type cannot be modified.
The project can have the following statuses:
|The project is stopped. All proxies are halted and removed.
|The project is in a sleep state. It maintains the minimum required proxies according to the project's settings and awaits the initial request (if
Auto Scale Up is enabled).
|The project is active. All proxies are started and await connections.
A connector has:
- Status: Indicates the current status of the connector (refer to the descriptions below);
- Name: The unique identifier assigned to the connector;
- Type: The classification of the connector, consistent with the credential type;
- Proxies count: Displays the number of online proxies and the maximum allowed proxies for this connector;
- Proxies control: Offers the option to increase or decrease the maximum number of proxies for this connector;
- Default: If enabled, designates this connector as the exclusive one used when the project status is
CALM. Only one connector can be set as the default for a project;
- Active: If enabled, initiate the proxies associated with this connector;
- Options: Provides a shortcut for scaling, updating, installing, or uninstalling the connector.
The connector can have the following statuses:
|The connector is online.
|The connector is offline.
|The connector has an error. Details are in the tooltip.
If you open the
Proxies section, Scrapoxy will present a list of proxies.
This compilation includes proxies from all connectors, with pagination to enhance performance.
A proxy is characterized by the following attributes:
- Connector: The name of the associated connector;
- Name: The identifier assigned to the proxy;
- IP Address: The IP address of the proxy along with geo-localization information;
- Status: Indicates the current status of the proxy (refer to the descriptions below);
- Received: The amount of bytes received by the proxy;
- Sent: The amount of bytes sent by the proxy;
- Requests: The number of requests made by the proxy;
- Uptime: The uptime of the proxy, representing the duration between its creation and the current moment;
- Delete: Manually remove the proxy from the pool.
The proxy has 2 sets of statuses:
- Instance Status: This reflects the overall operational status of the proxy;
- Connection Status: This denotes the status of the connection associated with the proxy.
|The proxy is starting.
|The proxy is started.
|The proxy is stopping.
|The proxy is stopped.
|The proxy has an issue.
|The connection is online.
|The connection is offline.
|The connection has an issue.
Scrapoxy will route traffic only when the proxy status is
STARTED and connection is
- When you access the
Coveragesection, Scrapoxy presents a world map showcasing the locations of all proxies. This visualization aids in comprehending the geographic distribution of your proxies;
- Scrapoxy furnishes metrics such as active proxies, countries, or cities;
- Click on the full-screen icon to expand the map for a more detailed view.
Checking the origin of your proxies is valuable for optimizing your scraping performance.
The map displays only proxies that have been used at least once, allowing Scrapoxy to fingerprint them.
The fullscreen mode is tailored to optimize map display on a TV. Additionally, it provides a ranking of top countries, cities, and ASNs (Autonomous System Numbers).
Metrics section, Scrapoxy provides a set of metrics for monitoring your project. These metrics include:
- Range: Adjusts the window of the metrics (e.g., 1 minute, 5 minutes);
- Received: Total amount of bytes received by all proxies;
- Sent: Total amount of bytes sent by all proxies;
- Received rate: Current rate of data received by all proxies;
- Sent rate: Current rate of data sent by all proxies;
- Requests: Total number of requests made by all proxies;
- Stops: Total number of "delete" delete orders;
- Proxies Created: Total number of proxies created;
- Proxies Removed: Total number of proxies deleted;
- Avg Requests Before Stop: Average number of requests made by proxy before removal from the pool;
- Avg Uptime Before Stop: Average uptime for a proxy before removal from the pool;
- Data received & sent: Amount of data received and sent by all proxies over time;
- Requests sent: Number of requests made by all proxies over time;
- Stop orders received: Number of "stop" orders received over time.
Only the last 3 charts use the window setting.
- Line charts are valuable for monitoring project usage.
- By analyzing "stop" orders, you can determine how much time a proxy is utilized before removal from the pool.
List of tasks
If you open the
Tasks section, Scrapoxy displays a list of executed tasks.
If you open a task, Scrapoxy presents detailed information about the task.
List of users
If you open the
Users section, Scrapoxy displays the list of users authorized to access the project.
- You can only add users who have previously been authenticated by Scrapoxy.
- It is impossible to remove yourself from the project.