Microsoft Azure Connector
Microsoft Azure is a cloud computing service created by Microsoft.
Prerequisites
An active Microsoft Azure subscription is required.
Azure Portal
Connect to Azure Portal.
Step 1: Register a new application
Search the Microsoft Entra ID (aka Active Directory).
- Click on
App registrations
; - And click
New registration
.
Complete the form with the following information:
- Name:
scrapoxy
- Supported account types:
Accounts in this organizational directory only (Microsoft only - Single tenant)
And click on Register
.
Remember:
- the
Application (client) ID
; - and the
Directory (tenant) ID
.
Step 2: Create a client secret
- Click on
Certificates & secrets
; - Click on
New client secret
; - Enter a description, select the maximum expiration time;
- Click on
Add
.
Remember the Value
of the secret.
Step 3: Get your subscription ID
Search the Subscription.
Select the first subscription.
Remember the Subscription ID
value.
Step 4: Add a role to the application
On the subscription:
- Click on
Access control (IAM)
; - And click on
Add
.
- On tab
Role
; - Select
Privileged administrator roles
; - Select
Contributor
; - Click on
Next
.
- On tab
Members
; - Click on
Select members
; - Select the application you created;
- Click on
Select
:
Finally, click on Review + assign
2 times.
Scrapoxy
Open Scrapoxy User Interface and select Marketplace
:
Step 1: Create a new credential
Select Azure
to create a new credential (use search if necessary).
Complete the form with the following information:
- Name: The name of the credential;
- Application (client) ID: The Application ID (aka Client ID) of the application;
- Directory (tenant) ID: The Directory ID (aka Tenant ID) of the application;
- Secret value: The Client Secret of the application;
- Subscription ID: The Subscription ID.
And click on Create
.
INFO
It may take Azure up to 10 minutes to reflect the changes.
Step 2: Create a new connector
Create a new connector:
Complete the form with the following information:
- Credential: The previous credential;
- Name: The name of the connector;
- # Proxies: The number of instances to create;
- Proxies Timeout: Maximum duration for connecting to a proxy before considering it as offline;
- Proxies Kick: If enabled, maximum duration for a proxy to be offline before being removed from the pool;
- Location: The region where the instances will be created;
- Port: The port of the proxy (on Azure);
- Resource Group Name: The resource group to host the instances;
- VM size: The size of the VM;
- Spot Instance: If enabled, the VM will be a spot instance;
- Storage Type: The type of storage;
- Prefix: The prefix for all resources created on Azure;
- Image Resource Group Name: The resource group where the image is stored.
And click on Create
.
Most default values can be retained if suitable for the use case.
WARNING
When setting up the connector in multiple locations, assign a unique Resource Group Name, Image Resource Group Name and distinct Prefix for each location. Without this, connectors may interfere with each other, shutting down instances from the same provider.
If you change the VM size or use Spot Instance, please uninstall and install connector.
Step 3: Install the connector
On the connector list, click on Install
.
Click on Install
.
Scrapoxy will start, install and stop the VM to create a custom image.
WARNING
Do not reuse an installation from a prior setup: Scrapoxy embeds a unique TLS certificate within the instance image to secure communication and communication with the Master will fail.
When the installation is finished, click on Connectors
.
- Start the project;
- Start the connector.
Other: Uninstall the connector
- Stop the connector;
- Wait for proxies to be removed.
On the connector list, click on Uninstall
.
Confirm the uninstallation.
Wait for the uninstallation to finish: Scrapoxy will delete the custom image.