Microsoft Azure Connector

Microsoft Azure is a cloud computing service created by Microsoft.

Prerequisites

An active Microsoft Azure subscription is required.

Azure Portal

Connect to Azure Portal.

Step 1: Register a new application

Search the Microsoft Entra ID (aka Active Directory).

---

1. Click on App registrations;
2. And click New registration.

---

Complete the form with the following information:

• Name: scrapoxy
• Supported account types: Accounts in this organizational directory only (Microsoft only - Single tenant)

And click on Register.

---

Remember:

1. the Application (client) ID;
2. and the Directory (tenant) ID.

Step 2: Create a client secret

1. Click on Certificates & secrets;
2. Click on New client secret;
3. Enter a description, select the maximum expiration time;
4. Click on Add.

---

Remember the Value of the secret.

Step 3: Get your subscription ID

Search the Subscription.

---

Select the first subscription.

---

Remember the Subscription ID value.

Step 4: Add a role to the application

On the subscription:

1. Click on Access control (IAM);
2. And click on Add.

---

1. On tab Role;
2. Select Privileged administrator roles;
3. Select Contributor;
4. Click on Next.

---

1. On tab Members;
2. Click on Select members;
3. Select the application you created;
4. Click on Select:

---

Finally, click on Review + assign 2 times.

Scrapoxy

Open Scrapoxy User Interface and select Marketplace:

Step 1: Create a new credential

Select Azure to create a new credential (use search if necessary).

---

Complete the form with the following information:

1. Name: The name of the credential;
2. Application (client) ID: The Application ID (aka Client ID) of the application;
3. Directory (tenant) ID: The Directory ID (aka Tenant ID) of the application;
4. Secret value: The Client Secret of the application;
5. Subscription ID: The Subscription ID.

And click on Create.

INFO

It may take Azure up to 10 minutes to reflect the changes.

Step 2: Create a new connector

Create a new connector:

Complete the form with the following information:

1. Credential: The previous credential;
2. Name: The name of the connector;
3. # Proxies: The number of instances to create;
4. Proxies Timeout: Maximum duration for connecting to a proxy before considering it as offline;
5. Proxies Kick: If enabled, maximum duration for a proxy to be offline before being removed from the pool;
6. Location: The region where the instances will be created;
7. Port: The port of the proxy (on Azure);
8. Resource Group Name: The resource group to host the instances;
9. VM size: The size of the VM;
10. Spot Instance: If enabled, the VM will be a spot instance;
11. Storage Type: The type of storage;
12. Prefix: The prefix for all resources created on Azure;
13. Image Resource Group Name: The resource group where the image is stored.

And click on Create.

Most default values can be retained if suitable for the use case.

WARNING

When setting up the connector in multiple locations, assign a unique Resource Group Name, Image Resource Group Name and distinct Prefix for each location. Without this, connectors may interfere with each other, shutting down instances from the same provider.

Step 3: Start the connector

1. Start the project;
2. Start the connector.

Step 4: Start the connector (optional)

1. Stop the connector;
2. Wait for proxies to be removed.