Skip to content

Microsoft Azure Connector

Azure

Microsoft Azure is a cloud computing service created by Microsoft.

Prerequisites

An active Microsoft Azure subscription is required.

Azure Portal

Connect to Azure Portal.

Step 1: Register a new application

Search Entra

Search the Microsoft Entra ID (aka Active Directory).


New Registration

  1. Click on App registrations;
  2. And click New registration.

App register

Complete the form with the following information:

  • Name: scrapoxy
  • Supported account types: Accounts in this organizational directory only (Microsoft only - Single tenant)

And click on Register.


App Info

Remember:

  1. the Application (client) ID;
  2. and the Directory (tenant) ID.

Step 2: Create a client secret

App Secret

  1. Click on Certificates & secrets;
  2. Click on New client secret;
  3. Enter a description, select the maximum expiration time;
  4. Click on Add.

App Secret Value

Remember the Value of the secret.

Step 3: Get your subscription ID

Subscription Search

Search the Subscription.


Subscription Select

Select the first subscription.


Subscription Info

Remember the Subscription ID value.

Step 4: Add a role to the application

IAM Add

On the subscription:

  1. Click on Access control (IAM);
  2. And click on Add.

IAM Role

  1. On tab Role;
  2. Select Privileged administrator roles;
  3. Select Contributor;
  4. Click on Next.

IAM Member

  1. On tab Members;
  2. Click on Select members;
  3. Select the application you created;
  4. Click on Select:

IAM Review

Finally, click on Review + assign 2 times.

Scrapoxy

Open Scrapoxy User Interface and select Marketplace:

Step 1: Create a new credential

Credential Select

Select Azure to create a new credential (use search if necessary).


Credential Form

Complete the form with the following information:

  1. Name: The name of the credential;
  2. Application (client) ID: The Application ID (aka Client ID) of the application;
  3. Directory (tenant) ID: The Directory ID (aka Tenant ID) of the application;
  4. Secret value: The Client Secret of the application;
  5. Subscription ID: The Subscription ID.

And click on Create.

INFO

It may take Azure up to 10 minutes to reflect the changes.

Step 2: Create a new connector

Create a new connector:

Connector Create

Complete the form with the following information:

  1. Credential: The previous credential;
  2. Name: The name of the connector;
  3. # Proxies: The number of instances to create;
  4. Proxies Timeout: Maximum duration for connecting to a proxy before considering it as offline;
  5. Proxies Kick: If enabled, maximum duration for a proxy to be offline before being removed from the pool;
  6. Location: The region where the instances will be created;
  7. Port: The port of the proxy (on Azure);
  8. Resource Group Name: The resource group to host the instances;
  9. VM size: The size of the VM;
  10. Spot Instance: If enabled, the VM will be a spot instance;
  11. Storage Type: The type of storage;
  12. Prefix: The prefix for all resources created on Azure;
  13. Image Resource Group Name: The resource group where the image is stored.

And click on Create.

Most default values can be retained if suitable for the use case.

WARNING

When setting up the connector in multiple locations, assign a unique Resource Group Name, Image Resource Group Name and distinct Prefix for each location. Without this, connectors may interfere with each other, shutting down instances from the same provider.

If you change the VM size or use Spot Instance, please uninstall and install connector.

Step 3: Install the connector

Connector Install Select

On the connector list, click on Install.


Connector Install

Click on Install.

Scrapoxy will start, install and stop the VM to create a custom image.

WARNING

Do not reuse an installation from a prior setup: Scrapoxy embeds a unique TLS certificate within the instance image to secure communication and communication with the Master will fail.


Connector Installed

When the installation is finished, click on Connectors.


Connector Start

  1. Start the project;
  2. Start the connector.

Other: Uninstall the connector

Connector Stop

  1. Stop the connector;
  2. Wait for proxies to be removed.

Connector Uninstall Select

On the connector list, click on Uninstall.


Connector Uninstalled

Confirm the uninstallation.

Connector Uninstalled

Wait for the uninstallation to finish: Scrapoxy will delete the custom image.

Released under the AGPLv3 License.