Scrapoxy

Appearance

Changelog ​

4.17.4

This release doesn't include any code changes. Simply update the repository!

BREAKING CHANGE

Please update your Docker image from fabienvauchelles/scrapoxy to scrapoxy/scrapoxy

Repositories

  • github: change repository from fabienvauchelles/scrapoxy to scrapoxy/scrapoxy

  • dockerhub: change repository from fabienvauchelles/scrapoxy to scrapoxy/scrapoxy

4.17.3

Features

  • gcp: add smaller machine types (thanks runa!)

4.17.2

Bug fixes

  • iproyal: add migration for file storage

  • website: documentation adjustments

4.17.1

Features

  • connection: respect keep-alive timeout for MITM connections

Bug Fixes

  • iproyal: remove API dependency for residential connection

4.17.0

Features

  • geonode: add Geonode proxies support

Bug fixes

  • freeproxies: change the way of generating ID hash

  • misc: minor fixes

4.16.1

Bug fixes

  • iproyal: update connector to follow the new API

4.16.0

Features

  • zyte: add support of the Zyte API in Proxy Mode

4.15.2

Bug fixes

  • master: keep non-scrapoxy request cookies with MITM

4.15.1

Features

  • zyte: customize URL of the Zyte Smart Proxy Manager

Bug fixes

  • frontend: fix button alignment introduces with Safari support

4.15.0

Features

  • tls: add an option to shuffle the ciphersuite to avoid TLS fingerprint

4.14.0

Features

  • headers: maintain the order and duplication of HTTP headers for MITM mode

Bug fixes

  • privacy: remove all Scrapoxy headers and cookies from responses sent to the target website.

  • connectors: prevent unnecessary data sharing with the frontend.

  • refactor: clean a lot of code

4.13.0

Features

Bug fixes

  • errors: add missing 557 HTTP status code.

4.12.0

BREAKING CHANGE

Possible breaking change on HTTP status code (see below)

Features

  • Possible breaking change:

  • errors: non-authentication errors now return a 557 HTTP status code.

Bug fixes

  • website: add documentation

  • idea: add run config for backend and frontend

  • frontend: fix card button for Safari browser

4.11.2

Bug fixes

  • gcp: upgrade template image to debian-12

4.11.1

Features

  • smartproxy: added support for Smartproxy datacenter pay per GB product

4.11.0

Features

  • smartproxy: added support for Smartproxy

  • task: enabled naming tasks for better identification

Bug fixes

  • test: fixed issues with fingerprints in tests.

  • misc: updated and corrected some documentation and naming conventions

4.10.2

Bug fixes

  • transport: fix usage of username & password for SOCKS proxy

4.10.1

Features

  • azure: add support for Spot Instance (thanks to Julien Maitrehenry)

  • fingerprint: use GET instead of POST method to access fingerprint server (to bypass firewall restrictions)

Bug fixes

  • azure: handle running deployment to avoid duplicate instances creation

4.10.0

Features

  • brightdata: add support for Residential and Mobile proxies for BrightData

  • fingerprint: use GET instead of POST method to access fingerprint server (to bypass firewall restrictions)

Bug fixes

  • website: misc fixes

4.9.0

Features

Bug fixes

  • sticky: correct cookie injection for sticky sessions

4.8.1

Bug fixes

  • ninjasproxy: update the API url

4.8.0

Features

  • success rate: add success rate metrics to proxies and project

Bug fixes

  • gcp: fix copy/paste with GCP private key

4.7.2

Bug fixes

  • freeproxy: include auth info in key generation

4.7.1

Bug fixes

  • frontend: correct proxy URL parsing

  • frontend: show proxies error message

  • transport: set transport type to proxy level

4.7.0

Features

  • brightdata: add BrightData connector for ISP and DC proxies

  • backend: add Swagger endpoint for scraper API

  • frontend: show coupon in marketplace

Bug fixes

  • misc: many small corrections

4.6.2

Features

  • frontend: add coupon field in marketplace

  • proxy-seller: add proxy-seller coupon

4.6.1

Features

  • master: add HTTPS support to master

4.6.0

Features

Bug fixes

  • metrics: include request and response headers size in metrics

  • website: add stickies session documentation

4.5.2

Bug fixes

  • aws: use the correct architecture to install and create instances

  • log: don't throw error if there is no configuration file at startup

4.5.1

Features

  • frontend: auto-generate a free name at credential and connector creation

  • website: add scrapy-impersonate documentation to scrapy

Bug fixes

  • aws: add new instance type like t4g.nano

  • freeproxy: allow bracket in URLs

  • transport: add missing timeout on connections

  • log: add more log at startup

  • website: reorganize the documentation tree

4.5.0

Features

  • freeproxies: collect freeproxies from an URL at specified interval

  • proxies: make timeout configurable to detect online/offline status of proxies

  • proxies: make timeout configurable for removing offline proxies

  • freeproxies: add a configurable timeout to detect online/offine status of freeproxies

  • freeproxies: add a configurable timeout for removing offline freeproxies

  • api: add an API endpoint to manage freeproxies and sources

Refactor

  • connectors: merge all connectors into 3 fixed directory (common, frontend, backend)

  • storage: merge all storages into 2 fixed directory (common, backend)

  • auth: merge all authentications into 2 fixed directory (common, backend)

  • website: reorganize website and documentation structure for connectors

  • website: manage plural text in toasts

Bug fixes

  • linter: fix linter for HTML and CSS

  • frontend: correct pagination component

  • socks: allow socks in connect tunnel mode

4.4.2

Bug fixes

  • proxy-cheap: fix website url

4.4.1

Bug fixes

  • github: add script to build and publish docker image and npm package

4.4.0

Features

  • proxy-cheap: add Proxy-Cheap connectors

  • hypeproxy: add HypeProxy connector

4.3.0

Features

  • nimble: add Nimble connector with Nimble IP

Bug fixes

  • refactor: rename cloud to datacenter

  • frontend: fix clipboard issue with old browser

  • misc: correct some naming, documentation

  • storage: allow empty file for file storage

4.2.3

Bug fixes

  • commander: log commander http error on console

  • frontend: more stable clipboard copy

4.2.2

Bug fixes

  • standalone: force standalone mode when no option is selected

4.2.1

Features

  • fingerprint: add useragent for all http calls (fingerprint and api)

  • frontend: split token in username and password in project settings

Bug fixes

  • package.json: add missing socks package

4.2.0

Features

  • freeproxies: support HTTPS proxies

  • freeproxies: support SOCKS4/SOCKS5 proxies

Bug fixes

  • protocol: Accept only HTTP and HTTPS requests from master

4.1.6

Bug fixes

  • frontend: correct pagination component

  • doc: replace local search with algolia

  • doc: misc documentation fixes

4.1.5

Bug fixes

  • auth: secure flag for the authentication cookie is now optional

  • doc: usage of username:password is better explained

4.1.4

Bug fixes

  • fingerprint: better fingerprint

4.1.3

Bug fixes

  • doc: correct link to picture

  • backend-app: add version to package-lock.json in build

4.1.2

Bug fixes

  • doc: add warning for xproxy installation

  • doc: improve documentation and website

  • eslint: merge scrapoxy-linter repository into scrapoxy repository

4.1.1

Bug fixes

  • iproyal: add migration layer for file storage

4.1.0

Features

  • iproyal: add support for Royal Residential proxies

  • marketplace: add marketplace menu with static proxies, dynamic proxies, hardware, datacenter providers and other

Bug fixes

  • transport: factorize code for residential proxies

4.0.1

Bug fixes

  • master: handle writeHead error after aborted connection

4.0.0

Scrapoxy 4 represents a complete rewrite, now developed in Typescript. It harnesses the capabilities of NestJS and Angular modules to enhance its functionality and efficiency.

BREAKING CHANGE

Scrapoxy 4 is not compatible with previous versions.

Features

  • datacenter providers: it supports main datacenter providers like AWS, Azure, GCP and more;

  • proxy services: it is compatible with proxy provider like Zyte, IPRoyal, Rayobyte and more;

  • hardware materials: it orchestrates 4G proxy farms hardware types, like Proxidize or XProxy.io;

  • free proxy lists: it handles manual lists of HTTP/HTTPS proxies;

  • auto-scale: it autoscale up and down proxies to optimize costs;

  • auto-rotate: it rotates proxies to avoid bans;

  • sticky sessions: it supports sticky sessions to keep the same IP address for a scraping session or a browser

  • traffic interception: it intercepts HTTP and HTTPS traffic to inject consistent headers or cookies;

  • monitoring: it monitors proxies and provides many statistics.

3.1.1

Bug fixes

  • master: use correctly writeEnd in socket and request (thanks to Ben Lavalley)

3.1.0

Features

  • mitm: decrypt & encrypt SSL requests to add headers (like x-cache-proxyname). Compatible with HTTPS requests in PhantomJS.

  • domains: manage whitelist or blacklist for urls (idea from Jonathan Wiklund)

  • docs: add ami-485fbba5 with type t2.micro

Bug fixes

  • logs: correct export path of logs

  • docs: correct documentation

  • ssl: add servername in the TLS connect (bug with HELLO)

  • pinger: use reject instead of throw error (crash program). Thanks to Anis Gandoura.

3.0.1

Features

  • digitalocean: support Digital Ocean tags on Droplets. Thanks to Ben Lavalley.

Bug fixes

  • digitalocean: use new image size (s-1vcpu-1gb instead of 512mb)

3.0.0

BREAKING CHANGE

The configuration of providers changes.

Features

  • providers: uses multiple providers at a time

  • awsec2: provider removes instances in batch every second (and no longer makes thousands of queries)

  • ovhcloud: provider creates instances in batch (new API route used)

Bug fixes

  • maxRunningInstances: remove blocking parameters maxRunningInstances

2.4.3

Bug fixes

  • node: change minimum version of Node.js to 8

  • dependencies: upgrade dependencies to latest version

2.4.2

Bug fixes

  • useragent: set useragent at instance creation, not at startup

  • instance: force crashed instance to be removed

2.4.1

Bug fixes

  • instance: correctly remove instance when instance is removed. Thanks to Étienne Corbillé.

2.4.0

Features

  • provider: add VScale.io provider. Thanks to Hotrush.

Bug fixes

  • proxy: use a valid startup script for init.d. Thanks to Hotrush.

  • useragent: change useragents with a fresh list for 2017

2.3.10

Features

  • docs: add ami-06220275 with type t2.nano

Bug fixes

  • instance: remove listeners on instance alive status on instance removal. Thanks to Étienne Corbillé.

2.3.9

Features

  • digitalocean: update Digital Ocean documentation

  • digitalocean: view only instances from selected region

  • instances: remove random instances instead of the last ones

  • pm2: add kill_timeout option for PM2 (thanks to cp2587)

Bug fixes

  • digitalocean: limit the number of created instances at each API request

  • digitalocean: don't remove locked instances

2.3.8

Features

  • docker: create the Docker image scrapoxy/scrapoxy

Bug fixes

  • template: limit max instances to 2

2.3.7

Features

  • connect: scrapoxy accepts now full HTTPS CONNECT method. It is useful for browser like PhantomJS. Thanks to Anis Gandoura

2.3.6

Bug fixes

  • template: replace old AWS AMI by ami-c74d0db4

2.3.5

BREAKING CHANGE

Please rebuild instance image.

Features

  • instance: change Node.js version to 6.x

  • ping: use an HTTP ping instead a TCP ping.

2.3.4

Features

  • stats: monitor stop count history

  • stats: add 3 more scales: 5m, 10m and 1h

  • logs: normalize logs and add more informations

  • scaling: pop a message when maximum number of instances is reached in a provider

  • scaling: add quick scaling buttons

  • docs: explain why Scrapoxy doesn't accept CONNECT mode

  • docs: explain how User Agent is overwritten

Bug fixes

  • dependencies: upgrade dependencies

  • ovh: monitor DELETED status

  • docs: add example to test scrapoxy with credentials

  • commander: manage twice instance remove

2.3.3

Bug fixes

  • master: sanitize bad request headers

  • proxy: catch all socket errors in the proxy instance

2.3.2

Bug fixes

  • docs: fallback to markdown for README (because npmjs doesn't like retext)

2.3.1

Features

  • docs: add tutorials for Scrapy and Node.js

Bug fixes

  • digitalocean: convert Droplet id to string

2.3.0

Features

  • digitalocean: add support for DigitalOcean provider

2.2.1

Misc

  • config: rename my-config.json to conf.json

  • doc: migrate documentation to ReadTheDocs

  • doc: link to the new website Scrapoxy.io

2.2.0

BREAKING CHANGE

node minimum version is now 4.2.1, to support JS class

Features

  • all: migrate core and gui to ES6, with all best practices

  • api: replace Express by Koa

Bug fixes

  • test: correct core e2e test

2.1.2

Bug fixes

  • gui: correct token encoding for GUI

2.1.1

Bug fixes

  • main: add message when all instances are stopped (at end)

  • doc: correct misc stuff in doc

2.1.0

Features

  • ovh: add OVH provider with documentation

  • security: add basic auth to Scrapoxy (RFC2617)

  • stats: add flow stats

  • stats: add scale for stats (1m/1h/1d)

  • stats: store stats on server

  • stats: add globals stats

  • doc: split of the documentation in 3 parts: quick start, standard usage and advanced usage

  • doc: add tutorials for AWS / EC2

  • gui: add a scaling popup instead of direct edit (with integrity check)

  • gui: add update popup when the status of an instance changes.

  • gui: add error popup when GUI cannot retrieve data

  • logs: write logs to disk

  • instance: add cloud name

  • instance: show instance IP

  • instance: always terminate an instance when stopping (prefer terminate instead of stop/start)

  • test: allow more than 8 requests (max 1000)

  • ec2: force to terminate/recreate instance instead of stop/restart

Bug fixes

  • gui: emit event when scaling is changed by engine (before, event was triggered by GUI)

  • stability: correct a lot of behavior to prevent instance cycling

  • ec2: use status name instead of status code

2.0.1

Features

  • test: specify the count of requests with the test command

  • test: count the requests by IP in the test command

  • doc: add GUI documentation

  • doc: add API documentation

  • doc: explain awake/asleep mode in user manual

  • log: add human readable message at startup

2.0.0

BREAKING CHANGE

API routes are prefixed with /api

Features

  • gui: add GUI to control Scrapoxy

  • gui: add statistics to the GUI (count of requests / minute, average delay of requests / minute)

  • doc: add doc about HTTP headers

1.1.0

Features

  • commander: stopping an instance returns the new count of instances

  • commander: password is hashed with base64

  • commander: read/write config with command (and live update of the scaling)

Misc

  • chore: force global install with NPM

1.0.2

Features

  • doc: add 2 AWS / EC2 tutorials

Bug fixes

  • template: correct template mechanism

  • config: correct absolute path for configuration

1.0.1

Misc

  • doc: change author and misc informations

1.0.0

Features

  • init: start of the project

Released under the AGPLv3 License.