HTTP代理使用样例¶

本文档包含编程请求http代理服务器的代码样例，供开发者参考。

样例使用说明

代码样例不能直接运行，因为代码中的订单号1834987042xxxxxx、代理IP和端口号117.69.63.102:43787、
用户名username、密码password都是虚构的，请替换成您自己的信息。
代码样例正常运行所需的运行环境和注意事项在样例末尾均有说明，使用前请仔细阅读。
使用代码样例过程中遇到问题请联系售后客服，我们会为您提供技术支持。

注意事项

以下样例均为基础案例,运行是并不能保证能成功爬取到目标网站,目标网站通常具有反爬重机制,如跳转时需要输入验证码的页面
建议您在开发过程中基于基础样例进行如下改进

添加IP池管理
合理控制对目标网站的请求频率，建议对同一网站1个代理IP每秒请求不超过1次；
发出的http请求尽可能带上完整的header信息。

Python3¶

request¶

request (推荐)

使用提示

基于requests的代码样例支持访问http,https网页，推荐使用
requests不是python原生库，需要安装才能使用: pip install requests

"""
使用requests请求代理服务器
请求http和https网页均适用
"""

import requests

# 提取代理API接口，获取1个代理IP
api_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006"

# 获取API接口返回的代理IP
proxy_ip = requests.get(api_url).text

# 验证方式：
# API返回账号密码验证
proxy_list = proxy_ip.split(":")
proxy_ip = proxy_list[0] + ":" + proxy_list[1]
username = proxy_list[2]
password = proxy_list[3]

# 账密固定
# username = "username"
# password = "password"

proxies = {
"http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
"https": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
}

# 验证方式：IP白名单（需提前设置白名单）
# proxies = {
#     "http": "http://%(proxy)s/" % {"proxy": proxy_ip},
#     "https": "http://%(proxy)s/" % {"proxy": proxy_ip},
# }

# 要访问的目标网页
target_url = "http://v2.api.juliangip.com/v2/dps/ip_search"

# 使用代理IP发送请求
response = requests.get(target_url, proxies=proxies)

# 获取页面内容
if response.status_code == 200:
print(response.text)

UrlLib¶

UrlLib

使用提示

基于UrlLib的代码样例同时支持访问http,https网页
运行环境要求Python3.x

  #!/usr/bin/env python
  # -*- coding: utf-8 -*-

  """
  使用urllib请求代理服务器
  请求http和https网页均适用
  """

  import urllib.request
  import ssl

  # 全局取消证书验证，避免访问https网页报错
  ssl._create_default_https_context = ssl._create_unverified_context

  # 提取代理API接口，获取1个代理IP
  api_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006"

  # 获取API接口返回的IP
  proxy_ip = urllib.request.urlopen(api_url).read().decode('utf-8')

  # 验证方式：
  # API返回账号密码验证
  proxy_list = proxy_ip.split(":")
  proxy_ip = proxy_list[0] + ":" + proxy_list[1]
  username = proxy_list[2]
  password = proxy_list[3]

  # 账密固定
  # username = "username"
  # password = "password"

  proxies = {
      "http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
  }

  # 验证方式：IP白名单（需提前设置白名单）
  # proxies = {
  #     "http": "http://%(proxy)s/" % {"proxy": proxy_ip},
  # }

  # 要访问的目标网页
  target_url = "http://v2.api.juliangip.com/v2/dps/ip_search"

  # 使用代理IP发送请求
  proxy_support = urllib.request.ProxyHandler(proxies)
  opener = urllib.request.build_opener(proxy_support)
  urllib.request.install_opener(opener)
  response = urllib.request.urlopen(target_url)

  # 获取页面内容
  if response.code == 200:
      print(response.read().decode('utf-8'))

AioHttp¶

AioHttp

使用提示

基于aiohttp的代码样例支持访问http,https网页
aiohttp不是python原生库，需要安装才能使用: pip install aiohttp
aiohttp只支持Python3.5及以上
Windows系统Python3.8使用aiohttp访问https网站会抛出异常，在import asyncio后调用 asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())即可解决。
aiohttp 抛出 ssl 错误，在 aiohttp 包的文件中找到 connector.py 这个文件(在 aiohttp 的根目录下)，将 runtime_has_start_tls = self._loop_supports_start_tls() 这一行注释掉，然后在其下面新增一行 runtime_has_start_tls = False if req.proxy.scheme != "https" else self._loop_supports_start_tls() 即可（Python 3.13.5 已修复）

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
使用aiohttp请求代理服务器
请求http和https网页均适用

"""
import random
import asyncio
# asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) windows系统请求https网站报错时调用此方法

import aiohttp
import requests

page_url = "http://v2.api.juliangip.com/v2/dps/ip_search"  # 要访问的目标网页

# 提取代理API接口
api_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006"

# 获取API接口返回的代理IP
proxy_ip = requests.get(api_url).text()

# API返回账号密码验证
proxy_list = proxy_ip.split(":")
proxy_ip = proxy_list[0] + ":" + proxy_list[1]
username = proxy_list[2]
password = proxy_list[3]

# 账密固定
# username = "username"
# password = "password"

proxy_auth = aiohttp.BasicAuth(username, password)

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
}
async with session.get(url, headers=headers, ssl=False, proxy=ip_port, proxy_auth=proxy_auth, timeout=10) as response:
    return await response.text(), response.status

async def main():
    try:
        async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(ssl=False), trust_env=True) as session:
            content, status = await fetch(session, 'https://www.baidu.com')
            print(f"status: {status}")
            print(f"content: {content}")
    except aiohttp.client_exceptions.ClientConnectorError as e:
        print(f'连接代理或目标服务器时发生错误: {e}')

if __name__ == '__main__':
    policy = asyncio.WindowsSelectorEventLoopPolicy()
    asyncio.set_event_loop_policy(policy)
    asyncio.run(main())

httpx¶

httpx

使用提示

基于httpx的代码样例支持访问http,https网页
httpx不是python原生库，需要安装才能使用: pip install httpx
httpx运行环境要求 Python3.8+
httpx暂时还不支持SOCKS代理
httpx的版本为：0.28.0 (注意：httpx的版本0.27.2版本不同写法不同，详情请查看httpx文档)

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
使用requests请求代理服务器
请求http和https网页均适用
"""

import random
import asyncio

import httpx
import requests

page_url = "http://v2.api.juliangip.com/v2/dps/ip_search"  # 要访问的目标网页

# 提取代理API接口
api_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006"

# 获取API接口返回的代理IP
proxy_ip = requests.get(api_url).text()

# API返回账号密码验证
proxy_list = proxy_ip.split(":")
proxy_ip = proxy_list[0] + ":" + proxy_list[1]
username = proxy_list[2]
password = proxy_list[3]

# 账密固定
# username = "username"
# password = "password"

# httpx的版本是0.27.2
# proxies = {
#     "http://": f"http://{username}:{password}@{proxy_ip}",
#     "https://": f"http://{username}:{password}@{proxy_ip}",
# }
# response = httpx.get(target_url, headers=headers, proxies=proxies, verify=False)
# print(f"Status Code: {response.status_code}")
# print(f"Response Text: {response.text}")

# 这是 httpx的0.28.0的版本写法
proxy = httpx.Proxy(
    url=f"http://{proxy_ip}",
    auth=(username, password),
)

with httpx.Client(proxy=proxy, headers=headers, verify=False, timeout=10) as client:
response = client.get(target_url)
print(f"Status Code: {response.status_code}")
print(f"Response Text: {response.text}")

websocket¶

websocket(长连接版)

使用提示

安装运行所需的客户端: pip install websocket-client
使用HTTP代理发送websocket请求
在IP可用的情况下，客户端长时间不发送消息，服务端会断开连接
运行环境要求 python3.x

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""
使用HTTP代理发送websocket请求
"""
import gzip
import zlib

import websocket

OPCODE_DATA = (websocket.ABNF.OPCODE_TEXT, websocket.ABNF.OPCODE_BINARY)

url = "ws://echo.websocket.org/"

proxies = {
    "http_proxy_host": "117.69.63.102",
    "http_proxy_port": 43787,
    "http_proxy_auth": ("username", "password"),
}

ws = websocket.create_connection(url, **proxies)

def recv():
    try:
        frame = ws.recv_frame()
    except websocket.WebSocketException:
        return websocket.ABNF.OPCODE_CLOSE, None
    if not frame:
        raise websocket.WebSocketException("Not a valid frame %s" % frame)
    elif frame.opcode in OPCODE_DATA:
        return frame.opcode, frame.data
    elif frame.opcode == websocket.ABNF.OPCODE_CLOSE:
        ws.send_close()
        return frame.opcode, None
    elif frame.opcode == websocket.ABNF.OPCODE_PING:
        ws.pong(frame.data)
        return frame.opcode, frame.data

    return frame.opcode, frame.data

def recv_ws():
    opcode, data = recv()
    if opcode == websocket.ABNF.OPCODE_CLOSE:
        return
    if opcode == websocket.ABNF.OPCODE_TEXT and isinstance(data, bytes):
        data = str(data, "utf-8")
    if isinstance(data, bytes) and len(data) > 2 and data[:2] == b'\037\213':  # gzip magick
        try:
            data = "[gzip] " + str(gzip.decompress(data), "utf-8")
        except Exception:
            pass
    elif isinstance(data, bytes):
        try:
            data = "[zlib] " + str(zlib.decompress(data, -zlib.MAX_WBITS), "utf-8")
        except Exception:
            pass
    if isinstance(data, bytes):
        data = repr(data)

    print("< " + data)

def main():
    print("Press Ctrl+C to quit")
    while True:
        message = input("> ")
        ws.send(message)
        recv_ws()

if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        print('\nbye')
    except Exception as e:
        print(e)

websocker(短连接版)

使用提示

安装运行所需的客户端: pip install websocket-client
使用HTTP代理发送websocket请求
运行环境要求 python3.x

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

import ssl
import websocket


def on_message(ws, message):
    print(message)


def on_error(ws, error):
    print(error)


def on_open(ws):
    data = '{}'  # 此处填入您需要传给目标网站的json格式参数，如{"type":"web","data":{"_id":"xxxx"}}
    ws.send(data)

def on_close(*args):
    print("### closed ###")

proxies = {
    "http_proxy_host": "117.69.63.102",
    "http_proxy_port": 43787,
    "http_proxy_auth": ("username", "password"),
}

def start():
    websocket.enableTrace(True)
    target_url = 'ws://127.0.0.1:5000/socket.io/?EIO=4&transport=websocket'  # 此处替换您的目标网站
    ws = websocket.WebSocketApp(
        url = target_url,
        header = [
            "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"
        ],
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )
    ws.on_open = on_open
    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE}, **proxies)


if __name__ == "__main__":
    start()

ProxyPool¶

ProxyPool

使用提示

此样例是动态代理简单IP池管理的实现
requests不是python原生库，需要安装才能使用: pip install requests
支持Python2.7和Python3

#!/usr/bin/env python
# -*- encoding: utf-8 -*-


import time
import random
import threading
import requests

class ProxyPool():

    def __init__(self, proxy_count):
        self.proxy_count = proxy_count if proxy_count < 50 else 50 # 池子维护的IP总数，建议一般不要超过50
        self.alive_proxy_list = []  # 活跃IP列表
        self.get_ip_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=json&trade_no=183498337042738379&sign=686f43f4b2d89d74c1680577d4ffbe47"

    def _fetch_proxy_list(self, count):
        try:
            proxy_list = []
            for i in range(count):
                res = requests.get(self.get_ip_url)
                res_data = res.json().get('data', {})
                res_proxy_list = res_data.get('proxy_list', [])
                for proxy_ip in res_proxy_list:
                    proxy_list.append(proxy_ip)
            return proxy_list
        except:
            print("API获取IP异常，请检查订单")
        return []

    def _init_proxy(self):
        """初始化IP池"""
        self.alive_proxy_list = self._fetch_proxy_list(self.proxy_count)

    def add_alive_proxy(self, add_count):
        """导入新的IP, 参数为新增IP数"""
        self.alive_proxy_list.extend(self._fetch_proxy_list(add_count))

    def get_proxy(self):
        """从IP池中获取IP"""
        return random.choice(self.alive_proxy_list) if self.alive_proxy_list else ""

    def run(self):
        sleep_seconds = 1
        self._init_proxy()
        while True:
            for proxy in self.alive_proxy_list:
                proxy[1] = float(proxy[1]) - sleep_seconds  # proxy[1]代表此IP的剩余可用时间
                if proxy[1] <= 3:
                    self.alive_proxy_list.remove(proxy)  # IP还剩3s时丢弃此IP
            if len(self.alive_proxy_list) < self.proxy_count:
                self.add_alive_proxy(self.proxy_count - len(self.alive_proxy_list))
            time.sleep(sleep_seconds)

    def start(self):
        """开启子线程更新IP池"""
        t = threading.Thread(target=self.run)
        t.setDaemon(True)  # 将子线程设为守护进程，主线程不会等待子线程结束，主线程结束子线程立刻结束
        t.start()


def parse_url(proxy):
    # 用户名密码认证(动态代理/独享代理)
    username = "username"
    password = "password"
    proxies = {
        "http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password,"proxy": proxy},
        "https": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password,"proxy": proxy},
    }

    # 白名单方式（需提前设置白名单）
    # proxies = {
    #     "http": "http://%(proxy)s/" % {"proxy": proxy_ip},
    #     "https": "http://%(proxy)s/" % {"proxy": proxy_ip},
    # }

    # 要访问的目标网页
    target_url = "http://v2.api.juliangip.com/v2/dps/ip_search"
    # 使用代理IP发送请求
    response = requests.get(target_url, proxies=proxies)
    # 获取页面内容
    if response.status_code == 200:
        print(response.text)


if __name__ == '__main__':
    proxy_pool = ProxyPool(30) # 池子中维护的IP数
    proxy_pool.start()
    time.sleep(1)  # 等待IP池初始化

    proxy = proxy_pool.get_proxy() # 从IP池中提取IP
    if proxy:
        parse_url(proxy)

Python2¶

request¶

request (推荐)

使用提示

基于requests的代码样例支持访问http,https网页，推荐使用
requests不是python原生库，需要安装才能使用: pip install requests

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
使用requests请求代理服务器
请求http和https网页均适用
"""

import requests

# 提取代理API接口，获取1个代理IP
api_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006"

# 获取API接口返回的代理IP
proxy_ip = requests.get(api_url).text

# 验证方式：
# API返回账号密码验证
proxy_list = proxy_ip.split(":")
proxy_ip = proxy_list[0] + ":" + proxy_list[1]
username = proxy_list[2]
password = proxy_list[3]

# 账密固定
# username = "username"
# password = "password"
proxies = {
    "http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
    "https": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
}

# 验证方式：IP白名单（需提前设置白名单）
# proxies = {
#     "http": "http://%(proxy)s/" % {"proxy": proxy_ip},
#     "https": "http://%(proxy)s/" % {"proxy": proxy_ip},
# }

# 要访问的目标网页
target_url = "http://v2.api.juliangip.com/v2/dps/ip_search"

# 使用代理IP发送请求
response = requests.get(target_url, proxies=proxies)

# 获取页面内容
if response.status_code == 200:
    print response.text

UrlLib2¶

UrlLib2

使用提示

基于urllib2的代码样例同时支持访问http和https网页
运行环境要求 python2.6 / 2.7

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
使用urllib2请求代理服务器
请求http和https网页均适用
"""

import urllib2
import ssl

# 全局取消证书验证，避免访问https网页报错
ssl._create_default_https_context = ssl._create_unverified_context  

# 提取代理API接口，获取1个代理IP
api_url = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006"

# 获取API接口返回的IP
proxy_ip = urllib2.urlopen(api_url).read()

# 验证方式：
# API返回账号密码验证
proxy_list = proxy_ip.split(":")
proxy_ip = proxy_list[0] + ":" + proxy_list[1]
username = proxy_list[2]
password = proxy_list[3]

# 账密固定
# username = "username"
# password = "password"
proxies = {
    "http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
}

# 验证方式：IP白名单（需提前设置白名单）
# proxies = {
#     "http": "http://%(proxy)s/" % {"proxy": proxy_ip},
# }

# 要访问的目标网页
target_url = "http://v2.api.juliangip.com/v2/dps/ip_search"

# 使用代理IP发送请求
proxy_support = urllib2.ProxyHandler(proxies)
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)
response = urllib2.urlopen(target_url)

# 获取页面内容
if response.code == 200:
    print response.read()

Python-Selenium¶

Chrome¶

Chrome (IP白名单,推荐)

使用提示

基于白名单方式使用Selenium+Chrome认证代理
运行环境要求python⅔ + selenium + Chrome + Chromedriver + Windows/Linux/macOS
下载chromedriver（注意chromedriver版本要和Chrome版本对应
selenium不是python原生库，需要安装才能使用：pip install selenium。selenium本身并不支持账密认证，只能使用IP白名单方式
请注意替换代码中的部分信息：
${ip:port}：代理IP:端口号，如："117.69.63.102:43787"
${chromedriver_path}：您本机chromedriver驱动存放路径，如："C:\chromedriver.exe"

#!/usr/bin/env python
# encoding: utf-8

from time import sleep
import requests
from selenium import webdriver
from selenium.common import TimeoutException
from selenium.webdriver.chrome.service import Service

# 获取代理IP
api_url = "http://v2.api.juliangip.com/company/dynamic/getips?auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=xxxxxxxxxxxxxx&sign=xxxxxxxxxxxx"
# # 目标网址
target_url = "https://cip.cc"

# 发送请求获取IP地址
proxy_ip = requests.get(api_url).text
print('提取的IP：' + proxy_ip)

# 目前只能用白名单方式
# options = webdriver.EdgeOptions()  # 初始化Edge浏览器
options = webdriver.ChromeOptions()  # 初始化Google浏览器

# options.add_argument('--headless')  # 无界面模式
options.add_argument('--disable-gpu')  # 禁用GPU加速
# options.add_argument('--blink-settings=imagesEnabled=false')  # 禁用图片展示
options.add_argument('--proxy-server=http://' + proxy_ip)  # 设置代理

# chrome 浏览器webdriver下载地址
# https://googlechromelabs.github.io/chrome-for-testing/
service = Service(executable_path=r"C:\Program Files\Google\Chrome\Application\chromedriver.exe")
browser = webdriver.Chrome(options=options, service=service)

# edge 浏览器webdriver下载地址
# https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/?form=MA13LH#downloads
# service = Service(executable_path=r"C:\Program Files (x86)\Microsoft\Edge\Application\msedgedriver.exe")
# browser = webdriver.Edge(options=options)
browser.set_page_load_timeout(10)  # 设置页面加载超时时间
try:
    browser.get(target_url)
    page_source = browser.page_source  # 获取页面源代码
    sleep(10)
    print(page_source)
except TimeoutException:
    print("页面加载超时，请检查网络连接或目标网站响应时间。")
    except Exception as e:
    print("发生错误：", str(e))
finally:
    browser.quit()

Chrome (用户名密码认证)

使用提示

基于用户名密码方式使用Selenium + Chrome认证代理(chrome86版已测试通过)
运行环境要求python⅔ + selenium + Chrome + Chromedriver + Windows/Linux/macOS
下载chromedriver（注意chromedriver版本要和Chrome版本对应）
selenium不是python原生库，需要安装才能使用：pip install selenium
请注意替换代码中的部分信息：
${proxy_ip}：代理IP
${proxy_port}：端口号
${username}}：用户名
${password}：密码
${chromedriver_path}：您本机chromedriver驱动存放路径，如："C:\chromedriver.exe"

#!/usr/bin/env python
# encoding: utf-8

from time import sleep
import requests
from seleniumwire import webdriver

# seleniumwire的依赖包blinker的版本需要指定的1.7.0版本 最新版会报错

api_url = "http://v2.api.juliangip.com/company/dynamic/getips?auth_type=2&auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=xxxxxxxxxxxx&sign=xxxxxxxxxxxxxxx"
# 获取API接口返回的代理IP
proxy_ip = requests.get(api_url).text
print(proxy_ip)

# # 用户名密码认证
proxy_list = proxy_ip.split(":")
proxy_ip = proxy_list[0]
proxy_port = proxy_list[1]
username = proxy_list[2]
password = proxy_list[3]

# proxy_ip = 't75.juliangip.cc'
# proxy_port = '10504'
# username = '15168637998'
# password = 'rPDEE0kJ'


ip_port = proxy_ip + ":" + proxy_port
user_pass = username + ":" + password
proxyIp = f"http://{user_pass}@{ip_port}"
print(proxyIp)
proxy = {
    'proxy': {
        'http': proxyIp,
        'https': proxyIp,
        'no_proxy': '127.0.0.1',
        'verify_ssl': False  # 禁用 SSL 证书验证
    }
}
# 配置 Edge 选项
# options = webdriver.EdgeOptions()
# driver = webdriver.Edge(options=options, seleniumwire_options=proxy)

# 配置 chrome 选项
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=options, seleniumwire_options=proxy)

# 打开目标网页
driver.get('https://cip.cc')
page_source = driver.page_source
sleep(500)
driver.close()
print(page_source)

使用提示

如需使用用户名密码+无界面方式认证代理,请使用使用 Selenium + PhantomJS

PhantomJS¶

用户名密码认证+无界面模式

使用提示

基于用户名密码+无界面方式使用 Selenium + PhantomJS认证代理
运行环境要求python⅔ + selenium + PhantomJS + Windows/Linux/macOS
点此下载PhantomJS（推荐使用2.1.1版）
${executable_path}：您本机PhantomJS驱动存放路径，如："C:\phantomjs-2.1.1-windows\bin\phantomjs.exe"

#!/usr/bin/env python
# encoding: utf-8

from selenium import webdriver
import time

#先下载phantomjs包文件,再填入phantomjs.exe的路径 (路径不要包含中文)
executable_path = '${executable_path}'
service_args=[
    '--proxy=host:port', #此处替换您的代理ip,如117.69.63.102:23918
    '--proxy-type=http',
    '--proxy-auth=username:password' #用户名密码
]
driver=webdriver.PhantomJS(service_args=service_args,executable_path=executable_path)
driver.get('http://v2.api.juliangip.com/v2/dps/ip_search')

print(driver.page_source)
time.sleep(3)
driver.close()

Firefox¶

Firefox (IP白名单,推荐)

使用提示

基于白名单方式使用Selenium+Firefox认证代理
运行环境要求python⅔ + selenium + Firefox + geckodriver + Windows/Linux/macOS
下载geckodriver（注意geckodriver版本要和Firefox版本对应）
selenium不是python原生库，需要安装才能使用：pip install selenium
请注意替换代码中的部分信息：
${ip:port}：代理IP:端口号，如："117.69.63.102:43787"
${geckodriver_path}：您本机geckodriver驱动存放路径，如："C:\geckodriver.exe"

#!/usr/bin/env python
# encoding: utf-8

from selenium import webdriver
import time

fp = webdriver.FirefoxProfile()
proxy = '${ip:port}'
ip, port = proxy.split(":")
port = int(port)

# 设置代理配置
fp.set_preference('network.proxy.type', 1)
fp.set_preference('network.proxy.http', ip)
fp.set_preference('network.proxy.http_port', port)
fp.set_preference('network.proxy.ssl', ip)
fp.set_preference('network.proxy.ssl_port', port)

driver = webdriver.Firefox(executable_path="${geckodriver_path}", firefox_profile=fp)
driver.get('http://v2.api.juliangip.com/v2/dps/ip_search')

# 获取页面内容
print(driver.page_source)

# 延迟3秒后关闭当前窗口，如果是最后一个窗口则退出
time.sleep(3)
driver.close()

Firefox (用户名密码认证)

使用提示

基于白名单方式使用Selenium+Firefox认证代理
运行环境要求python⅔ + selenium + Firefox + geckodriver + Windows/Linux/macOS
下载geckodriver（注意geckodriver版本要和Firefox版本对应）
selenium不是python原生库，需要安装才能使用：pip install selenium
请注意替换代码中的部分信息：
${ip:port}：代理IP:端口号，如："117.69.63.102:43787"
${geckodriver_path}：您本机geckodriver驱动存放路径，如："C:\geckodriver.exe"

#!/usr/bin/env python
# encoding: utf-8

import time

from seleniumwire import webdriver # pip install selenium-wire

username = 'username' # 请替换您的用户名和密码
password = 'password'
proxy_ip = '117.69.63.102:43787' # 请替换您提取到的代理ip
options = {
        'proxy': {
            'http': "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip},
            'https': "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": proxy_ip}
        }
    }

driver = webdriver.Firefox(seleniumwire_options=options,executable_path="${geckodriver_path}")

driver.get('http://v2.api.juliangip.com/v2/dps/ip_search')

# 获取页面内容
print(driver.page_source)

# 延迟3秒后关闭当前窗口，如果是最后一个窗口则退出
time.sleep(3)
driver.close()

Python-Scrapy¶

使用提示

http/https网页均可适用
scrapy不是python原生库，需要安装才能使用: pip install scrapy

Scrapy项目目录

运行命令：scrapy startproject tutorial 新建Scrapy项目，创建包含下列内容的tutorial目录

tutorial/ 
    scrapy.cfg # 项目的配置文件 
    tutorial/ # 该项目的python模块。之后您将在此加入代码 
        __init__.py 
        items.py # 项目中的item文件 
        pipelines.py # 项目中的pipelines文件 
        settings.py # 项目的设置文件 
        spiders/ # 放置spider代码的目录 __init__.py 
        ...

myextend.py

添加自定义扩展（Extend）：在tutorial/ 目录下新建myextend.py文件,调用时只需修改api_url以及在time.sleep处设置提取IP的间隔时间即可

#!/usr/bin/env python 
# -- coding: utf-8 -- 
import time
import threading

import requests
from scrapy import signals

# 提取代理IP的api
api_url = 'http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=json&trade_no=1834987042738379&sign=686f43f4b2d89d74c1680577d4ffbe47'
foo = True

class Proxy:

    def __init__(self, ):
        self._proxy_list = requests.get(api_url).json().get('data').get('proxy_list')

    @property
    def proxy_list(self):
        return self._proxy_list

    @proxy_list.setter
    def proxy_list(self, list):
        self._proxy_list = list


pro = Proxy()
print(pro.proxy_list)


class MyExtend:

    def __init__(self, crawler):
        self.crawler = crawler
        # 将自定义方法绑定到scrapy信号上,使程序与spider引擎同步启动与关闭
        # scrapy信号文档: https://www.osgeo.cn/scrapy/topics/signals.html
        # scrapy自定义拓展文档: https://www.osgeo.cn/scrapy/topics/extensions.html
        crawler.signals.connect(self.start, signals.engine_started)
        crawler.signals.connect(self.close, signals.spider_closed)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def start(self):
        t = threading.Thread(target=self.extract_proxy)
        t.start()

    def extract_proxy(self):
        while foo:
            pro.proxy_list = requests.get(api_url).json().get('data').get('proxy_list')
            #设置每15秒提取一次ip
            time.sleep(15)

    def close(self):
        global foo
        foo = False

middlewares.py

middlewares.py中新增ProxyDownloaderMiddleware即代理中间件
请注意替换代码中的部分信息： ${username}：用户名 ${password}：密码

#!/usr/bin/env python 
# -- coding: utf-8 -- 

from scrapy import signals
from .myextend import pro
from w3lib.http import basic_auth_header
import random


class ProxyDownloaderMiddleware:

    def process_request(self, request, spider):
        proxy = random.choice(pro.proxy_list)
        request.meta['proxy'] = "http://%(proxy)s" % {'proxy': proxy}
        # 用户名密码认证(动态代理/独享代理)
        request.headers['Proxy-Authorization'] = basic_auth_header('${username}', '${password}')  # 白名单认证可注释此行
        return None

settings.py

settings.py中激活ProxyDownloaderMiddleware代理中间件和自定义拓展

# -- coding: utf-8 --

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    'tutorial.middlewares.ProxyDownloaderMiddleware': 100,
}
#注意路径
EXTENSIONS = {
    'tutorial.myextend.MyExtend': 300,
}

Java¶

okhttp¶

okhttp3-3.8.1

使用提示

此样例同时支持访问http和https网页
使用用户名密码验证时必须重写 Authenticator 的 authenticate方法
添加依赖

import okhttp3.*;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;

public class TestProxyOKHttpClient {
    public static void main(String args[]) throws IOException {
        // 目标网站
        String targetUrl = "http://v2.api.juliangip.com/v2/dps/ip_search";

        // 代理信息
        String api_url = "http://v2.api.juliangip.com/dynamic/getips?auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=eb93bf75e229386757485eb5d7dc8427";
        String res = HttpUtil.get(api_url);

        // API返回账号密码验证
        String ip = res.split(":")[0];
        int port = Integer.parseInt(res.split(":")[1]);
        String username = res.split(":")[2];
        String password = res.split(":")[3];

        // 账密固定
        // final String username = "username";
        // final String password = "password";

        Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ip, port));

        Authenticator authenticator = new Authenticator() {
            @Override
            public Request authenticate(Route route, Response response) throws IOException {
                String credential = Credentials.basic(username, password);
                return response.request().newBuilder()
                        .header("Proxy-Authorization", credential)
                        .build();
            }
        };
        OkHttpClient client = new OkHttpClient.Builder()
                .proxy(proxy)
                .proxyAuthenticator(authenticator)
                .build();

        Request request = new Request.Builder()
                .url(targetUrl)
                .addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3100.0 Safari/537.36")
                .build();

        Response response = client.newCall(request).execute();
        System.out.println(response.body().string());

    }
}

Java-Selenium¶

Chrome (IP白名单,推荐)

使用提示

基于白名单方式使用Selenium+Chrome认证代理
运行环境要求jdk1.8+ + selenium + Chrome + Chromedriver + Windows/Linux/macOS
下载chromedriver（注意chromedriver版本要和Chrome版本对应)
selenium不是JDK原生库，需要添加相关依赖才能使用

    import org.openqa.selenium.By;
    import org.openqa.selenium.Proxy;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.WebElement;
    import org.openqa.selenium.chrome.ChromeDriver;
    import org.openqa.selenium.chrome.ChromeOptions;

    public class SeleniumTest {
        public static void main(String[] args) throws InterruptedException {
            // 目标网站
            String targetUrl = "https://www.baidu.com";

            // 代理ip: port
            String proxyServer = "103.43.135.4:50568";
            // 创建webdriver驱动，设置代理
            System.setProperty("webdriver.chrome.driver", "C:\\Program Files\\Google\\Chrome\\Application\\chromedriver.exe"); // webdriver驱动路径
            Proxy proxy = new Proxy().setHttpProxy(proxyServer).setSslProxy(proxyServer);
            ChromeOptions options = new ChromeOptions();
            options.setProxy(proxy);
            WebDriver driver = new ChromeDriver(options);

            // 发起请求
            driver.get(targetUrl);
            WebElement element = driver.findElement(By.xpath("/html"));
            String resText = element.getText();
            System.out.println(resText);
            Thread.sleep(10000);

            // 关闭webdriver
            driver.quit();
        }
    }

httpclient¶

HttpClient-4.5.6

使用提示

此样例同时支持访问http和https网页
使用用户名密码访问的情况下，每次请求httpclient会发送两次进行认证从而导致请求耗时增加，建议使用白名单访问
若有多个用户名、密码进行认证需要在代码中须添加AuthCacheValue.setAuthCache(new AuthCacheImpl());
依赖包下载：
httpclient-4.5.6.jar
httpcore-4.4.10.jar
commons-codec-1.10.jar
commons-logging-1.2.jar

import java.net.URL;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

/**
* 使用httpclient请求代理服务器 请求http和https网页均适用
*/
public class TestProxyHttpClient {

    private static String pageUrl = "http://v2.api.juliangip.com/v2/dps/ip_search"; // 要访问的目标网页

    public static void main(String[] args) throws Exception {
        // 代理信息
        String api_url = "http://v2.api.juliangip.com/dynamic/getips?auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=eb93bf75e229386757485eb5d7dc8427";
        String res = HttpUtil.get(api_url);

        // API返回账号密码验证
        String proxyIp = res.split(":")[0];
        int proxyPort = Integer.parseInt(res.split(":")[1]);
        String username = res.split(":")[2];
        String password = res.split(":")[3];

        // 账密固定
        // final String username = "username";
        // final String password = "password";

        // JDK 8u111版本后，目标页面为HTTPS协议，启用proxy用户密码鉴权
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");

        CredentialsProvider credsProvider = new BasicCredentialsProvider();
        credsProvider.setCredentials(new AuthScope(proxyIp, proxyPort),
                new UsernamePasswordCredentials(username, password));
        CloseableHttpClient httpclient = HttpClients.custom().setDefaultCredentialsProvider(credsProvider).build();
        try {
            URL url = new URL(pageUrl);
            HttpHost target = new HttpHost(url.getHost(), url.getDefaultPort(), url.getProtocol());
            HttpHost proxy = new HttpHost(proxyIp, proxyPort);

            /*
            httpclient各个版本设置超时都略有不同, 此处对应版本4.5.6
            setConnectTimeout：设置连接超时时间
            setConnectionRequestTimeout：设置从connect Manager获取Connection 超时时间
            setSocketTimeout：请求获取数据的超时时间
            */
            RequestConfig config = RequestConfig.custom().setProxy(proxy).setConnectTimeout(6000)
                    .setConnectionRequestTimeout(2000).setSocketTimeout(6000).build();
            HttpGet httpget = new HttpGet(url.getPath());
            httpget.setConfig(config);
            httpget.addHeader("Accept-Encoding", "gzip"); // 使用gzip压缩传输数据让访问更快
            httpget.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36");
            CloseableHttpResponse response = httpclient.execute(target, httpget);
            try {
                System.out.println(response.getStatusLine());
                System.out.println(EntityUtils.toString(response.getEntity()));
            } finally {
                response.close();
            }
        } finally {
            httpclient.close();
        }
    }
}

jsoup¶

使用jsoup发起请求

使用提示

此样例同时支持访问http和https网页
若有多个用户名、密码进行认证需要在代码中须添加AuthCacheValue.setAuthCache(new AuthCacheImpl());
jsoup-1.13.1.jar

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class jsoupTest {

    public static void main(String[] args) {
        try {
            String proxyHost = "36.134.55.149";
            int proxyPort = 11125;
            String proxyUser = "";
            String proxyPassword = "";

            HttpHost proxy = new HttpHost(proxyHost, proxyPort);

            BasicCredentialsProvider credsProvider = new BasicCredentialsProvider();
            credsProvider.setCredentials(
                    new AuthScope(proxy),
                    new UsernamePasswordCredentials(proxyUser, proxyPassword)
            );

            try (CloseableHttpClient httpClient = HttpClients.custom()
                    .setProxy(proxy)
                    .setDefaultCredentialsProvider(credsProvider)
                    .build()) {

                HttpGet httpGet = new HttpGet("http://v2.api.juliangip.com/v2/dps/ip_search?ip=0.0.0.0");
                String html = httpClient.execute(httpGet, response ->
                        EntityUtils.toString(response.getEntity())
                );

                Document doc = Jsoup.parse(html);
                System.out.println(doc.body().text());
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

hutool¶

使用hutool发起请求

使用提示

此样例同时支持访问http和https网页
使用用户名密码访问的情况下，每次请求会发送两次进行认证从而导致请求耗时增加，建议使用白名单访问(企业池目前不支持账密模式)
依赖包下载：
hutool-all-5.5.4.jar

import java.net.Authenticator;
import java.net.PasswordAuthentication;
import cn.hutool.http.HttpResponse;
import cn.hutool.http.HttpRequest;

public class TestProxyHutool {

    public static void main(String[] args) {
        // 代理信息
        String api_url = "http://v2.api.juliangip.com/dynamic/getips?auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=eb93bf75e229386757485eb5d7dc8427";
        String res = HttpUtil.get(api_url);

        // API返回账号密码验证 （企业池不支持账密模式，只能使用白名单模式）
        String ProxyHost = res.split(":")[0];
        int ProxyPort = Integer.parseInt(res.split(":")[1]);
        String ProxyUser = res.split(":")[2];
        String ProxyPass = res.split(":")[3];

        // 账密固定
        // final String ProxyUser = "username";
        // final String ProxyPass = "password";

        // 目标网站
        String url = "http://v2.api.juliangip.com/v2/dps/ip_search";
        // JDK 8u111版本后，目标页面为HTTPS协议，启用proxy用户密码鉴权
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
        // 设置请求验证信息
        Authenticator.setDefault(new Authenticator() {
            @Override
            protected PasswordAuthentication getPasswordAuthentication() {
                return new PasswordAuthentication(ProxyUser, ProxyPass.toCharArray());
            }
        });

        // 发送请求
        HttpResponse result = HttpRequest.get(url)
                .setHttpProxy(ProxyHost, ProxyPort)
                .timeout(20000)//设置超时，毫秒
                .execute();
        System.out.println(result.body());
    }
}

GoLang¶

标准库¶

标准库

使用提示

·http和https网页均可适用

// 请求代理服务器
// http和https网页均适用

package main

import (
    "compress/gzip"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "net/url"
    "os"
)

func main() {

    // 代理信息
    apiUrl := "http://v2.api.juliangip.com/dynamic/getips?auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=eb93bf75e229386757485eb5d7dc8427"
    client := &http.Client{}
    s, _ := http.NewRequest("GET", apiUrl, nil)
    resp, _ := client.Do(s)

    var proxyIp string
    if resp.StatusCode == 200 {
        body, _ := io.ReadAll(resp.Body)
        proxyIp = string(body)
    }

    // API返回账号密码验证
    proxyList := strings.Split(proxyIp, ":")
    proxyRaw := proxyList[0] + ":" + proxyList[1]
    username := proxyList[2]
    password := proxyList[3]

    // 账密固定
    // username := "username"
    // password := "password"

    proxyStr := fmt.Sprintf("http://%s:%s@%s", username, password, proxyRaw)
    proxy, err := url.Parse(proxyStr)

    // 目标网站
    pageUrl := "http://v2.api.juliangip.com/v2/dps/ip_search"

    reqClient := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxy)}}
    req, _ := http.NewRequest("GET", pageUrl, nil)
    req.Header.Add("Accept-Encoding", "gzip") //使用gzip压缩传输数据让访问更快
    res, err := reqClient.Do(req)

    if err != nil {
        // 请求发生异常
        fmt.Println(err.Error())
    } else {
        defer res.Body.Close() //保证最后关闭Body

        fmt.Println("status code:", res.StatusCode) // 获取状态码

        // 有gzip压缩时,需要解压缩读取返回内容
        if res.Header.Get("Content-Encoding") == "gzip" {
            reader, _ := gzip.NewReader(res.Body) // gzip解压缩
            defer reader.Close()
            io.Copy(os.Stdout, reader)
            os.Exit(0) // 正常退出
        }

        // 无gzip压缩, 读取返回内容
        body, _ := ioutil.ReadAll(res.Body)
        fmt.Println(string(body))
    }
}

goKit¶

goKit

使用提示

·http和https网页均可适用

·go get github.com/xingcxb/goKit

package main

import(
    "fmt"
    "github.com/xingcxb/goKit/core/httpKit"
)

func main() {
    /// get请求
    // 白名单认证
    fmt.Println(httpKit.HttpProxyGet("https://cip.cc", "22.33.44.55:59582"))
    fmt.Println("------------------>")
    // 用户名密码认证
    fmt.Println(httpKit.HttpProxyGetFull("https://cip.cc", nil, nil,
        "", 300, "http", "username", "password",
        "22.33.44.55:59582"))
    fmt.Println("------------------>")
    /// post请求
    // 白名单认证
    fmt.Println(httpKit.HttpProxyPost("https://cip.cc", nil, "22.33.44.55:59582"))
    fmt.Println("------------------>")
    // 用户名密码认证
    fmt.Println(httpKit.HttpProxyPostFull("https://cip.cc", nil, nil,
        "", 300, "http", "username", "password", "22.33.44.55:59582"))
}

CSharp¶

标准库¶

标准库

使用提示

·http和https网页均可适用

using System;
using System.Text;
using System.Net;
using System.IO;
using System.IO.Compression;

namespace csharp_http
{
    class Program
    {
        static void Main(string[] args) {
            // 代理信息
            string apiUrl = "http://v2.api.juliangip.com/dynamic/getips?auto_white=1&num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=eb93bf75e229386757485eb5d7dc8427";
            HttpWebRequest apiRequest = (HttpWebRequest)WebRequest.Create(apiUrl);
            apiRequest.Method = "GET";
            apiRequest.ContentType = "application/x-www-form-urlencoded";
            apiRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64)";
            apiRequest.Timeout = 10000;

            using (HttpWebResponse apiResponse = (HttpWebResponse)apiRequest.GetResponse()) {
                using (Stream apiResponseStream = apiResponse.GetResponseStream()) {
                    using (StreamReader apiReader = new StreamReader(apiResponseStream)) {
                        string apiResponseContent = apiReader.ReadToEnd();
                        Console.WriteLine(apiResponseContent);

                        // API返回账号密码验证
                        string[] parts = apiResponseContent.Split(new[] { ':' });
                        string proxy_ip = parts[0];
                        int proxy_port = int.Parse(parts[1]);
                        string username = parts[2];
                        string password = parts[3];

                        // 账密固定
                        // string username = "username";
                        // string password = "password";

                        // 要访问的目标网页
                        string targetUrl = "http://v2.api.juliangip.com/v2/dps/ip_search";

                        // 构造请求
                        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(targetUrl);
                        request.Method = "GET"; // 设置请求方法为GET
                        request.ContentType = "application/x-www-form-urlencoded"; // 设置内容类型，根据需要调整
                        request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"; // 设置User-Agent，模仿浏览器访问
                        request.Timeout = 10000; // 设置超时时间为10秒

                        // 设置代理 (已添加白名单)
                        // request.Proxy = new WebProxy(proxy_ip, proxy_port);

                        // 设置代理 (未添加白名单)
                        WebProxy proxy = new WebProxy();
                        proxy.Address = new Uri(String.Format("http://{0}:{1}", proxy_ip, proxy_port));
                        proxy.Credentials = new NetworkCredential(username, password);
                        request.Proxy = proxy;

                        try {
                            using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) {
                                Console.WriteLine("目标站返回码：" + (int)response.StatusCode);
                                if (response.StatusCode == HttpStatusCode.OK) {
                                    using (Stream responseStream = response.GetResponseStream()) {
                                        using (StreamReader reader = new StreamReader(responseStream)) {
                                            string responseContent = reader.ReadToEnd();
                                            Console.WriteLine(responseContent); // 输出网页内容到控制台
                                        }
                                    }
                                } else {
                                    Console.WriteLine("Error: " + response.StatusCode); // 输出错误状态码，例如404, 500等。
                                }
                            }
                        }
                        catch (WebException e) {
                            Console.WriteLine("Error: " + e.Status + " - " + e.Message); // 输出异常信息。
                        }
                    }
                }
            }
        }
    }
}

Node.js¶

标准库(http+url)¶

标准库(http,https均适用)

使用提示

·http和https网页均可适用

const http = require("http");
const zlib = require('zlib');

// 提取代理API接口，获取1个代理IP
const apiUrl = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006";
const apiObj = new URL(apiUrl);
http.request({
    hostname: apiObj.hostname,
    port: 80,
    path: apiObj.pathname + apiObj.search,
    method: 'GET'
}, (res) => {
    let proxy = '';

    res.on('data', (c) => {
        proxy += c;
    });

    res.on('end', () => {
        // API返回账号密码验证
        let proxy_list = proxy.split(':');
        let proxyIp = proxy_list[0];
        let proxyPort = proxy_list[1];
        let username = proxy_list[2];
        let password = proxy_list[3];

        // 账密固定
        // const username = "username";
        // const password = "password";

        // 要访问的目标网页
        const targetUrl = "http://v2.api.juliangip.com/v2/dps/ip_search";
        const base64 = Buffer.from(username + ":" + password).toString("base64");
        const urlObj = new URL(targetUrl);
        const options = {
            host: proxyIp,
            port: proxyPort,
            path: urlObj.pathname + urlObj.search,
            method: "GET",
            headers: {
                "Host": urlObj.hostname,
                "Proxy-Authorization": "Basic " + base64
            }
        };

        http.request(options, (res) => {
            console.log("got response: " + res.statusCode);

            let data = '';

            if (res.headers['content-encoding'] && res.headers['content-encoding'].indexOf('gzip') !== -1) {
                const unzip = zlib.createGunzip();
                res.pipe(unzip);

                unzip.on('data', (chunk) => {
                    data += chunk;
                });

                unzip.on('end', () => {
                    console.log("Response Data:");
                    console.log(data);
                });

                unzip.on('error', (err) => {
                    console.error("Error decompressing data: " + err.message);
                });
            } else {
                res.on('data', (chunk) => {
                    data += chunk;
                });

                res.on('end', () => {
                    console.log("Response Data:");
                    console.log(data);
                });
            }

        }).on("error", (err) => {
            console.error("Error: " + err.message);
        }).end();

    });
}).on("error", (err) => {
    console.error("Error: " + err.message);
}).end();

标准库(http+tls+util)¶

标准库(http,https均适用)

使用提示

·http和https网页均可适用

let http = require('http'); // 引入内置http模块
let tls = require('tls'); // 引入内置tls模块
let util = require('util');

// 用户名密码认证(动态代理/独享代理)
const username = 'username';
const password = 'password';
const auth = 'Basic ' + new Buffer.from(username + ':' + password).toString('base64');

// 代理服务器ip和端口
let proxy_ip = '117.69.63.102';
let proxy_port = 43787;

// 要访问的主机和路径
let remote_host = 'http://v2.api.juliangip.com/v2/dps/ip_search';
let remote_path = '/';

// 发起CONNECT请求
let req = http.request({
    host: proxy_ip,
    port: proxy_port,
    method: 'CONNECT',
    path: util.format('%s:443', remote_host),
    headers: {
        "Host": remote_host,
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3100.0 Safari/537.36",
        "Proxy-Authorization": auth,
        "Accept-Encoding": "gzip"   // 使用gzip压缩让数据传输更快
    }
});


req.on('connect', function (res, socket, head) {
    // TLS握手
    let tlsConnection = tls.connect({
        host: remote_host,
        socket: socket
    }, function () {
        // 发起GET请求
        tlsConnection.write(util.format('GET %s HTTP/1.1\r\nHost: %s\r\n\r\n', remote_path, remote_host));
    });

    tlsConnection.on('data', function (data) {
        // 输出响应结果(完整的响应报文串)
        console.log(data.toString());
    });
});

req.end();

request¶

标准库(http,https均适用)

使用提示

·请先安装request库: npm install request ·http网页和https网页均可适用

let request = require('request'); // 引入第三方request库
let util = require('util');
let zlib = require('zlib');

// 用户名密码认证(动态代理/独享代理)
const username = 'username';
const password = 'password';

// 要访问的目标地址
let page_url = 'http://v2.api.juliangip.com/v2/dps/ip_search'

// 代理服务器ip和端口
let proxy_ip = '117.69.63.102';
let proxy_port = 43787;

// 完整代理服务器url
let proxy = util.format('http://%s:%s@%s:%d', username, password, proxy_ip, proxy_port);  

// 发起请求
request({
    url: page_url,
    method: 'GET',
    proxy: proxy,
    headers: {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3100.0 Safari/537.36",
        "Accept-Encoding": "gzip"   // 使用gzip压缩让数据传输更快
    },
    encoding: null,  // 方便解压缩返回的数据
}, function(error, res, body) {
    if (!error && res.statusCode == 200) {
        // 输出返回内容(使用了gzip压缩)
        if (res.headers['content-encoding'] && res.headers['content-encoding'].indexOf('gzip') != -1) {
            zlib.gunzip(body, function(err, dezipped) {
                console.log(dezipped.toString()); 
            });
        } else {
            // 输出返回内容(没有使用gzip压缩)
            console.log(body);
        }
    } else {
        console.log(error);
    }
});

puppeteer¶

puppeteer (IP白名单)

使用提示

·基于用户名密码认证的http/https代理Puppeteer ·运行环境要求: node7.6.0或以上 + puppeteer ·请先安装puppeteer: npm i puppeteer

// 引入puppeteer模块
const puppeteer = require('puppeteer');

// 要访问的目标网页
const url = 'http://v2.api.juliangip.com/v2/dps/ip_search';

// 添加headers
const headers = {
    'Accept-Encoding': 'gzip' // 使用gzip压缩让数据传输更快
};

// 代理服务器ip和端口
let proxy_ip = '117.69.63.102'
let proxy_port = 43787

(async ()=> {
    // 新建一个浏览器实例
    const browser = await puppeteer.launch({
        headless: false,  // 是否不显示窗口, 默认为true, 设为false便于调试
        args: [
            `--proxy-server=${proxy_ip}:${proxy_port}`,
            '--no-sandbox',
            '--disable-setuid-sandbox'
        ]
    });

    // 打开一个新页面
    const page = await browser.newPage();

    // 设置headers
    await page.setExtraHTTPHeaders(headers);

    // 访问目标网页
    await page.goto(url);

})();

puppeteer(用户名密码认证)

使用提示

·基于白名单的http/https代理Puppeteer ·运行环境要求: node7.6.0或以上 + puppeteer ·请先安装puppeteer: npm i puppeteer

// 引入puppeteer模块
const puppeteer = require('puppeteer');

// 要访问的目标网页
const url = 'http://v2.api.juliangip.com/v2/dps/ip_search';

// 添加headers
const headers = {
    'Accept-Encoding': 'gzip' // 使用gzip压缩让数据传输更快
};

// 代理服务器ip和端口
let proxy_ip = '117.69.63.102'
let proxy_port = 43787

// 用户名密码认证(动态代理/独享代理)
const username = 'username';
const password = 'password';

(async ()=> {
    // 新建一个浏览器实例
    const browser = await puppeteer.launch({
        headless: false,  // 是否不显示窗口, 默认为true, 设为false便于调试
        args: [
            `--proxy-server=${proxy_ip}:${proxy_port}`,
            '--no-sandbox',
            '--disable-setuid-sandbox'
        ]
    });

    // 打开一个新页面
    const page = await browser.newPage();

    // 设置headers
    await page.setExtraHTTPHeaders(headers);

    // 用户民密码认证
    await page.authenticate({username: username, password: password});

    // 访问目标网页
    await page.goto(url);
})();

axios¶

http,https均适用

使用提示

终端执行 npm install axios-https-proxy-fix 下载相应的包

// 引入包：axios-https-proxy-fix
let axios = require('axios-https-proxy-fix')
// 目标网站
let targetUrl = 'https://cip.cc' //要访问的目标站点
//服务器信息
let serverURL = '***********' //代理服务器地址
let serverPort = '***********' //代理服务器端口

// 设置代理
let proxy = {
    host: serverURL,
    port: serverPort,
// 使用白名单可不添加 auth 字段
//     auth: {
//         username: authKey,
//         password: authPwd
//     }
}
axios.get(targetUrl, {proxy: proxy}) //请求时设置调用代理信息
    .then((res) => {
        console.log(res.data)
    }).catch((err) => {
    console.log(err.message)
})

Ruby¶

net/http¶

net/http(IP白名单)

使用提示

基于用户名密码认证的http/https代理net/http

# -*- coding: utf-8 -*-

require 'net/http'  # 引入内置net/http模块
require 'zlib'
require 'stringio'

# 代理服务器ip 和 端口
proxy_ip = '117.69.63.102'
proxy_port = 43787

    # 目标站，此处为示例
page_url = "http://v2.api.juliangip.com/v2/dps/ip_search"
uri = URI(page_url)

# 新建代理实例
proxy = Net::HTTP::Proxy(proxy_ip, proxy_port)

# 创建新的请求对象 
req = Net::HTTP::Get.new(uri)
# 设置User-Agent
req['User-Agent'] = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50'
req['Accept-Encoding'] = 'gzip'  # 使用gzip压缩传输数据让访问更快



# 使用代理发起请求, 若访问的是http网页, 请将use_ssl设为false
res = proxy.start(uri.hostname, uri.port, :use_ssl => true) do |http|
    http.request(req)
end

# 输出状态码
puts "status code: #{res.code}"

# 输出响应体
if  res.code.to_i != 200 then
    puts "page content: #{res.body}"
else
    gz = Zlib::GzipReader.new(StringIO.new(res.body.to_s))
    puts "page content: #{gz.read}" 
end

net/http(用户名密码认证)

使用提示

基于用户名密码认证的http/https代理net/http

# -*- coding: utf-8 -*-

require 'net/http'  # 引入内置net/http模块
require 'zlib'
require 'stringio'

# 代理服务器ip 和 端口
proxy_ip = '117.69.63.102'
proxy_port = 43787

# 用户名密码认证(动态代理/独享代理)
username = 'username'
password = 'password'

# 要访问的目标网页, 此处为示例
page_url = "http://v2.api.juliangip.com/v2/dps/ip_search"
uri = URI(page_url)

# 新建代理实例
proxy = Net::HTTP::Proxy(proxy_ip, proxy_port, username, password)

# 创建新的请求对象 
req = Net::HTTP::Get.new(uri)
# 设置代理用户名密码认证(动态代理/独享代理)
req.basic_auth(username, password)
# 设置User-Agent
req['User-Agent'] = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50'
req['Accept-Encoding'] = 'gzip'  # 使用gzip压缩传输数据让访问更快



# 使用代理发起请求, 若访问的是http网页, 请将use_ssl设为false
res = proxy.start(uri.hostname, uri.port, :use_ssl => true) do |http|
    http.request(req)
end

# 输出状态码
puts "status code: #{res.code}"

# 输出响应体
if  res.code.to_i != 200 then
    puts "page content: #{res.body}"
else
    gz = Zlib::GzipReader.new(StringIO.new(res.body.to_s))
    puts "page content: #{gz.read}" 
end

php¶

curl¶

curl

使用提示

1.此样例同时支持访问http和https网页 2.curl不是php原生库，需要安装才能使用:
Ubuntu/Debian系统：apt-get install php5-curl
CentOS系统：yum install php-curl

<?php 
// 提取代理API接口，获取1个代理IP
$api_url  = "http://v2.api.juliangip.com/dynamic/getips?num=1&pt=1&result_type=text&split=1&trade_no=1834987042xxxxxx&sign=9e489baa3bf149593f149d7252efd006";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $api_url);
curl_setopt($curl, CURLOPT_HTTPGET, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$re = curl_exec($curl);
curl_close($curl);
if ($re !== '未检索到满足要求的代理IP，请调整筛选条件后再试，或联系客服处理！') {
    // API返回账号密码验证
    $proxy_list = explode(':', $re);
    $proxy = "http://$proxy_list[0]:$proxy_list[1]";
    $auth = "$proxy_list[2]:$proxy_list[3]";

    // 账密固定
    // $auth = "username:password";

    // 要访问的目标网页
    $page_url = "http://v2.api.juliangip.com/v2/dps/ip_search";

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $page_url);

    // 自定义header
    $headers               = [];
    $headers["user-agent"] = 'User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0);';
    $headers[]             = 'Content-Type: application/json';
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

    // 发送post请求
    // curl_setopt($ch, CURLOPT_POST, 1);
    // $requestData["id"] = 1;
    // curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($requestData));

    // get请求
    curl_setopt($ch, CURLOPT_HTTPGET, true);

    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
    curl_setopt($ch, CURLOPT_PROXY, $proxy);
    curl_setopt($ch, CURLOPT_PROXYAUTH, CURLAUTH_BASIC);
    curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth);

    curl_setopt($ch, CURLOPT_COOKIE, ''); //自定义cookie
    curl_setopt($ch, CURLOPT_ENCODING, 'gzip'); //使用gzip压缩传输数据让访问更快
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5); //设置连接超时时间
    curl_setopt($ch, CURLOPT_TIMEOUT, 10); //设置超时时间
    curl_setopt($ch, CURLOPT_HEADER, true); //返回头信息
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); //返回数据不直接输出

    $result = curl_exec($ch);
    curl_close($ch);
    halt($result);
}

?>

没有您在使用的编程语言示例？

因开发语言太多，如果没有找到您的编程语言示例，请您联系客服处理！