本文共 3196 字，大约阅读时间需要 10 分钟。

Python爬虫：爬取某鱼颜值主播图片并保存到本地2.0版——解決爬取多页报错問題及簡化图片重命名

一、准备

1.创建Scrapy项目

使用Scrapy框架创建爬虫项目，步骤如下：

scrapy startproject douyucd douyuscrapy genspider spider "www.douyu.com"

2.创建启动文件start.py

from scrapy import cmdlinecmdline.execute("scrapy crawl douyu".split())

二、分析

1.进入斗鱼平台

通过浏览斗鱼官网，找到需要爬取的数据路径。主要关注的页面URL为：

https://www.douyu.com/gapi/rknc/directory/yzRec/1

2.解析数据结构

通过爬取页面数据，发现数据以JSON格式返回，具体结构如下：

{  "data": {    "rl": [      {        "nn": "主播名称",        "rs1": "图片URL"      },      ...    ]  }}

需要提取的信息：主播名称nn和图片URLrs1。

三、代码实现

1.Spider文件spider.py

import scrapyfrom douyu.items import DouyuItemclass DouyuSpider(scrapy.Spider):    name = 'douyu'    start_urls = ['https://www.douyu.com/gapi/rknc/directory/yzRec/1']    offset = 1  # 爬取多页的限制    def parse(self, response):        data_list = json.loads(response.body)["data"]["rl"]        for data in data_list:            nn = data["nn"]            img_url = data["rs1"]            item = DouyuItem(nn=nn, img_url=img_url)            yield item        self.offset += 1        if self.offset < 4:            num = int(str(response).split(" ")[1].replace(">", "").split("/")[-1])            num += 1            url = f"https://www.douyu.com/gapi/rknc/directory/yzRec/{num}"            print(url)            yield scrapy.Request(url=url, callback=self.parse, encoding="utf-8", dont_filter=True)

2.Pipeline文件pipelines.py

import scrapyfrom scrapy.pipelines.images import ImagesPipelineclass DouyuPipeline(ImagesPipeline):    def get_media_requests(self, item, info):        image_link = item["img_url"]        image_name = item['nn']        yield scrapy.Request(image_link, meta={"image_name": image_name})    def file_path(self, request, response=None, info=None):        category = request.meta['image_name']        return f"{category}.jpg"

3.Items文件items.py

import scrapyclass DouyuItem(scrapy.Item):    nn = scrapy.Field()  # 主播名称    img_url = scrapy.Field()  # 直播间封面图片URL

四、完整代码

spider.py（主要爬虫逻辑）

import scrapyimport jsonfrom douyu.items import DouyuItemclass DouyuSpider(scrapy.Spider):    name = 'douyu'    start_urls = ['https://www.douyu.com/gapi/rknc/directory/yzRec/1']    offset = 1    def parse(self, response):        data_list = json.loads(response.body)["data"]["rl"]        for data in data_list:            yield DouyuItem(nn=data["nn"], img_url=data["rs1"])        self.offset += 1        if self.offset < 4:            num = int(str(response).split(" ")[1].replace(">", "").split("/")[-1]) + 1            url = f"https://www.douyu.com/gapi/rknc/directory/yzRec/{num}"            print(url)            yield scrapy.Request(url=url, callback=self.parse, encoding="utf-8", dont_filter=True)

pipelines.py（图片下载与重命名）

import scrapyfrom scrapy.pipelines.images import ImagesPipelineclass DouyuPipeline(ImagesPipeline):    def get_media_requests(self, item, info):        image_link = item["img_url"]        image_name = item['nn']        yield scrapy.Request(image_link, meta={"image_name": image_name})    def file_path(self, request, response=None, info=None):        return f"{request.meta['image_name']}.jpg"

五、总结

通过以上代码实现，可以轻松爬取斗鱼主播的图片，并将其保存到本地。代码逻辑清晰，支持多页爬取，并通过Pipeline优化了图片下载和重命名流程。如果需要更高效的爬取效率，可通过设置CONCURRENT_REQUESTS和DOWNLOAD_DELAY来优化。