当前位置：首页 > news >正文

万物可爬（以爬取浏览器井盖图片和豆瓣电影名字为例）

news 2024/12/14 0:26:26

我们以爬取井盖图片这个链接中的图片为例：

点击F12 并选中其中一张图片，得到它的信息。具体如下：我们可以编写对应的正则表达式：

'<img[^>]*src="(.*?)"[^>]*alt="井盖图片 的图像结果"[^>]*>'

1 <img 匹配开头

2 [^>]* 匹配非>的任意多个字符（为了不匹配到结尾）

3 src="(.*?)" 进行非贪婪匹配

4 alt="井盖图片的图像结果" 进行元素限制

5 > 结尾

完整爬虫代码：

# -*- coding: utf-8 -*-
import re
import requests
from urllib import error
from bs4 import BeautifulSoup
import osfile = ''
List = []#爬取图片链接
def Find(url, A):global Listprint('正在检测图片总数，请稍等.....')s = 0try:Result = A.get(url, timeout=7, allow_redirects=False)except BaseException:print("error");else:result = Result.textpic_url = re.findall('<img[^>]*src="(.*?)"[^>]*alt="井盖图片 的图像结果"[^>]*>', result)  # 先利用正则表达式找到图片urls += len(pic_url)if len(pic_url) == 0:print("没读到")else:List.append(pic_url)return s#下载图片
def dowmloadPicture():num = 1for each in List[0]:print('正在下载第' + str(num) + '张图片，图片地址:' + str(each))try:if each is not None:pic = requests.get(each, timeout=7)else:continueexcept BaseException:print('错误，当前图片无法下载')continueelse:if len(pic.content) < 200:continuestring = file + r'\\'  + str(num) + '.jpg'fp = open(string, 'wb')fp.write(pic.content)fp.close()num+=1if __name__ == '__main__':  # 主函数入口headers = {'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2','Connection': 'keep-alive','User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0','Upgrade-Insecure-Requests': '1'}A = requests.Session()A.headers = headersurl = 'https://cn.bing.com/images/search?q=%E4%BA%95%E7%9B%96%E5%9B%BE%E7%89%87&form=IQFRML&first=1'total = Find(url, A)print('经过检测图片共有%d张' % (total))file = input('请建立一个存储图片的文件夹，输入文件夹名称即可: ')y = os.path.exists(file)if y == 1:print('该文件已存在，请重新输入')file = input('请建立一个存储图片的文件夹，)输入文件夹名称即可: ')os.mkdir(file)else:os.mkdir(file)dowmloadPicture()print('当前爬取结束，感谢使用')

浏览器：edge

缺陷：不能匹配划页元素和翻页元素，待完善。

爬取豆瓣电影名字

# -*- coding: utf-8 -*-
import re
import requestsimport osfile = ''
List = []#爬取图片链接
def Find(url, A):global Listprint('正在检测图片总数，请稍等.....')s = 0try:Result = A.get(url, timeout=7, allow_redirects=False)except BaseException:print("error");else:result = Result.texttexts = re.findall('<li class="title">\s*<a[^>]*>(.*?)</a>\s*</li>', result)  # 先利用正则表达式找到图片urls += len(texts)if len(texts) == 0:print("没读到")else:List.append(texts)return s
#下载图片
def dowmloadPicture():num = 1for each in List[0]:print('正在下载第' + str(num) + '张图片，图片地址:' + str(each))try:if each is not None:pic = requests.get(each, timeout=7)else:continueexcept BaseException:print('错误，当前图片无法下载')continueelse:if len(pic.content) < 200:continuestring = file + r'\\'  + str(num) + '.jpg'fp = open(string, 'wb')fp.write(pic.content)fp.close()num+=1if __name__ == '__main__':  # 主函数入口headers = {'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2','Connection': 'keep-alive','User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0','Upgrade-Insecure-Requests': '1'}A = requests.Session()A.headers = headersurl = 'https://movie.douban.com/'#total = Find(url, A)for texts in List:for text in texts:print(text)

注意\s*是用来匹配任意个数的空白字符的：

查看全文

http://www.mrgr.cn/news/79378.html