这个和50章节的下拉菜单有什么区别呢，都是点击之后再点击，为什么我用50章的方法就不行

2024-05-20 20:19:12 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: scrapy05)

2024-05-20 20:19:12 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.11.7, cssselect 1.1.0, parsel 1.6.0, w3lib 2.1.2, Twisted 22.4.0, Python 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1n  15 Mar 2022), cryptography 36.0.2, Platform Windows-10-10.0.22621-SP0

2024-05-20 20:19:12 [scrapy.crawler] INFO: Overridden settings:

{'BOT_NAME': 'scrapy05',

 'FEED_EXPORT_ENCODING': 'utf-8',

 'NEWSPIDER_MODULE': 'scrapy05.spiders',

 'SPIDER_MODULES': ['scrapy05.spiders'],

 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',

 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '

               '(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36'}

2024-05-20 20:19:12 [asyncio] DEBUG: Using selector: SelectSelector

2024-05-20 20:19:12 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor

2024-05-20 20:19:12 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop

2024-05-20 20:19:12 [scrapy.extensions.telnet] INFO: Telnet Password: 00e3679f6ed09808      

2024-05-20 20:19:12 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

 'scrapy.extensions.telnet.TelnetConsole',

 'scrapy.extensions.logstats.LogStats']

2024-05-20 20:19:12 [scrapy.middleware] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',

 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',

 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',

 'scrapy.downloadermiddlewares.retry.RetryMiddleware',

 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',

 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',

 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',

 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',

 'scrapy.downloadermiddlewares.stats.DownloaderStats']

2024-05-20 20:19:12 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',

 'scrapy.spidermiddlewares.referer.RefererMiddleware',

 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',

 'scrapy.spidermiddlewares.depth.DepthMiddleware']

2024-05-20 20:19:12 [scrapy.middleware] INFO: Enabled item pipelines:

['scrapy.pipelines.images.ImagesPipeline']

2024-05-20 20:19:12 [scrapy.core.engine] INFO: Spider opened

2024-05-20 20:19:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2024-05-20 20:19:12 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023

2024-05-20 20:19:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://desk.zol.com.cn/bizhi/9812_118227_2.html> (referer: None)

--- Logging error ---

Traceback (most recent call last):

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 1110, in emit      

    msg = self.format(record)

          ^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 953, in format     

    return fmt.format(record)

           ^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 695, in format     

    record.exc_text = self.formatException(record.exc_info)

                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 645, in formatException

    traceback.print_exception(ei[0], ei[1], tb, None, sio)

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 124, in print_exception   

    te = TracebackException(type(value), value, tb, limit=limit, compact=True)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 690, in __init__

    self.stack = StackSummary._extract_from_extended_frame_gen(

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 416, in _extract_from_extended_frame_gen

    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 353, in _walk_tb_with_full_positions

    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)

                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 366, in _get_code_position

    positions_gen = code.co_positions()

                    ^^^^^^^^^^^^^^^^^

AttributeError: '_Code' object has no attribute 'co_positions'

Call stack:

  File "d:\code\python\spider_code\scrapy05\scrapy05\begin.py", line 3, in <module>

    execute(['scrapy','crawl','zol'])

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\cmdline.py", line 145, in execute

    _run_print_help(parser, _run_command, cmd, args, opts)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\cmdline.py", line 100, in _run_print_help

    func(*a, **kw)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\cmdline.py", line 153, in _run_command

    cmd.run(args, opts)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\commands\crawl.py", line 27, in run

    self.crawler_process.start()

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\crawler.py", line 346, in start  

    reactor.run(installSignalHandlers=False)  # blocking call

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\asyncioreactor.py", line 255, in run

    self._asyncioEventloop.run_forever()

  File "E:\Program Files\Python\Python311\Lib\asyncio\base_events.py", line 607, in run_forever

    self._run_once()

  File "E:\Program Files\Python\Python311\Lib\asyncio\base_events.py", line 1922, in _run_once

    handle._run()

  File "E:\Program Files\Python\Python311\Lib\asyncio\events.py", line 80, in _run

    self._context.run(self._callback, *self._args)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\asyncioreactor.py", line 271, in _onTimer

    self.runUntilCurrent()

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\base.py", line 991, in runUntilCurrent

    call.func(*call.args, **call.kw)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\task.py", line 680, in _tick

    taskObj._oneWorkUnit()

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\task.py", line 526, in _oneWorkUnit

    result = next(self._iterator)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\utils\defer.py", line 86, in <genexpr>

    work = (callable(elem, *args, **named) for elem in iterable)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\core\scraper.py", line 207, in _process_spidermw_output

    dfd.addBoth(self._itemproc_finished, output, response, spider)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 538, in addBoth

    return self.addCallbacks(

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 477, in addCallbacks

    self._runCallbacks()

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 857, in _runCallbacks

    current.result = callback(  # type: ignore[misc]

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\core\scraper.py", line 267, in _itemproc_finished

    logger.log(*logformatter_adapter(logkws), extra={'spider': spider},

Message: 'Error processing %(item)s'

Arguments: {'item': {'image_urls': [['https://desk-fd.zol-img.com.cn/t_s960x600c5/g6/M00/0F/09/ChMkKWF4tn6IeIGwAD5r3l1JSDcAAU-5QFAFu4APmv2924.jpg']]}}

2024-05-20 20:19:13 [scrapy.core.engine] INFO: Closing spider (finished)

2024-05-20 20:19:13 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 316,

 'downloader/request_count': 1,

 'downloader/request_method_count/GET': 1,

 'downloader/response_bytes': 9708,

 'downloader/response_count': 1,

 'downloader/response_status_count/200': 1,

 'elapsed_time_seconds': 0.339944,

 'finish_reason': 'finished',

 'finish_time': datetime.datetime(2024, 5, 20, 12, 19, 13, 237143),

 'httpcompression/response_bytes': 33746,

 'httpcompression/response_count': 1,

 'log_count/DEBUG': 4,

 'log_count/ERROR': 1,

 'log_count/INFO': 10,

 'response_received_count': 1,

 'scheduler/dequeued': 1,

 'scheduler/dequeued/memory': 1,

 'scheduler/enqueued': 1,

 'scheduler/enqueued/memory': 1,

 'start_time': datetime.datetime(2024, 5, 20, 12, 19, 12, 897199)}

2024-05-20 20:19:13 [scrapy.core.engine] INFO: Spider closed (finished)

老师麻烦看下哪里错了

Message: 'Error processing %(item)s'

老师，scrapy可以像requests.session一样会话保持吗？

SDKmanager卡住了

import requests
from fake_useragent import UserAgent
from lxml import etree
import os

def get_header():
    location = os.getcwd() + '/fake_useragent.json'
    ua = UserAgent(path=location)
    return ua.chrome

#发送请求
class Downloader():
    def do_download(self,url):
        print(url)
        hearders = {'User-Agent':get_header()}
        resp = requests.get(url,hearders=hearders)
        if resp.status_code == 200:
            resp.encoding = 'utf-8'
            return resp.text

#数据解析
class Parser():
    def do_parse(self,html):
        e = etree.HTML(html)
        contents = [div.xpath('string(.)').strip() for div in e.xpath('//div[@class="content"]')]
        urls = ['https://www.qiushibaike.com{}'.format(url) for url in e.xpath('//ul[@class="pagination"]/li/a/@href')]
        return contents,urls

#数据保存
class Dataoutput():
    def do_save(self,datas):
        with open('duanzi.txt1','a',encoding='utf-8') as f:
            for data in datas:
                f.write(data + '\n')



#url管理器
class URLManager():
    def __init__(self):
        self.new_url = set()
        self.old_url = set()
        #加入一个url
    def add_new_url(self,url):
        if url is not None and url != '' and url not in self.old_url:
            self.new_url.add(url)
        #加入多个url
    def add_new_urls(self,urls):
        for url in urls:
            self.add_new_url(url)
        #获取一个url
    def get_new_url(self):
        url = self.new_url.pop()
        self.old_url.add(url)
        return url
        #获取还有多少个url要爬取
    def get_new_url_size(self):
        return len(self.new_url)

        #获取是否还有url要爬取
    def have_new_url(self):
        return self.get_new_url_size()>0

#调度器
class Scheduler:
    def __init__(self):
        self.downloader = Downloader()
        self.parser = Parser()
        self.data_out_put = Dataoutput()
        self.url_manager = URLManager()
    def start(self,url):
        self.url_manager.add_new_url(url)
        while self.url_manager.have_new_url():
            url = self.url_manager.get_new_url()
            html = self.downloader.do_download(url)
            datas,urls = self.parser.do_parse(html)
            self.data_out_put.do_save(datas)
            self.url_manager.add_new_urls(urls)



if __name__ == '__main__':
    scheduler = Scheduler()
    url = 'https://www.qiushibaike.com/text/'
    scheduler.start(url)

老师，这个报错是哪儿有问题呢，没想明白

自己敲了一遍代码，运行无结果也不报错

原代码：

import requests

from fake_useragent import UserAgent

from bs4 import BeautifulSoup

url = "https://www.maoyan.com/films?showType=3"

headers = {"User-Agent":UserAgent().edge}

resp = requests.get(url,headers=headers)

bs = BeautifulSoup(resp.text,"lxml")

names = bs.select("dl.movie-list > div.channel-detail.movie-item-title > a")

scores = bs.select("dl.movie-list > div.channel-detail.channel-detail-orange > i")

for n,s in zip(names,scores):

    print(f"电影名称：{n}---评分：{s}")

然后按照视频中的内容复制一份运行也没结果，不报错

原代码：

import requests

from fake_useragent import UserAgent

from bs4 import BeautifulSoup

url = "https://www.maoyan.com/films?showType=3"

headers = {"User-Agent":UserAgent().edge}

resp = requests.get(url,headers=headers)

bs = BeautifulSoup(resp.text,"lxml")

names = [div.text.strip("") for div in bs.select("div.channel-detail.movie-item-title")]

scores = [div.text.strip("") for div in bs.select(".channel-detail.channel-detail-orange")]

for n,s in zip(names,scores):

    print(f"电影名称：{n}---评分：{s}")

在爬虫这一章，好几次视频操作的网站，运行结果都是这样，无结果也不报错，换百战的官网有时候试试又可以了，不知道哪儿有问题

问题如下：

如何爬取每个城市的数字编号？

网址：51job招聘网

要爬出这种结果：如何爬?

#pip install Appium-Python-Client==2.6.1

#selenium                  4.11.2

from appium import webdriver

from selenium.webdriver.common.by import By

from time import sleep

#设置设备信息

desired = {

    "platformName": "Android",

    "platformVersion": "5.1.1",

    "deviceName": "VOG-AL10",

    "appPackage": "com.android.browser",

    "appActivity": "com.android.browser.BrowserActivity"

}

#设置appium的服务

server = 'http://localhost:4723/wd/hub'

# 新建一个driver

driver = webdriver.Remote(server, options=desired)

#蒙版元素，需要点击一下

value = '/hierarchy/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout[2]/android.widget.LinearLayout/android.widget.FrameLayout/android.webkit.WebView'

driver.find_element(By.XPATH,value).click()

sleep(1)

#定位输入框

input_text = '/hierarchy/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout[2]/android.widget.LinearLayout/android.widget.FrameLayout/android.webkit.WebView/android.webkit.WebView/android.view.View[2]/android.view.View[3]/android.view.View/android.widget.EditText'

input = driver.find_element(By.XPATH,input_text)

#输入内容,清空数据

input.clear()

input.send_keys('音乐')

sleep(1)

#定位按钮

button_text = '/hierarchy/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout[2]/android.widget.LinearLayout/android.widget.FrameLayout/android.webkit.WebView/android.webkit.WebView/android.view.View[2]/android.view.View[3]/android.view.View/android.widget.Button'

button = driver.find_element(By.XPATH,button_text)

#点击按钮

button.click()

sleep(1)

#com.android.browser/.BrowserActivity

启动报错：

Traceback (most recent call last):

  File "d:\python\spiderProject\移动端爬虫\appium的实战.py", line 21, in <module>

    driver = webdriver.Remote(server, options=desired)

  File "D:\virtualenv\spider_env\lib\site-packages\appium\webdriver\webdriver.py", line 230, in __init__

    super().__init__(

TypeError: WebDriver.__init__() got an unexpected keyword argument 'desired_capabilities'

我也是复制全部代码再删的，但出现错误了

return A[e].call(t.exports, t, t.exports, o),

                    ^

TypeError: Cannot read properties of undefined (reading 'call')

老师我想问一个问题，我们做项目的时候用不用scrapy是不是根据项目大小，数据量的多少来决定吗？还是都可以用scrapy？

老师，这个Android是安装不了软件？

老师，

    节日快乐~

    请问课件上用的etree.parse 和代码实例里的etree.HTML的应用场景有什么区别呢？

    我自己随便找了个网页爬取数据parse和html方法都无法使用，请问问题出在哪里呢？

import requests
from fake_useragent import UserAgent
from lxml import etree
url='http://lishishangdejintian.51240.com/'
headers={'User-Agent':UserAgent().random}
proxies={
            "http":"http://61.135.186.80:80"
}
resp=requests.get(url,headers=headers,proxies=proxies,timeout=1)
# resp.encoding='utf-8'
print(resp.text)
e = etree.HTML(resp)
# print(e)
# test = e.xpath('//ul[@class="list"]')
# print(test)

老师，能把能用正则把图片中标记字体匹配一下

老师为啥我这个Tomcat里面啥命令也没有，没有像视频中一样可以显示IP地址

老师，切换虚拟环境用哪个快捷键？