老师，我这个是？

PS D:\vscodeproject2\爬虫\Scarpy\scarpy05> & D:/python_env/spider2_env_/Scripts/Activate.ps1

(spider2_env_) PS D:\vscodeproject2\爬虫\Scarpy\scarpy05> & D:/python_env/spider2_env_/Scripts/python.exe d:/vscodeproject2/爬虫/Scarpy/scarpy05/scarpy05/begin.py

2023-12-26 22:01:30 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: scarpy05)

2023-12-26 22:01:30 [scrapy.utils.log] INFO: Versions: lxml 4.8.0.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1n  15 Mar 2022), cryptography 36.0.2, Platform Windows-10-10.0.19045-SP0

2023-12-26 22:01:30 [scrapy.crawler] INFO: Overridden settings:

{'BOT_NAME': 'scarpy05',

 'NEWSPIDER_MODULE': 'scarpy05.spiders',

 'SPIDER_MODULES': ['scarpy05.spiders']}

2023-12-26 22:01:30 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor

2023-12-26 22:01:30 [scrapy.extensions.telnet] INFO: Telnet Password: 2e9c559873783f27

2023-12-26 22:01:30 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

 'scrapy.extensions.telnet.TelnetConsole',

 'scrapy.extensions.logstats.LogStats']

2023-12-26 22:01:30 [scrapy.middleware] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',   

 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',

 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',

 'scrapy.downloadermiddlewares.retry.RetryMiddleware',

 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',

 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',

 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',

 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',

 'scrapy.downloadermiddlewares.stats.DownloaderStats']

2023-12-26 22:01:30 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',

 'scrapy.spidermiddlewares.referer.RefererMiddleware',

 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',

 'scrapy.spidermiddlewares.depth.DepthMiddleware']

Unhandled error in Deferred:

2023-12-26 22:01:30 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\crawler.py", line 206, in crawl

    return self._crawl(crawler, *args, **kwargs)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\crawler.py", line 210, in _crawl

    d = crawler.crawl(*args, **kwargs)

  File "D:\python_env\spider2_env_\lib\site-packages\twisted\internet\defer.py", line 1905, in unwindGenerator

    return _cancellableInlineCallbacks(gen)

  File "D:\python_env\spider2_env_\lib\site-packages\twisted\internet\defer.py", line 1815, in _cancellableInlineCallbacks

    _inlineCallbacks(None, gen, status)

--- <exception caught here> ---

  File "D:\python_env\spider2_env_\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks

    result = current_context.run(gen.send, result)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\crawler.py", line 102, in crawl

    self.engine = self._create_engine()

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\crawler.py", line 116, in _create_engine

    return ExecutionEngine(self, lambda _: self.stop())

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\core\engine.py", line 84, in __init__

    self.scraper = Scraper(crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\core\scraper.py", line 75, in __init__

    self.itemproc = itemproc_cls.from_crawler(crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\middleware.py", line 59, in from_crawler

    return cls.from_settings(crawler.settings, crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\middleware.py", line 41, in from_settings

    mw = create_instance(mwcls, settings, crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\utils\misc.py", line 166, in create_instance

    instance = objcls.from_crawler(crawler, *args, **kwargs)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\media.py", line 76, in from_crawler

    pipe = cls.from_settings(crawler.settings)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\images.py", line 112, in from_settings

    return cls(store_uri, settings=settings)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\images.py", line 55, in __init__

    super().__init__(store_uri, settings=settings, download_func=download_func)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\files.py", line 329, in __init__

    self.store = self._get_store(store_uri)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\files.py", line 378, in _get_store

    store_cls = self.STORE_SCHEMES[scheme]

builtins.KeyError: 'd'

2023-12-26 22:01:30 [twisted] CRITICAL:

Traceback (most recent call last):

  File "D:\python_env\spider2_env_\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks

    result = current_context.run(gen.send, result)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\crawler.py", line 102, in crawl

    self.engine = self._create_engine()

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\crawler.py", line 116, in _create_engine

    return ExecutionEngine(self, lambda _: self.stop())

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\core\engine.py", line 84, in __init__

    self.scraper = Scraper(crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\core\scraper.py", line 75, in __init__

    self.itemproc = itemproc_cls.from_crawler(crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\middleware.py", line 59, in from_crawler

    return cls.from_settings(crawler.settings, crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\middleware.py", line 41, in from_settings

    mw = create_instance(mwcls, settings, crawler)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\utils\misc.py", line 166, in create_instance

    instance = objcls.from_crawler(crawler, *args, **kwargs)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\media.py", line 76, in from_crawler

    pipe = cls.from_settings(crawler.settings)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\images.py", line 112, in from_settings

    return cls(store_uri, settings=settings)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\images.py", line 55, in __init__

    super().__init__(store_uri, settings=settings, download_func=download_func)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\files.py", line 329, in __init__

    self.store = self._get_store(store_uri)

  File "D:\python_env\spider2_env_\lib\site-packages\scrapy\pipelines\files.py", line 378, in _get_store

    store_cls = self.STORE_SCHEMES[scheme]

KeyError: 'd'

(spider2_env_) PS D:\vscodeproject2\爬虫\Scarpy\scarpy05>

老师安装这个scrapy模块总是报错安装不成功怎么办

老师我一直都在循环的拿到第一页的三十条数据，九十条数据都是三十条三十条一重复。？？

老师，如何在网页源代码里面调出query黑色命令行

在爬取小说的时候，每个章节里面的xml格式不一样，都要重新写prase方法么？这么爬取的效率也不要高啊，我现在爬取的

http://www.boquku.com/book/47192/140257486.html

开始就乱了，不是乱一两页，是后面的很多都乱了，这样的要怎么爬

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver import ChromeOptions
from time import sleep

# 将浏览器设置为无头模式
def test_headless():
    # 设置参数，将浏览器设置为无头浏览器
    option = ChromeOptions()  # 设置属性
    option.add_argument('--headless')  # 增加内容

    # 设置驱动
    s = Service(executable_path='./chromedriver.exe')
    # 启动Chrome浏览器
    driver = webdriver.Chrome(service=s,options=option)
    # 访问页面
    driver.get('http://www.baidu.com')
    # 打印代码
    print(driver.page_source)
    # 关闭浏览器
    driver.quit()


# 给浏览器增加代理
def test_proxy1():
    # 设置参数，将浏览器设置为无头浏览器
    option = ChromeOptions()
    # 设置属性option.add_argument('--proxy-server=ip:port')
    option.add_argument('--proxy-server=http://221.199.36.122:35414')  # 增加内容

    # 设置驱动
    s = Service(executable_path='./chromedriver.exe')
    # 启动Chrome浏览器
    driver = webdriver.Chrome(service=s,options=option)
    # 访问页面
    driver.get('http://httpbin.org/get')
    # 打印代码
    print(driver.page_source)
    # 关闭浏览器
    driver.quit()


# 防止检测
def test_find():
    # 老版
    options = ChromeOptions()
    options.add_experimental_option('excludeSwitches', ['enable - automation'])
    options.add_experimental_option('useAutomationExtension', False)

    # 新版
    chrome = webdriver.Chrome(chrome_options=options)
    chrome.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": """
       Object.defineProperty(navigator,
    'webdriver', {
       get: () => false
       })
     """
    })
    chrome.get('http://httpbin.org/get')
    info = chrome.page_source
    print(info)
    sleep(100)



if __name__ == '__main__':
    test_find()

是这样的

from urllib.request import urlopen
url = '
response =urlopen(url)
info = response.read()
print(info)

运行时，出现如下错误，是什么原因？

/Users/kola/Downloads/python百战课程/1/venv/bin/python /Users/kola/Downloads/python百战课程/1/pachong.py

Traceback (most recent call last):

  File "/Users/kola/Downloads/python百战课程/1/pachong.py", line 4, in <module>

    response=urlopen(url)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 222, in urlopen

    return opener.open(url, data, timeout)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 522, in open

    req = meth(req)

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1274, in do_request_

    raise URLError('no host given')

urllib.error.URLError: <urlopen error no host given>

Process finished with exit code 1

这个文档的资料，关于这一节Docker文档的资料

老师，我没使用cookie怎么直接登上来了，我试验网易云音乐也是，直接登上来了。

from urllib.request import Request,urlopen
from fake_useragent import UserAgent

url='https://www.itbaizhan.cn/'

headers={
    "User-Agent":UserAgent().random
}
request=Request(url,headers=headers)

rep=urlopen(request)
print(rep.read().decode())

老师，如何在网页源代码里面调出query黑色命令行

这个代码拿到了然后呢

老师请问一下，创建scrapy怎么总是缺失错误，该怎么解决？

老师，css选择器怎么获取文本啊，还有特殊属性的获取

现在瓜子网用这个方式也加载不出来了呀，docker这块的文档，下载里也没有

老师，设置存储库的时候，报错，网络不可达