视频中讲解的是

scrapy crawl douban2 -o douban2.json

可以生成json文件

而我自己的必须是

scrapy crawl douban -o douban2.json

才可以生成json文件

老师这种情况是 猫眼的反爬吗 爬了个空页面

为啥打不开网页？

PS D:\PROJECT_python_project\python01\反爬\proxy_sys> uvicorn proxy_api:app --reload    

uvicorn : 无法将“uvicorn”项识别为 cmdlet、函数、脚本文件或可运行程序的名称。请检查名称的

拼写，如果包括路径，请确保路径正确，然后再试一次。

所在位置 行:1 字符: 1

+ uvicorn proxy_api:app --reload

+ ~~~~~~~

    + CategoryInfo          : ObjectNotFound: (uvicorn:String) [], CommandNotFoundExceptio  

   n

    + FullyQualifiedErrorId : CommandNotFoundException

def dai_li_66(page):

    web = WebRequest()

    for i in range(1,page+1):

        url = f"http://www.66ip.cn/{i}.html"

        web.get(url=url)

        e = web.get_tree()

        # 解析数据

        ips = e.xpath('//*[@id="main"]/div[1]/div[2]/div[1]/table/tbody/tr/td[1]/text()')

        ports = e.xpath('//*[@id="main"]/div[1]/div[2]/div[1]/table/tbody/tr/td[2]/text()')

        print(ips)

        for i_,p in zip(ips,ports):

            print({'ip':i_,'port':p,"_type":'http'})

        sleep(3)

为啥返回的是一个空列表呢？

这个报错是什么原因？

import requests
from fake_useragent import UserAgent
from bs4 import BeautifulSoup

def bs4_test():
    url='https://www.maoyan.com/films?showType=3'
    headers={'User-Agennt':UserAgent().chrome}
    resp = requests.get(url,headers=headers)
    soup = BeautifulSoup(resp.text,'lxml')
    names = [div.text.strip('\n') for div in soup.select('div.channel-detail.movie-item-title')]
    # sorces = [div.text.strip('') for div in soup.select('div.channel-detail.channel-detail-orange')]
    # for n,s in zip(names,sorces): 
    #     print(f'名称:{n} 评分:{s}')
    print(names)
    print(resp.text)
if __name__ == '__main__':
    bs4_test()

测试后返回这个错误是什么原因？

我开启断点启用调试模式后，结果就直接运行出来了，在变量区域看不到变量，该怎么设置呢？

File "d:\software\programminglanauageenvironment\pytho3.8.6\lib\runpy.py", line 194, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "d:\software\programminglanauageenvironment\pytho3.8.6\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "D:\software\python_env\alway_env\Scripts\scrapy.exe\__main__.py", line 4, in <module>

  File "D:\software\python_env\alway_env\lib\site-packages\scrapy\__init__.py", line 12, in <module>

    from scrapy.spiders import Spider

  File "D:\software\python_env\alway_env\lib\site-packages\scrapy\spiders\__init__.py", line 93, in <module>

    from scrapy.spiders.crawl import CrawlSpider, Rule

  File "D:\software\python_env\alway_env\lib\site-packages\scrapy\spiders\crawl.py", line 14, in <module>

    from scrapy.utils.spider import iterate_spider_output

  File "D:\software\python_env\alway_env\lib\site-packages\scrapy\utils\spider.py", line 5, in <module>

    from scrapy.utils.defer import deferred_from_coro

  File "D:\software\python_env\alway_env\lib\site-packages\scrapy\utils\defer.py", line 24, in <module>

    from scrapy.utils.reactor import is_asyncio_reactor_installed

  File "D:\software\python_env\alway_env\lib\site-packages\scrapy\utils\reactor.py", line 5, in <module>

    from twisted.internet import asyncioreactor, error

  File "D:\software\python_env\alway_env\lib\site-packages\twisted\internet\asyncioreactor.py", line 19, in <module>

    from twisted.internet.posixbase import (

  File "D:\software\python_env\alway_env\lib\site-packages\twisted\internet\posixbase.py", line 19, in <module>

    from twisted.internet import error, tcp, udp

  File "D:\software\python_env\alway_env\lib\site-packages\twisted\internet\tcp.py", line 38, in <module>

    from twisted.internet._newtls import (

  File "D:\software\python_env\alway_env\lib\site-packages\twisted\internet\_newtls.py", line 18, in <module>

    from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol

  File "D:\software\python_env\alway_env\lib\site-packages\twisted\protocols\tls.py", line 42, in <module>

    from OpenSSL.SSL import Connection, Error, SysCallError, WantReadError, ZeroReturnError

  File "D:\software\python_env\alway_env\lib\site-packages\OpenSSL\__init__.py", line 8, in <module>

    from OpenSSL import SSL, crypto

  File "D:\software\python_env\alway_env\lib\site-packages\OpenSSL\SSL.py", line 19, in <module>

    from OpenSSL.crypto import (

  File "D:\software\python_env\alway_env\lib\site-packages\OpenSSL\crypto.py", line 3224, in <module>

    utils.deprecated(

这是什么原因？pip install cryptography==36.0.2换成这个版本以后报这个错误

没有验证码跟课堂上不一样

没有验证码是什么原因,我扫描登录的

老师，查找scrapyd的安装目录还有个更简单的方法， find / -name scrapyd

找了所有浏览器连负载都没有都不介绍一下

为啥上一秒 还能爬取成功 下一秒就爬取不成功了 这个网站