半夜突然醒来，闲着无聊敲了个爬虫的代码，用selenium实现一个自动登录之类的，代码如下所示：

"""
   用 selenium 实现对中国大学mocc的登录
   程序运行报错：正在处理中
"""
from selenium import webdriver
from time import sleep
fox = webdriver.Firefox()
url ='https://www.icourse163.org/member/login.htm?returnUrl=aHR0cHM6Ly93d3cuaWNvdXJzZTE2My5vcmcvaW5kZXguaHRt#/webLoginIndex'
fox.get(url)
sleep(3)

# 点击登录按钮,弹出登录界面
fox.find_element_by_css_selector('#auto-id-1628452551743').click()
sleep(1)
# 选择其他登录方式
fox.find_element_by_css_selector('#login-cnt > div > div > div > div.ux-login-set-scan-code_ft > span').click()
# 获取账号框和密码框，输入密码
fox.find_element_by_css_selector('#auto-id-1628452775609').send_keys('*************')
sleep(1)
fox.find_element_by_css_selector('#auto-id-1628452775612').send_keys('*************')
sleep(1)
fox.find_element_by_css_selector('#dologin').click()
sleep(1)

print(fox.current_url)
print(fox.page_source)
sleep(5)


sleep(2)
fox.quit()

报错信息如下：

C:\Users\Administrator\AppData\Local\Programs\Python\Python39\python.exe D:/pythonProject2/实战python网络爬虫/selenium的使用/selenium_03.py

Traceback (most recent call last):

  File "D:\pythonProject2\实战python网络爬虫\selenium的使用\selenium_03.py", line 13, in <module>

    fox.find_element_by_css_selector('#auto-id-1628452551743').click()

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 598, in find_element_by_css_selector

    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element

    return self.execute(Command.FIND_ELEMENT, {

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute

    self.error_handler.check_response(response)

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response

    raise exception_class(message, screen, stacktrace)

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: #auto-id-1628452551743

无法定位到元素，敢问老师怎么处理？

进程已结束，退出代码为 1

什么你这docker服务已经开启了，你是开启了，我们怎么开

为什么我跑完程序 里面没有user-agent这个说明？

from fake_useragent import UserAgent
import ssl
import requests
from lxml import etree

from time import sleep

def get_html(url):
    '''
    :param url: 要爬取的地址
    :return: 返回html
    '''
    headers = {"User_Agent": UserAgent().random}
    resp = requests.get(url,headers=headers)

    #status_code   返回状态码
    if resp.status_code == 200:
        resp.encoding='utf-8'
        return resp.text
    else:
        return None

def parse_list(html):
    '''
    :param html: 传递进来有一个电影列表的的html
    :return: 返回一个电影的url
    '''

    e = etree.HTML(html)
    # 解决验证CA
    # ssl._create_default_https_context = ssl._create_unverified_context
    list_url = ['https://www.qidian.com{}'.format(url)for url in e.xpath('//div[@class="book-img-box"]/a/@href')]
    return list_url

def parse_index(html):
    '''
    :param html: 传递一个有电影信息的html
    :return: 已经提取好的电影信息
    '''
    e = etree.HTML(html)
    name = e.xpath('//div/h1/span/a/text()')
    return name

def main():
    num = int(input("请输入要获取多少页："))
    for page in range(num):
        url = 'https://www.qidian.com/all?&page={}'.format(page+1)
        list_html = get_html(url)
        list_url = parse_list(list_html)
        for url in list_url:
            info_html = get_html(url)
            move = parse_index(info_html)
            print(move)

if __name__ == '__main__':
    main()

老师，您帮我看一下，为什么我这个最后返回的是空列表啊，我debug看了一下是这个出问题了。返回了空值，但是我使用插件看了看没问题啊

name = e.xpath('//div/h1/span/a/text()')

存在两个疑惑，希望老师可以答疑：

1、scrapy 是否可以理解为已经写好的框架类设计，而前面学习的动态UA，会话cookie在这里依旧有效（实现同样的功能，只是可能表现的方法不同）

2、对于选择器的概念还不是很明白，有没有更浅显直白的相关扩展资料来参考下

老师为啥我按视频里的步骤做完了怎么还弹出来这个

  老师您好，请教几个个非技术性问题：

  （1）就是爬虫学到哪了程度了可以尝试去找工作？是否需要比较精才可以？

     简单介绍一下自己的情况，现在已经学完了爬虫课程，可以登录一些网站爬取信息（主要用的还是cookies登录，验证码登录时而成功时而失败）并保存，scrapy基本操作会了，但真正编写爬虫的时候需要时不时翻阅资料。

     简单概括就是，基本会登录网站爬取数据简单处理之后报存在数据库。

  （2）另外还想问一下，企业爬虫一般用那种登录方式比较多，个人感觉是cookies，不知道是否正确？

  （3）我个人是想往爬虫方向发展的，网络编程基础只学了html、css、javascript，没学框架，这样学习是否合理？

  （4）除了尚学堂的课程，请老师推荐精通爬虫的书籍。

   期待回复，谢谢。

为什么会有这种错误，老师。setting配置和你的一样啊，网上的解决方法都不行啊，是不是scrapy的版本不行，我的是1.8.0的

刚开始那个get请求和老师结果一样，现在这个不知道什么情况？

title乱码正常吗？

老师，pip install fake-useragent命令是下面这种情况

2024-05-22 15:42:18 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: scrapy07)

2024-05-22 15:42:18 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.11.7, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1n  15 Mar 2022), cryptography 36.0.2, Platform Windows-10-10.0.22621-SP0

2024-05-22 15:42:18 [scrapy.crawler] INFO: Overridden settings:

{'BOT_NAME': 'scrapy07',

 'NEWSPIDER_MODULE': 'scrapy07.spiders',

 'SPIDER_MODULES': ['scrapy07.spiders']}

2024-05-22 15:42:18 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor

2024-05-22 15:42:18 [scrapy.extensions.telnet] INFO: Telnet Password: c658f43dc33b4451

2024-05-22 15:42:18 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

 'scrapy.extensions.telnet.TelnetConsole',

 'scrapy.extensions.logstats.LogStats']

Unhandled error in Deferred:

2024-05-22 15:42:18 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\crawler.py", line 206, in crawl  

    return self._crawl(crawler, *args, **kwargs)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\crawler.py", line 210, in _crawl 

    d = crawler.crawl(*args, **kwargs)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 1905, in unwindGenerator

    return _cancellableInlineCallbacks(gen)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 1815, in _cancellableInlineCallbacks

    _inlineCallbacks(None, gen, status)

--- <exception caught here> ---

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks

    result = current_context.run(gen.send, result)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\crawler.py", line 101, in crawl  

    self.spider = self._create_spider(*args, **kwargs)

  File "D:\python_env\spider2_env\Lib\site-packages\scrapy\crawler.py", line 113, in _create_spider

    return self.spidercls.from_crawler(self, *args, **kwargs)

  File "D:\code\python\spider_code\scrapy07\scrapy07\spiders\selenium.py", line 29, in from_crawler

    s = Service(executable_path='scrapy07/chromedriver.exe')

builtins.TypeError: Service.__init__() got an unexpected keyword argument 'executable_path' 

--- Logging error ---

Traceback (most recent call last):

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 1110, in emit      

    msg = self.format(record)

          ^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 953, in format     

    return fmt.format(record)

           ^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 695, in format     

    record.exc_text = self.formatException(record.exc_info)

                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\logging\__init__.py", line 645, in formatException

    traceback.print_exception(ei[0], ei[1], tb, None, sio)

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 124, in print_exception   

    te = TracebackException(type(value), value, tb, limit=limit, compact=True)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 690, in __init__

    self.stack = StackSummary._extract_from_extended_frame_gen(

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 416, in _extract_from_extended_frame_gen

    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 353, in _walk_tb_with_full_positions

    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)

                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "E:\Program Files\Python\Python311\Lib\traceback.py", line 366, in _get_code_position

    positions_gen = code.co_positions()

                    ^^^^^^^^^^^^^^^^^

AttributeError: '_Code' object has no attribute 'co_positions'

Call stack:

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\internet\defer.py", line 344, in __del__

    log.failure(format, self.failResult, debugInfo=debugInfo)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\logger\_logger.py", line 190, in failure

    self.emit(level, format, log_failure=failure, **kwargs)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\logger\_logger.py", line 142, in emit

    self.observer(event)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\logger\_observer.py", line 81, in __call__

    observer(event)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\logger\_legacy.py", line 90, in __call__

    self.legacyObserver(event)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\python\log.py", line 579, in emit

    _publishNew(self._newObserver, eventDict, textFromEventDict)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\logger\_legacy.py", line 147, in publishToNewObserver

    observer(eventDict)

  File "D:\python_env\spider2_env\Lib\site-packages\twisted\logger\_stdlib.py", line 112, in __call__

    self.logger.log(stdlibLevel, StringifiableFromEvent(event), exc_info=excInfo)

Message: <twisted.logger._stdlib.StringifiableFromEvent object at 0x0000026D24D85550>       

Arguments: ()

老师这个报错怎么解决啊？

rs = base64.decodebytes(data.encode())

decodebytes和b64decode是不是都能用

，这里data已经转码成字节类型

我开启断点启用调试模式后，结果就直接运行出来了，在变量区域看不到变量，该怎么设置呢？

老师 mongodb的_ip能不能设计成ip:port?这样就不用写指纹了，mongodb默认是不收录重复的_ip的

老师，之前创建还好好的，现在就不行了，咋回事。