Scrapy 表单未正确提交

Question

我想提交包含以下 5 项数据的表单。通过提交表格，我可以得到重定向 URL。我不知道问题出在哪里。谁能帮我提交包含所需信息的表格。获取下一页 URL.

供您参考的代码：

import requests
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "uksite"
    login_url = 'https://myeplanning.oxfordshire.gov.uk/Disclaimer/Accept?returnUrl=%2FSearch%2FAdvanced'
    start_urls = [login_url]
    def parse(self, response):
        token = response.css('input[name="__RequestVerificationToken"]::attr(value)').extract_first()
        data = {'__RequestVerificationToken': token,
                'DateReceivedFrom': '2021-04-07',
                'DateReceivedTo': '2021-04-08',
                'AdvancedSearch': 'True',
                'SearchPlanning': 'True',
                }


        yield scrapy.FormRequest.from_response(response,
                                               url=self.login_url,
                                               formdata= data,
                                               clickdata={'class': 'occlss-button occlss-button--primary decompress'},
                                               callback = self.value,
                                               )
    def value(self, response):
   
        print(response._url)

输入URL ==> https://myeplanning.oxfordshire.gov.uk/Disclaimer/Accept?returnUrl=%2FSearch%2FAdvanced

给定输入的输出 URL ==> https://myeplanning.oxfordshire.gov.uk/Planning/Display/MW.0047/21

Answer 1

好的，这样就可以了。

class MyePlanningSpider(scrapy.Spider):
    name = "myeplanning"
    start_urls = ['https://myeplanning.oxfordshire.gov.uk/Disclaimer/Accept?returnUrl=%2FSearch%2FAdvanced']
    login_url = 'https://myeplanning.oxfordshire.gov.uk/Search/Results'

    def parse(self, response):
        data = {
            '__RequestVerificationToken': response.css('input[name="__RequestVerificationToken"]::attr(value)').get(),
            'DateReceivedFrom': '2021-04-07',
            'DateReceivedTo': '2021-04-08',
            'AdvancedSearch': 'True',
            'SearchPlanning': 'True',
        }

        yield scrapy.FormRequest(
            url=self.login_url,
            formdata= data,
            callback=self.parse_value,
        )

    def parse_value(self, response):
        print(response.url)

Scrapy 表单未正确提交

Scrapy form not submitting properly

python

scrapy

web-scraping

python-3.x