多POST查询（会话模式）
Multi POST query (session mode)
我正在尝试查询此 site 以获取报价列表。
问题是在收到最终结果之前，我们需要填写 2 个表格（2 POST 个查询）。
这是我目前所做的：
首先，我在设置 cookie 后发送第一个 POST：
library(httr)
set_cookies(.cookies = c(a = "1", b = "2"))
first_url <- "https://compare.switchon.vic.gov.au/submit"
body <- list(energy_category="electricity",
location="home",
"location-home"="shift",
"retailer-company"="",
postcode="3000",
distributor=7,
zone=1,
energy_concession=0,
"file-provider"="",
solar=0,
solar_feedin_tariff="",
disclaimer_chkbox="disclaimer_selected")
qr<- POST(first_url,
encode="form",
body=body)
然后尝试使用第二个 post 查询检索报价：
gov_url <- "https://compare.switchon.vic.gov.au/energy_questionnaire/submit"
qr1<- POST(gov_url,
encode="form",
body=list(`person-count`=1,
`room-count`=1,
`refrigerator-count`=1,
`gas-type`=4,
`pool-heating`=0,
spaceheating="none",
spacecooling="none",
`cloth-dryer`=0,
waterheating="other"),
set_cookies(a = 1, b = 2))
)
library(XML)
dc <- htmlParse(qr1)
但不幸的是，我收到一条指示会话结束的消息。非常感谢您为解决此问题提供的任何帮助。
更新添加 cookie：
我添加了 cookie 和中间 GET，但我仍然没有任何结果。
library(httr)
first_url <- "https://compare.switchon.vic.gov.au/submit"
body <- list(energy_category="electricity",
location="home",
"location-home"="shift",
"retailer-company"="",
postcode=3000,
distributor=7,
zone=1,
energy_concession=0,
"file-provider"="",
solar=0,
solar_feedin_tariff="",
disclaimer_chkbox="disclaimer_selected")
qr<- POST(first_url,
encode="form",
body=body,
config=set_cookies(a = 1, b = 2))
xx <- GET("https://compare.switchon.vic.gov.au/energy_questionnaire",config=set_cookies(a = 1, b = 2))
gov_url <- "https://compare.switchon.vic.gov.au/energy_questionnaire/submit"
qr1<- POST(gov_url,
encode="form",
body=list(
`person-count`=1,
`room-count`=1,
`refrigerator-count`=1,
`gas-type`=4,
`pool-heating`=0,
spaceheating="none",
spacecooling="none",
`cloth-dryer`=0,
waterheating="other"),
config=set_cookies(a = 1, b = 2))
library(XML)
dc <- htmlParse(qr1)
使用具有以下数据的 python requests.Session 对象进入结果页面：
form1 = {"energy_category": "electricity",
"location": "home",
"location-home": "shift",
"distributor": "7",
"postcode": "3000",
"energy_concession": "0",
"solar": "0",
"disclaimer_chkbox": "disclaimer_selected",
}
form2 = {"person-count":"1",
"room-count":"4",
"refrigerator-count":"0",
"gas-type":"3",
"pool-heating":"0",
"spaceheating[]":"none",
"spacecooling[]":"none",
"cloth-dryer":"0",
"waterheating[]":"other"}
sub_url = "https://compare.switchon.vic.gov.au/submit"
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
r = s.get("https://compare.switchon.vic.gov.au/offers")
print(r.content)
您应该在页面上看到的返回 html 中看到匹配的 h1：
<h1>Your electricity offers</h1>
或使用 scrapy 表单请求：
import scrapy
class Spider(scrapy.Spider):
name = 'comp'
start_urls = ['https://compare.switchon.vic.gov.au/energy_questionnaire/submit']
form1 = {"energy_category": "electricity",
"location": "home",
"location-home": "shift",
"distributor": "7",
"postcode": "3000",
"energy_concession": "0",
"solar": "0",
"disclaimer_chkbox": "disclaimer_selected",
}
sub_url = "https://compare.switchon.vic.gov.au/submit"
form2 = {"person-count":"1",
"room-count":"4",
"refrigerator-count":"0",
"gas-type":"3",
"pool-heating":"0",
"spaceheating[]":"none",
"spacecooling[]":"none",
"cloth-dryer":"0",
"waterheating[]":"other"}
def start_requests(self):
return [scrapy.FormRequest(
self.sub_url,
formdata=form1,
callback=self.parse
)]
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata=form2,
callback=self.after
)
def after(self, response):
print("<h1>Your electricity offers</h1>" in response.body)
我们可以验证的是 "<h1>Your electricity offers</h1>":
2016-03-07 12:27:31 [scrapy] DEBUG: Crawled (200) <GET https://compare.switchon.vic.gov.au/offers#list/electricity> (referer: https://compare.switchon.vic.gov.au/energy_questionnaire)
True
2016-03-07 12:27:31 [scrapy] INFO: Closing spider (finished)
下一个问题是实际数据是动态呈现的，如果您查看结果页面的来源，您可以验证这一点，您实际上可以获得 json 格式的所有提供程序：
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
r = s.get("https://compare.switchon.vic.gov.au/service/offers")
print(r.json())
其中的一个片段是：
{u'pageMetaData': {u'showDual': False, u'isGas': False, u'showTouToggle': True, u'isElectricityInitial': True, u'showLoopback': False, u'isElectricity': True}, u'offersList': [{u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.peopleenergy.com.au', u'offerId': u'PEO33707SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1410, u'offerType': u'Standing offer', u'offerName': u'Residential 5-Day Time of Use', u'conditionalPrice': 1410, u'fullDiscountedPrice': 1390, u'greenPower': 0, u'retailerName': u'People Energy', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'1300 788 970', u'isPartDual': False, u'retailerId': u'7322', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1645', u'exitFeeCount': 0, u'timeDefinition': u'Local time', u'retailerImageUrl': u'img/retailers/big/peopleenergy.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.peopleenergy.com.au', u'offerId': u'PEO33773SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1500, u'offerType': u'Standing offer', u'offerName': u'Residential Peak Anytime', u'conditionalPrice': 1500, u'fullDiscountedPrice': 1480, u'greenPower': 0, u'retailerName': u'People Energy', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'1300 788 970', u'isPartDual': False, u'retailerId': u'7322', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1649', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/peopleenergy.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.energythatcould.com.au', u'offerId': u'PAC33683SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1400, u'offerType': u'Standing offer', u'offerName': u'Vic Home Flex', u'conditionalPrice': 1400, u'fullDiscountedPrice': 1400, u'greenPower': 0, u'retailerName': u'Pacific Hydro Retail Pty Ltd', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Flexible Pricing', u'retailerPhone': u'1800 010 648', u'isPartDual': False, u'retailerId': u'15902', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1666', u'exitFeeCount': 0, u'timeDefinition': u'Local time', u'retailerImageUrl': u'img/retailers/big/pachydro.jpg'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.energythatcould.com.au', u'offerId': u'PAC33679SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1340, u'offerType': u'Standing offer', u'offerName': u'Vic Home Flex', u'conditionalPrice': 1340, u'fullDiscountedPrice': 1340, u'greenPower': 0, u'retailerName': u'Pacific Hydro Retail Pty Ltd', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'1800 010 648', u'isPartDual': False, u'retailerId': u'15902', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1680', u'exitFeeCount': 0, u'timeDefinition': u'Local time', u'retailerImageUrl': u'img/retailers/big/pachydro.jpg'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 10, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30367MR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': True, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1370, u'offerType': u'Market offer', u'offerName': u'Citipower Commander Residential Market Offer (CE3CPR-MAT1 + PF1/TF1/GF1)', u'conditionalPrice': 1370, u'fullDiscountedPrice': 1160, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': True, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2384', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 10, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30359MR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': True, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1330, u'offerType': u'Market offer', u'offerName': u'Citipower Commander Residential Market Offer (Flexible Pricing (Peak, Shoulder and Off Peak) (CE3CPR-MCFP1 + PF1/TF1/GF1)', u'conditionalPrice': 1330, u'fullDiscountedPrice': 1140, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': True, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2386', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 10, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E33241MR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': True, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1300, u'offerType': u'Market offer', u'offerName': u'Citipower Commander Residential Market Offer (Peak / Off Peak) (CE3CPR-MPK1OP1)', u'conditionalPrice': 1300, u'fullDiscountedPrice': 1100, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': True, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2389', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30379SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1370, u'offerType': u'Standing offer', u'offerName': u'Citipower Commander Residential Standing Offer (CE3CPR-SAT1 + PF1/TF1/GF1)', u'conditionalPrice': 1370, u'fullDiscountedPrice': 1370, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2391', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30369SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1330, u'offerType': u'Standing offer', u'offerName': u'Citipower Commander Residential Standing Offer (Flexible Pricing (Peak, Shoulder and Off Peak) (CE3CPR-SCFP1 + PF1/TF1/GF1)', u'conditionalPrice': 1330, u'fullDiscountedPrice': 1330, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2393', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30375SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1300, u'offerType': u'Standing offer', u'offerName': u'Citipower Commander Residential Standing Offer (Peak / Off Peak) (CE3CPR-SPK1OP1)', u'conditionalPrice': 1300, u'fullDiscountedPrice': 1300, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2395', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.dodo.com/powerandgas', u'offerId': u'DOD32903SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1320, u'offerType': u'Standing offer', u'offerName': u'Citipower Res No Term Standing Offer (Common Form Flex Plan) (E3CPR-SCFP1)', u'conditionalPrice': 1320, u'fullDiscountedPrice': 1320, u'greenPower': 0, u'retailerName': u'Dodo Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType':
然后如果您稍后查看请求，例如当您单击结果页面上的比较所选按钮时，会有这样的请求：
https://compare.switchon.vic.gov.au/service/offer/tariff/9090/9092
因此，您可以通过使用关税或一些变化进行过滤来模拟发生的情况。
如果您在表格中输入与以下相同的值，您实际上可以获得 json 的所有数据：
form1 = {"energy_category": "electricity",
"location": "home",
"location-home": "shift",
"distributor": "7",
"postcode": "3000",
"energy_concession": "0",
"solar": "0",
"disclaimer_chkbox": "disclaimer_selected"
}
form2 = {"person-count":"1",
"room-count":"1",
"refrigerator-count":"1",
"gas-type":"4",
"pool-heating":"0",
"spaceheating[]":"none",
"spacecooling[]":"none",
"cloth-dryer":"0",
"cloth-dryer-freq-weekday":"",
"waterheating[]":"other"}
import json
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
js = s.get("https://compare.switchon.vic.gov.au/service/offers").json()["offersList"]
by_discount = sorted(js, key=lambda d: d["offerDetails"][0]["fullDiscountedPrice"])
如果我们只从按总折扣价排序的列表中提取前两个值：
from pprint import pprint as pp
pp(by_discount[:2])
您将获得：
[{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 980,
u'conditionalPrice': 980,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 10,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 1,
u'fullDiscountedPrice': 660,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': True,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': False,
u'offerId': u'GLO40961MR',
u'offerKey': u'7636',
u'offerName': u'GLO SWITCH',
u'offerType': u'Market offer',
u'retailerId': u'31206',
u'retailerImageUrl': u'img/retailers/big/globird.jpg',
u'retailerName': u'GloBird Energy',
u'retailerPhone': u'(03) 8560 4199',
u'retailerUrl': u'http://www.globirdenergy.com.au/switchon/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Single rate',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0},
{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 1080,
u'conditionalPrice': 1080,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 10,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 1,
u'fullDiscountedPrice': 720,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': True,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': True,
u'offerId': u'GLO41009MR',
u'offerKey': u'7642',
u'offerName': u'GLO SWITCH',
u'offerType': u'Market offer',
u'retailerId': u'31206',
u'retailerImageUrl': u'img/retailers/big/globird.jpg',
u'retailerName': u'GloBird Energy',
u'retailerPhone': u'(03) 8560 4199',
u'retailerUrl': u'http://www.globirdenergy.com.au/switchon/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Time of use',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0}]
当您单击 "DISCOUNTED PRICE" 过滤器按钮时，它应该与您在页面上看到的相匹配。
对于正常视图，它似乎按 conditionalPrice 或 basePrice 排序，再次拉取前两个值应该与您在网页上看到的相匹配：
base = sorted(js, key=lambda d: d["offerDetails"][0]["conditionalPrice"])
from pprint import pprint as pp
pp(base[:2])
[{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 740,
u'conditionalPrice': 740,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 0,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 0,
u'fullDiscountedPrice': 740,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': False,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': False,
u'offerId': u'NEX42694SR',
u'offerKey': u'9092',
u'offerName': u'Citpower Single Rate Residential',
u'offerType': u'Standing offer',
u'retailerId': u'35726',
u'retailerImageUrl': u'img/retailers/big/nextbusinessenergy.jpg',
u'retailerName': u'Next Business Energy Pty Ltd',
u'retailerPhone': u'1300 466 398',
u'retailerUrl': u'http://www.nextbusinessenergy.com.au/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Single rate',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0},
{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 780,
u'conditionalPrice': 780,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 0,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 0,
u'fullDiscountedPrice': 780,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': False,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': False,
u'offerId': u'NEX42699SR',
u'offerKey': u'9090',
u'offerName': u'Citpower Residential Flexible Pricing',
u'offerType': u'Standing offer',
u'retailerId': u'35726',
u'retailerImageUrl': u'img/retailers/big/nextbusinessenergy.jpg',
u'retailerName': u'Next Business Energy Pty Ltd',
u'retailerPhone': u'1300 466 398',
u'retailerUrl': u'http://www.nextbusinessenergy.com.au/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Flexible Pricing',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0}]
如果您单击 https://compare.switchon.vic.gov.au/service/offers 获取条目然后点击 response，您可以在 firebug 控制台中看到返回的所有 json:
您应该能够从中提取您想要的每个字段。
输出实际上有一些额外的结果，除非您切换下面的 tou 按钮，否则您不会在页面上看到这些结果：
您可以从结果中过滤掉这些结果，以便与默认输出完全匹配，或者提供一个选项以包含辅助函数：
def order_by(l, k, is_tou=False):
if not is_tou:
filt = filter(lambda x: not x["offerDetails"][0]["isTouOffer"], l)
return sorted(filt, key=lambda d: d["offerDetails"][0][k])
return sorted(l, key=lambda d: d["offerDetails"][0][k])
import json
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
js = s.get("https://compare.switchon.vic.gov.au/service/offers").json()["offersList"]
by_price = by_discount(js, "conditionalPrice", False)
print(by_price[:3)
如果你检查输出，你会看到第三个来源能源在结果中的价格为 840 开关打开或 AGL 关闭时为 860，你可以将相同的应用到折扣输出：
常规输出似乎也按 conditionalPrice 排序，如果您检查源代码，则调用排序的两个 js 函数是：
ng-click="changeSortingField('conditionalPrice')"
ng-click="changeSortingField('fullDiscountedPrice')"
所以现在应该完全匹配站点输出。
我正在尝试查询此 site 以获取报价列表。 问题是在收到最终结果之前，我们需要填写 2 个表格（2 POST 个查询）。
这是我目前所做的：
首先，我在设置 cookie 后发送第一个 POST：
library(httr)
set_cookies(.cookies = c(a = "1", b = "2"))
first_url <- "https://compare.switchon.vic.gov.au/submit"
body <- list(energy_category="electricity",
location="home",
"location-home"="shift",
"retailer-company"="",
postcode="3000",
distributor=7,
zone=1,
energy_concession=0,
"file-provider"="",
solar=0,
solar_feedin_tariff="",
disclaimer_chkbox="disclaimer_selected")
qr<- POST(first_url,
encode="form",
body=body)
然后尝试使用第二个 post 查询检索报价：
gov_url <- "https://compare.switchon.vic.gov.au/energy_questionnaire/submit"
qr1<- POST(gov_url,
encode="form",
body=list(`person-count`=1,
`room-count`=1,
`refrigerator-count`=1,
`gas-type`=4,
`pool-heating`=0,
spaceheating="none",
spacecooling="none",
`cloth-dryer`=0,
waterheating="other"),
set_cookies(a = 1, b = 2))
)
library(XML)
dc <- htmlParse(qr1)
但不幸的是，我收到一条指示会话结束的消息。非常感谢您为解决此问题提供的任何帮助。
更新添加 cookie：
我添加了 cookie 和中间 GET，但我仍然没有任何结果。
library(httr)
first_url <- "https://compare.switchon.vic.gov.au/submit"
body <- list(energy_category="electricity",
location="home",
"location-home"="shift",
"retailer-company"="",
postcode=3000,
distributor=7,
zone=1,
energy_concession=0,
"file-provider"="",
solar=0,
solar_feedin_tariff="",
disclaimer_chkbox="disclaimer_selected")
qr<- POST(first_url,
encode="form",
body=body,
config=set_cookies(a = 1, b = 2))
xx <- GET("https://compare.switchon.vic.gov.au/energy_questionnaire",config=set_cookies(a = 1, b = 2))
gov_url <- "https://compare.switchon.vic.gov.au/energy_questionnaire/submit"
qr1<- POST(gov_url,
encode="form",
body=list(
`person-count`=1,
`room-count`=1,
`refrigerator-count`=1,
`gas-type`=4,
`pool-heating`=0,
spaceheating="none",
spacecooling="none",
`cloth-dryer`=0,
waterheating="other"),
config=set_cookies(a = 1, b = 2))
library(XML)
dc <- htmlParse(qr1)
使用具有以下数据的 python requests.Session 对象进入结果页面：
form1 = {"energy_category": "electricity",
"location": "home",
"location-home": "shift",
"distributor": "7",
"postcode": "3000",
"energy_concession": "0",
"solar": "0",
"disclaimer_chkbox": "disclaimer_selected",
}
form2 = {"person-count":"1",
"room-count":"4",
"refrigerator-count":"0",
"gas-type":"3",
"pool-heating":"0",
"spaceheating[]":"none",
"spacecooling[]":"none",
"cloth-dryer":"0",
"waterheating[]":"other"}
sub_url = "https://compare.switchon.vic.gov.au/submit"
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
r = s.get("https://compare.switchon.vic.gov.au/offers")
print(r.content)
您应该在页面上看到的返回 html 中看到匹配的 h1：
<h1>Your electricity offers</h1>
或使用 scrapy 表单请求：
import scrapy
class Spider(scrapy.Spider):
name = 'comp'
start_urls = ['https://compare.switchon.vic.gov.au/energy_questionnaire/submit']
form1 = {"energy_category": "electricity",
"location": "home",
"location-home": "shift",
"distributor": "7",
"postcode": "3000",
"energy_concession": "0",
"solar": "0",
"disclaimer_chkbox": "disclaimer_selected",
}
sub_url = "https://compare.switchon.vic.gov.au/submit"
form2 = {"person-count":"1",
"room-count":"4",
"refrigerator-count":"0",
"gas-type":"3",
"pool-heating":"0",
"spaceheating[]":"none",
"spacecooling[]":"none",
"cloth-dryer":"0",
"waterheating[]":"other"}
def start_requests(self):
return [scrapy.FormRequest(
self.sub_url,
formdata=form1,
callback=self.parse
)]
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata=form2,
callback=self.after
)
def after(self, response):
print("<h1>Your electricity offers</h1>" in response.body)
我们可以验证的是 "<h1>Your electricity offers</h1>":
2016-03-07 12:27:31 [scrapy] DEBUG: Crawled (200) <GET https://compare.switchon.vic.gov.au/offers#list/electricity> (referer: https://compare.switchon.vic.gov.au/energy_questionnaire)
True
2016-03-07 12:27:31 [scrapy] INFO: Closing spider (finished)
下一个问题是实际数据是动态呈现的，如果您查看结果页面的来源，您可以验证这一点，您实际上可以获得 json 格式的所有提供程序：
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
r = s.get("https://compare.switchon.vic.gov.au/service/offers")
print(r.json())
其中的一个片段是：
{u'pageMetaData': {u'showDual': False, u'isGas': False, u'showTouToggle': True, u'isElectricityInitial': True, u'showLoopback': False, u'isElectricity': True}, u'offersList': [{u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.peopleenergy.com.au', u'offerId': u'PEO33707SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1410, u'offerType': u'Standing offer', u'offerName': u'Residential 5-Day Time of Use', u'conditionalPrice': 1410, u'fullDiscountedPrice': 1390, u'greenPower': 0, u'retailerName': u'People Energy', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'1300 788 970', u'isPartDual': False, u'retailerId': u'7322', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1645', u'exitFeeCount': 0, u'timeDefinition': u'Local time', u'retailerImageUrl': u'img/retailers/big/peopleenergy.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.peopleenergy.com.au', u'offerId': u'PEO33773SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1500, u'offerType': u'Standing offer', u'offerName': u'Residential Peak Anytime', u'conditionalPrice': 1500, u'fullDiscountedPrice': 1480, u'greenPower': 0, u'retailerName': u'People Energy', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'1300 788 970', u'isPartDual': False, u'retailerId': u'7322', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1649', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/peopleenergy.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.energythatcould.com.au', u'offerId': u'PAC33683SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1400, u'offerType': u'Standing offer', u'offerName': u'Vic Home Flex', u'conditionalPrice': 1400, u'fullDiscountedPrice': 1400, u'greenPower': 0, u'retailerName': u'Pacific Hydro Retail Pty Ltd', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Flexible Pricing', u'retailerPhone': u'1800 010 648', u'isPartDual': False, u'retailerId': u'15902', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1666', u'exitFeeCount': 0, u'timeDefinition': u'Local time', u'retailerImageUrl': u'img/retailers/big/pachydro.jpg'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.energythatcould.com.au', u'offerId': u'PAC33679SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1340, u'offerType': u'Standing offer', u'offerName': u'Vic Home Flex', u'conditionalPrice': 1340, u'fullDiscountedPrice': 1340, u'greenPower': 0, u'retailerName': u'Pacific Hydro Retail Pty Ltd', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'1800 010 648', u'isPartDual': False, u'retailerId': u'15902', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'1680', u'exitFeeCount': 0, u'timeDefinition': u'Local time', u'retailerImageUrl': u'img/retailers/big/pachydro.jpg'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 10, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30367MR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': True, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1370, u'offerType': u'Market offer', u'offerName': u'Citipower Commander Residential Market Offer (CE3CPR-MAT1 + PF1/TF1/GF1)', u'conditionalPrice': 1370, u'fullDiscountedPrice': 1160, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': True, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2384', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 10, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30359MR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': True, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1330, u'offerType': u'Market offer', u'offerName': u'Citipower Commander Residential Market Offer (Flexible Pricing (Peak, Shoulder and Off Peak) (CE3CPR-MCFP1 + PF1/TF1/GF1)', u'conditionalPrice': 1330, u'fullDiscountedPrice': 1140, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': True, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2386', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 10, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E33241MR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': True, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1300, u'offerType': u'Market offer', u'offerName': u'Citipower Commander Residential Market Offer (Peak / Off Peak) (CE3CPR-MPK1OP1)', u'conditionalPrice': 1300, u'fullDiscountedPrice': 1100, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': True, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2389', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30379SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1370, u'offerType': u'Standing offer', u'offerName': u'Citipower Commander Residential Standing Offer (CE3CPR-SAT1 + PF1/TF1/GF1)', u'conditionalPrice': 1370, u'fullDiscountedPrice': 1370, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Single rate', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': False, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2391', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30369SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1330, u'offerType': u'Standing offer', u'offerName': u'Citipower Commander Residential Standing Offer (Flexible Pricing (Peak, Shoulder and Off Peak) (CE3CPR-SCFP1 + PF1/TF1/GF1)', u'conditionalPrice': 1330, u'fullDiscountedPrice': 1330, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2393', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.commander.com', u'offerId': u'M2E30375SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1300, u'offerType': u'Standing offer', u'offerName': u'Citipower Commander Residential Standing Offer (Peak / Off Peak) (CE3CPR-SPK1OP1)', u'conditionalPrice': 1300, u'fullDiscountedPrice': 1300, u'greenPower': 0, u'retailerName': u'Commander Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType': u'Time of use', u'retailerPhone': u'13 39 14', u'isPartDual': False, u'retailerId': u'13667', u'isTouOffer': True, u'solarType': None, u'estimatedSolarCredit': 0, u'offerKey': u'2395', u'exitFeeCount': 0, u'timeDefinition': u'AEST only', u'retailerImageUrl': u'img/retailers/big/commanderpowergas.png'}], u'isClosed': False, u'isChecked': False, u'offerFuelType': 0}, {u'offerDetails': [{u'coolingOffPeriod': 0, u'retailerUrl': u'www.dodo.com/powerandgas', u'offerId': u'DOD32903SR', u'contractLengthCount': 1, u'exitFee': [0], u'hasIncentive': False, u'tariffDetails': {}, u'greenpowerAmount': 0, u'isDirectDebitOnly': False, u'basePrice': 1320, u'offerType': u'Standing offer', u'offerName': u'Citipower Res No Term Standing Offer (Common Form Flex Plan) (E3CPR-SCFP1)', u'conditionalPrice': 1320, u'fullDiscountedPrice': 1320, u'greenPower': 0, u'retailerName': u'Dodo Power & Gas', u'intrinsicGreenpowerPercentage': u'0.0000', u'contractLength': [u'None'], u'hasPayOnTimeDiscount': False, u'greenpowerChargeType': None, u'tariffType':
然后如果您稍后查看请求，例如当您单击结果页面上的比较所选按钮时，会有这样的请求：
https://compare.switchon.vic.gov.au/service/offer/tariff/9090/9092
因此，您可以通过使用关税或一些变化进行过滤来模拟发生的情况。
如果您在表格中输入与以下相同的值，您实际上可以获得 json 的所有数据：
form1 = {"energy_category": "electricity",
"location": "home",
"location-home": "shift",
"distributor": "7",
"postcode": "3000",
"energy_concession": "0",
"solar": "0",
"disclaimer_chkbox": "disclaimer_selected"
}
form2 = {"person-count":"1",
"room-count":"1",
"refrigerator-count":"1",
"gas-type":"4",
"pool-heating":"0",
"spaceheating[]":"none",
"spacecooling[]":"none",
"cloth-dryer":"0",
"cloth-dryer-freq-weekday":"",
"waterheating[]":"other"}
import json
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
js = s.get("https://compare.switchon.vic.gov.au/service/offers").json()["offersList"]
by_discount = sorted(js, key=lambda d: d["offerDetails"][0]["fullDiscountedPrice"])
如果我们只从按总折扣价排序的列表中提取前两个值：
from pprint import pprint as pp
pp(by_discount[:2])
您将获得：
[{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 980,
u'conditionalPrice': 980,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 10,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 1,
u'fullDiscountedPrice': 660,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': True,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': False,
u'offerId': u'GLO40961MR',
u'offerKey': u'7636',
u'offerName': u'GLO SWITCH',
u'offerType': u'Market offer',
u'retailerId': u'31206',
u'retailerImageUrl': u'img/retailers/big/globird.jpg',
u'retailerName': u'GloBird Energy',
u'retailerPhone': u'(03) 8560 4199',
u'retailerUrl': u'http://www.globirdenergy.com.au/switchon/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Single rate',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0},
{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 1080,
u'conditionalPrice': 1080,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 10,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 1,
u'fullDiscountedPrice': 720,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': True,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': True,
u'offerId': u'GLO41009MR',
u'offerKey': u'7642',
u'offerName': u'GLO SWITCH',
u'offerType': u'Market offer',
u'retailerId': u'31206',
u'retailerImageUrl': u'img/retailers/big/globird.jpg',
u'retailerName': u'GloBird Energy',
u'retailerPhone': u'(03) 8560 4199',
u'retailerUrl': u'http://www.globirdenergy.com.au/switchon/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Time of use',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0}]
当您单击 "DISCOUNTED PRICE" 过滤器按钮时，它应该与您在页面上看到的相匹配。
对于正常视图，它似乎按 conditionalPrice 或 basePrice 排序，再次拉取前两个值应该与您在网页上看到的相匹配：
base = sorted(js, key=lambda d: d["offerDetails"][0]["conditionalPrice"])
from pprint import pprint as pp
pp(base[:2])
[{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 740,
u'conditionalPrice': 740,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 0,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 0,
u'fullDiscountedPrice': 740,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': False,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': False,
u'offerId': u'NEX42694SR',
u'offerKey': u'9092',
u'offerName': u'Citpower Single Rate Residential',
u'offerType': u'Standing offer',
u'retailerId': u'35726',
u'retailerImageUrl': u'img/retailers/big/nextbusinessenergy.jpg',
u'retailerName': u'Next Business Energy Pty Ltd',
u'retailerPhone': u'1300 466 398',
u'retailerUrl': u'http://www.nextbusinessenergy.com.au/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Single rate',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0},
{u'isChecked': False,
u'isClosed': False,
u'offerDetails': [{u'basePrice': 780,
u'conditionalPrice': 780,
u'contractLength': [u'None'],
u'contractLengthCount': 1,
u'coolingOffPeriod': 0,
u'estimatedSolarCredit': 0,
u'exitFee': [0],
u'exitFeeCount': 0,
u'fullDiscountedPrice': 780,
u'greenPower': 0,
u'greenpowerAmount': 0,
u'greenpowerChargeType': None,
u'hasIncentive': False,
u'hasPayOnTimeDiscount': False,
u'intrinsicGreenpowerPercentage': u'0.0000',
u'isDirectDebitOnly': False,
u'isPartDual': False,
u'isTouOffer': False,
u'offerId': u'NEX42699SR',
u'offerKey': u'9090',
u'offerName': u'Citpower Residential Flexible Pricing',
u'offerType': u'Standing offer',
u'retailerId': u'35726',
u'retailerImageUrl': u'img/retailers/big/nextbusinessenergy.jpg',
u'retailerName': u'Next Business Energy Pty Ltd',
u'retailerPhone': u'1300 466 398',
u'retailerUrl': u'http://www.nextbusinessenergy.com.au/',
u'solarType': None,
u'tariffDetails': {},
u'tariffType': u'Flexible Pricing',
u'timeDefinition': u'Local time'}],
u'offerFuelType': 0}]
如果您单击 https://compare.switchon.vic.gov.au/service/offers 获取条目然后点击 response，您可以在 firebug 控制台中看到返回的所有 json:
您应该能够从中提取您想要的每个字段。
输出实际上有一些额外的结果，除非您切换下面的 tou 按钮，否则您不会在页面上看到这些结果：
您可以从结果中过滤掉这些结果，以便与默认输出完全匹配，或者提供一个选项以包含辅助函数：
def order_by(l, k, is_tou=False):
if not is_tou:
filt = filter(lambda x: not x["offerDetails"][0]["isTouOffer"], l)
return sorted(filt, key=lambda d: d["offerDetails"][0][k])
return sorted(l, key=lambda d: d["offerDetails"][0][k])
import json
with requests.Session() as s:
s.post(sub_url, data=form1)
r = (s.get("https://compare.switchon.vic.gov.au/energy_questionnaire"))
s.post("https://compare.switchon.vic.gov.au/energy_questionnaire/submit",
data=form2)
js = s.get("https://compare.switchon.vic.gov.au/service/offers").json()["offersList"]
by_price = by_discount(js, "conditionalPrice", False)
print(by_price[:3)
如果你检查输出，你会看到第三个来源能源在结果中的价格为 840 开关打开或 AGL 关闭时为 860，你可以将相同的应用到折扣输出：
常规输出似乎也按 conditionalPrice 排序，如果您检查源代码，则调用排序的两个 js 函数是：
ng-click="changeSortingField('conditionalPrice')"
ng-click="changeSortingField('fullDiscountedPrice')"
所以现在应该完全匹配站点输出。