logo
down
shadow

Getting error trying to formrequest login page via scrapy shell


Getting error trying to formrequest login page via scrapy shell

By : user2185532
Date : November 22 2020, 04:01 AM
I wish did fix the issue. In the formdata, you are passing is_rememberme as an integer (0), try to put it as a string:
code :
frmdata = {"captcha":"","email":"MYEMAIL","password":"MYPASS","is_rememberme":"0","locale":"en_US","auth_type":"bearer_token","date":"2018-12-13T09:56:22.957Z"}


Share : facebook icon twitter icon
Scrapy FormRequest , trying to send a post request (FormRequest) with currency change formdata

Scrapy FormRequest , trying to send a post request (FormRequest) with currency change formdata


By : Wayne Chen
Date : March 29 2020, 07:55 AM
hope this fix your issue It is all about the frontend cookie, I will show you how to do it with requests first, the logic will be exactly the same with Scrapy:
code :
head = {        "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"}
#
import requests
from bs4 import BeautifulSoup

with requests.Session() as s:
    soup = BeautifulSoup(s.get("https://www.mooda.com/en/").content)
    r2 = s.get(soup.select_one("#selectCurrency option[value*=SAR]")["value"])
    r = s.get("https://www.mooda.com/en/", params={"currency": "sar"}, headers=head, cookies=dict(r2.cookies.items()))
    soup2 = BeautifulSoup(r.content)
    print(soup2.select_one(".price").text)
In [9]: with requests.Session() as s:
   ...:         soup = BeautifulSoup(s.get("https://www.mooda.com/en/").content,"lxml")
   ...:         r2 = s.get(soup.select_one("#selectCurrency option[value*=SAR]")["value"])
   ...:         r = s.get("https://www.mooda.com/en/", params={"currency": "sar"}, headers=head, cookies=dict(r2.cookies.items()))
   ...:         soup2 = BeautifulSoup(r.content,"lxml")
   ...:         print(soup2.select_one(".price").text)
   ...:     

825 SR
class S(Spider):
    name = "foo"
    allowed_domains = ["www.mooda.com"]
    start_urls = ["https://www.mooda.com/en"]

    def parse(self, resp):
        curr = resp.css("#selectCurrency option[value*='SAR']::attr(value)").extract_first()
        return Request(curr, callback=self.parse2)

    def parse2(self, resp):
        print( resp.headers.getlist('Set-Cookie'))
        return Request("https://www.mooda.com/en?currency=sar",cookies=cookies, callback=self.parse3)

    def parse3(self, resp):
        print(resp.css('.price').xpath('text()').extract())
['frontend=c95er9h1at2srhtqu5rkfo13g0; expires=Wed, 28-Jun-2017 08:56:08 GMT; path=/; domain=www.mooda.com', 'currency=SAR; expires=Wed, 28-Jun-2017 08:56:08 GMT; path=/; domain=www.mooda.com']


[u'825 SR', u'1,575 SR', u'1,181 SR', u'4,377 SR', u'769 SR']
Using Scrapy Shell with FormRequest

Using Scrapy Shell with FormRequest


By : Camilo Slr
Date : March 29 2020, 07:55 AM
Hope that helps You are most likely missing some headers in your FormRequest.
Open up networks tab in your browser tools, find the requests you are looking for and look under "request headers" part (see related issue Can scrapy be used to scrape dynamic content from websites that are using AJAX?). Some of the headers are not necessary and some are already included by FormRequest, however some are not so you need to replicate those.
code :
headers = {
    'Content-Type': 'json/...',
}
req = FormRequest(url, formdata=form, headers=headers)
Scrapy FormRequest login not working

Scrapy FormRequest login not working


By : kahsky
Date : March 29 2020, 07:55 AM
this one helps. Scrapy by default try to populate your email and password in the first clickable input field (in login page it's search form). You need to specify input field by formname or formid e.g. FormRequest.from_response(response, formid="login-form", formdata={"email": "XXXXX", "password": "XXXXX"}, callback=self.start_crawl). See docs
Can't Fill Login Form with FormRequest in Scrapy

Can't Fill Login Form with FormRequest in Scrapy


By : kitty ahuja
Date : March 29 2020, 07:55 AM
around this issue Your Selenium code is almost correct. The problem is that authentication form located inside an iframe and you should switch to it to be able to handle inputs:
code :
driver.switch_to.frame(driver.find_element_by_css_selector('iframe._loadEvent'))
driver.find_element_by_id('loginName').send_keys('g2387744@nwytg.com')
driver.find_element_by_id('password').send_keys('123456789')
driver.find_element_by_css_selector('button[type="submit"]').click()
Scrapy : Can't get hidden data from page with FormRequest

Scrapy : Can't get hidden data from page with FormRequest


By : CableGuy
Date : March 29 2020, 07:55 AM
will be helpful for those in need I have been trying to scrape this page. , They have this in HTML source:
code :
                <li class='link-visible'>                                   <a href='/pages/detail2_cat_est_ordinateurs_puis_rubrique_est_wh_tabmul_puis_ref_est_in10092521.html'>
                                apple ipad mini 4 7.9'' retina 128 go wi-fi gris sideral (2015)                                     &nbsp;<strong>(MK9N2NF/A</strong>)                                  </a>
                        </li>
                    <li class='link-visible'>                                   <a href='/pages/detail2_cat_est_ordinateurs_puis_rubrique_est_wh_tabmul_puis_ref_est_in10103554.html'>
                                Apple iPad 32 Go Wi-Fi Or (2017)                                        &nbsp;<strong>(MPGT2NF/A</strong>)                                  </a>
                        </li>
                    <li>                                    <a href='/pages/detail2_cat_est_ordinateurs_puis_rubrique_est_wh_tabmul_puis_ref_est_in10103538.html'>
                                apple ipad 32 go wi-fi gris sideral (2017)                                      &nbsp;<strong>(MP2F2NF/A</strong>)                                  </a>
                        </li>
                    <li>                                    <a href='/pages/detail2_cat_est_ordinateurs_puis_rubrique_est_wh_tabmul_puis_ref_est_in10103545.html'>
                                Apple iPad 32 Go Wi-Fi Silver (2017)                                        &nbsp;<strong>(MP2G2NF/A</strong>)                                  </a>
                        </li>
Related Posts Related Posts :
  • Autofac - how to register a type used as a constructor parameter that requires the resolving (constructor) type as a gen
  • Tkinter filedialog is stealing focus and not returning it without "Alt-tab" in Python 3.4.1
  • jmeter dashboard report includes transaction controller children
  • Visual Branching in SourceTree
  • Doctrine2 orderBy in Symfony4 UnitTest
  • Match table-record to main-report dataset in Jaspersoft Studio
  • Drupal 8 custom module getting page not found
  • Is it possible that a container instance isn't really a container?
  • Padding not being understood in the UpSampling2D layer of a sequential layer
  • What is the incentive for a participating node in a "Proof of Stake" consensus model of Blockchain?
  • Radial gradient on a node in cytoscape
  • How to create operators from list in Airflow?
  • Is bitly supported for Native Script?
  • Reading log data records from a BLE device
  • How to add aliases in yii2?
  • XPages - Bootstrap popover
  • Misleading exception message in GatewayMethodInboundMessageMapper with un-annotated parameters
  • Inno Setup Disable Next button using multiple validation expressions (when input value matches one of multiple values)
  • 'reference to setCapability is ambiguous' on Appium Java project for Android Caps
  • Quartz .net - Abort/Stop Current Execution of Job & Pause All the triggers
  • Calculate length of road in a Polygon
  • Can Signal have zero recipients in BPMN?
  • Perl6: .sort() doesn't use overridden cmp
  • Find a directory using wildcard in Inno Setup
  • Botframework Dialog migration v3 to v4
  • How to load resources from classpath in itext7?
  • How to inject a java value in XSLT Saxon template?
  • Pagination in Microsoft Graph APi to get users
  • Changing Access modifier using reflection in kotlin
  • Karate - actual value is not a string
  • Is there a way of knowing if data is being piped into a express response?
  • Previous tab components are visible on parsing the json datas - cn1
  • Caret location QtQuick TextField
  • How to run Fabric-Ca server/client on multiple hosts with tls enabled?
  • Porting software / firmware from one architecture to another confusion
  • Why does removing yaml front matter stop Jekyll from converting md file to html
  • Fancybox with Owl carousel (lazyLoad)
  • pouchdb doesn't sync all doc to couchdb
  • Karate API Testing - How do we extract values from Content-Type: text/html kind of response?
  • Show newest inserted values in chart with TimeSeriesCollection when zoomed
  • Different random variable for same HTTP request when load in distributed mode
  • Why doesn't my gremlin query give an output?
  • TestFlight public link doesn't accept new beta testers
  • Is there a simpler way to bind the entire match when using multiple matches?
  • Does apache ignite support as keycloak shared data base?
  • Lotus @-formula - Is it possible to create dynamic selection formula for view?
  • Multiple AND and OR conditons for Filter Expression with DynamoDbMapper
  • ncurses: how to refresh a single panel
  • Why do certain POSIX system calls return an integer instead of a boolean?
  • Correctly apply transformation when moving shapes out of group onto layer
  • Couldn't find executable named camaracalibrator.py [ROS]
  • How do I disable the device back button on Android (react-native)?
  • maximum reclen ifort 18 vs ifort 19
  • How to run static code analysis in netbeans on save
  • What does upload_only=3 mean in extended BitTorrent handshake
  • Navigate ListView with Keyboard without selecting (UWP)
  • Mapping multiple objects in dapper using Split-on and Query Multiple together
  • Number of Performance Monitoring Units in ARM Cortex-A53
  • How to get the bounding box from a Revit Element with Revit API, then call to center of that bounding box
  • Microsoft Graph API to get list of AAD groups owned by a user
  • shadow
    Privacy Policy - Terms - Contact Us © bighow.org