0 votes
12 views

Really need the help from this community.

My question is that when I used the code

========================================================================= response.xpath("//div[contains(@class,'check-prices-widget-not-sponsored')]/a/div[contains(@class,'check-prices-widget-not-sponsored-link')]").extract() enter image description here to extract the vendor name in scrapy shell, the output is empty. I really did not know why that happened, and it seems to me that the problem might be the website info is updating dynamically?

The url for this web scraping is: https://cruiseline.com/cruise/7-night-bahamas-florida-new-york-roundtrip-32860, and what I need is the Vendor name and Price for each vendor. Besides the attached pic is the screenshot of "the inspect".

Really appreciate the help!

by (305 points)   | 12 views

1 Answer

0 votes

You need to always check HTML source code in your browser (usually with Ctrl+U).

This way you'll find that information you want is embedded inside Javascript variables using JSON:

var partnerPrices = [{"pool":"9a316391b6550eef969c8559c14a380f","partner":"ncl.com","priority":0,"currency":"USD","data":{"32860":{"2018-02-25":{"Inside":579,"Suite":1199,"Balcony":699,"Oceanview":629},.... var sponsored_partners = [{"code":"CDCNA","name":"cruises.com","value":"cruises.com","logo":"\/images\/partner-logo-cruises-sm.png","logo_sprite":"partner-logo-cruises-com"},...

So you need to import json, parse response.body (using re or another method) and next json.loads() parsed JSON strings to iterate through two arrays.

by (305 points)  

Related questions

+1 vote
1 answer
asked May 26, 2020 by toheed (1.4k points) | 20 views
+2 votes
0 answers
asked Jun 17, 2020 by abrishnoor (1.7k points) | 166 views
+1 vote
0 answers
asked Jun 16, 2020 by noor (1.1k points) | 24 views
+1 vote
0 answers
asked Apr 28, 2020 by athar (3.3k points) | 55 views
0 votes
1 answer
+2 votes
2 answers
+1 vote
0 answers
0 votes
1 answer
asked Jul 2, 2020 by toheed (1.4k points) | 67 views




Welcome to Meansflow - Where Developers and Students Learn, Share, & Build Careers, where you can ask questions and receive answers from other members of the community.
131 questions
94 answers
44 comments
134 users