So, when a website does not provide an API, the only way to get the data from the website is to scrape it off yourself.Web Scraper Cloud can be managed via an HTTPS JSON API. In most cases, you are provided with an API, but that’s not always plausible. There are times when we want to extract data from a website.
I want to scrape all of the pages. The site basically shows articles like a shopping site would: ten items per page, each article is an element that consists of title, a short description and so on.The whole list consists of about 80-90 articles, spread over 8-9 pages. Use our PHP SDK when developing.One thing i don't get is how pagination works.I'm scraping a PHP web page with research updates.
Even if i scrape 20 seconds apart (and the site hasn't changed) the results are different.Does anyone know what's going on? I have no idea myself, probably because i don't understand how pagination works. As noted above, the program goes through the pages twice, but some of the articles are listed three times in my scraped list. So it visits each page twice, and saves the info from each article twice (at least)2) The list of data gets a different number of lines every time. But i bump into the following problems:1) Web scraper goes through all of the pages and then goes back.
There are infinite CSS selector tutorials if you find the specification I linked to is too low-levelYou will gain a much better grasp if you open the page source, and/or use the Chrome developer tools to inspect the elements on the page - you will without any question whatsoever need to be familiar with both of those things for any success. Fa-angle-right is a CSS class selector, and corresponds to class="fa-angle-right" found on the tag in the page source. The page is in php, and the url for each of the pages looks like this: ".php?start=10" (or 20, or 30 and so on.)For context, I'll be discussing page 11 and page 12, the next to the last one, has the same "next" markup as do all of its friends: Which is ul.pagination li a i.fa-angle-right (you can omit the li if you wish, or you can make it super specific by mandating they be in that exact structure with ul.pagination > li > a > i.fa-angle-right - that can be good or bad, depending on the circumstance)Thankfully, page 12 (the last one) does what I thought it would: the "next" button disappears: there is no such selector i.fa-angle-right and instead the ul.pagination li.active:last-of-type a is the last li there is: 12And that selector will only ever match if you have run out of "next" linksI've given the CSS selectors but every one of them has a corresponding XPath selector, so you can use whichever makes the most sense to you and/or is supported by the scraping tool you're usingSee how much easier it is with actual HTML and links to discuss? :-)It's helpful if you use the backticks to format code snippets, so they are easier to distinguish from english: `code snippet` is the syntaxSee "i.fa-angle-right" (selector, right? Or element?)It's a selector, not an element is the element, and corresponds to the i in the selector. I just don't get what I'm doing. But how does it know which one to open? I mean, on the starting page there is a 1, a 2 and a right arrow, but when you are on page 2, it has a left arrow, a 1, a 3, and a right arrow.* The selector says "ul.pagination a" as in the tutorial, but I've also tried stuff like "ul.pagination li:nth-of-type(2)" and other similar lines.
If you went to wikipedia.org and tried to use that selector, nothing would happen. Also, saying "it doesn't work" is meaningless without saying what did happen, and under what circumstances you tried to have that take place. Chrome's javascript console has a handy built-in shortcut $$('i.fa-angle-right') to try out selectors, but that $$ is just an alias for document.querySelectorAll, so don't think of it as "chrome magic" (although be aware $$ is only in the chrome console, and not in the webpage itself)Unfortunately, it doesn't find anything at all when i scrapeThen the tool you are using is clearly defective, because I didn't pull those out of thin-air.
0 kommentar(er)
