[NScrape Web Scraping] Scraper is not waiting for dynamic JS to populate DOM
Question
Forge component by André Vieira

I have a website that will load some pertinent information after initial DOM render. When I get the HTML using:

ScrapePage -> HTMLDocument_Load

and I drill down to the larger DIV I want to extract, I can see child DIVs with missing inner HTML content. I can see other static content in the HTML that shows on the actual website but even on the website you can clearly tell it is fetching some dynamic data after load.

Any ideas into adding a "wait for full render" in the extension to mitigate this?

Hi Nicholas, were you able to achieve this? I have the same issue, as posted here. Was hoping that the NScrape project page had already the solution to this, though... perhaps on the library documentation or its discussion forums... but couldn't find it yet.

Unfortunately not, but there may be some options for implementing an implicit or explicit wait function like in the Selenium WebDriver. Maybe that is an enhancement opportunity.

Regardless, I plan to explore some RPA options and use OutSystems to drive that bot. That way I can utilize runtime actions to generate the data I need from AJAX and then scrape it into OutSystems.

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.