how to use beautifulsoup to fetch data in slim scroll window

Recently I came across a problem of trying to fetch data from slim scroll window in a website. I can see all the tags and their class/id/type in Chrome Inspect element with not problem.

The webpage is https://chart.cryptocoinsnews.com/bitfinex_frame.html The table I am looking at is

From soup, I can still visit the parent {div class="panel-body table-pannel-body"} but not this tag {div class = "slimScrollDiv" style=........} which contains the table I am interested.

It is confusing to me why I can see them on Chrome but would not be able to fetch by soup. Appreciate if anyone could help. Thanks a lot!

Answers


Due to it's dynamic nature the table is being created/updated from javascript.

You are likely only downloading the HTML before the DOM is constructred dynamically with javascript. So you'll need to use a browser, e.g. with selenium, to see the same DOM that you can have in chrome.

Note that selenium can use headless browsers, like PhantomJS.


Need Your Help

What is the scope of a Static Class?

c# static biztalk applicationdomain

I have an assembly that may be used by more than one process at a time. If I am using a static class, would the multiple processes all use the same "instance" of that class?

C# System.InvalidOperationException: Collection was modified; enumeration operation may not execute

c# invalidoperationexception

I have researched this issue and haven't found what pertains to me yet. I am not trying to edit something I am looping through.