BeautifulSoup get innerhtml data

I am trying to read data from a website. I can see the value I need but the value does not appear in the downloaded html code (using urllib2). The value is created by some js file and embedded into the webpage as innerhtml for that id. PS: How can that be extracted? raw source code cannot render js unlike the browsers!

Answers


Another way of getting data is leaving the browser do all the stuff using Selenium and read the rendered html. A bit slow but surely effective.

Here you can find a getting started guide for using Selenium with Python: http://jimmyg.org/blog/2009/getting-started-with-selenium-and-python.html


You have two options: Have the browser save the DOM (this includes all changes made by scripts) or use a JavaScript engine to execute the embedded scripts.

For the latter route, try a Java based engine like Rhino and emulate the browser with env.js.


Need Your Help

python/pygame image cant be found the the same folder

python pygame

hello im new to python/pygame and tried to make a basic project. it did not turn out as planned as i don't understand this error if you can tell my why my image is not loading it will very much