How to search multiple files for textblocks and write those textblocks to another file

Here is an example of an input file:

<html xml:lang="en" lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
</head>
<body>
    HERE IS A LOT OF TEXT, THAT IS NOT INTERESTING 
    <br>
      <div id="text"><div id="text-interesting1">11/222-AA</div>
      <h2>This is the title</h2>

            <P>Here is some multiline desc-<br>
                 cription about what is <br><br>
                 going on here
      </div>

      <div id="text2"><div id="text-interesting2">IV-VI</div>
        <br>
        <h1> Some really interesting text</h1>
</body>
</html>

Now I want to grep multiple blocks of this file, like that between <div id="text-interesting1"> and </div> then between <P> and </div> then between <div id="text-interesting2"> and </div> and many more. The point is, there are multiple values that I want to retrieve.

I want to write those values to a file, e.g. comma separated. How can that be done?

From the example that Luke provided I made the following:

import os, re
path = 'C:/Temp/Folder1/allTexts'
listing = os.listdir(path)
for infile in listing:
    text = open(path + '/' + infile).read()
    match = re.search('<div id="text-interesting1">', text)
    if match is None:
        continue
    start = match.end()
    end = re.search('</div>', text).start()
    print (text[start:end])


    match = re.search('<h2>', text)
    if match is None:
        continue
    start = match.end()
    end = re.search('</h2>', text).start()
    print (text[start:end])


    match = re.search('<P>', text)
    if match is None:
        continue
    start = match.end()
    end = re.search('</div>', text).start()
    print (text[start:end])


    match = re.search('<div id="text-interesting2">', text)
    if match is None:
        continue
    start = match.end()
    end = re.search('</div>', text).start()
    print (text[start:end])


    match = re.search('<h1>', text)
    if match is None:
        continue
    start = match.end()
    end = re.search('</h1>', text).start()
    print (text[start:end])

    print ('--------------------------------------')

Output is:

11/222-AA
This is the title


 Some really interesting text
--------------------------------------
22/4444-AA
22222 This is the title2


22222222222222222222222
--------------------------------------
33/4444-AA
3333 This is the title3


333333333333333333333333
--------------------------------------

Why does the

part not work?

Answers


Here's a start:

import os, re
path = 'C:/Temp/Folder1/allTexts'
listing = os.listdir(path)
for infile in listing:
    text = open(path + '/' + infile).read()
    match = re.search('<div id="text-interesting1">', text)
    if match is None:
        continue
    start = match.start()
    end = re.search('<div id="text-interesting2">', text).start()
    print text[start:end]

Another strategy is to parse the XML. You will need to tidy your file up since strict XML requires matching tags, case consistency, etc. Here is an example:

from xml.etree import ElementTree
from cStringIO import StringIO
import sys
tree = ElementTree.ElementTree()
tree.parse(StringIO(sys.stdin.read()))
print "All tags:"
for e in tree.getiterator():
    print e.tag
    print e.text
print "Only div:"
for i in tree.find("{http://www.w3.org/1999/xhtml}body").findall("{http://www.w3.org/1999/xhtml}div"):
    print i.text

Run on a slight modification of your file:

<html xml:lang="en" lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
</head>
<body>
HERE IS A LOT OF TEXT, THAT IS NOT INTERESTING 
<br></br>
<div id="text"><div id="text-interesting1">11/222-AA</div>
  <h2>This is the title</h2>
  <p>Here is some multiline desc-<br></br>
      cription about what is <br></br><br></br>
    going on here</p>
</div>
  <div id="text-interesting2">IV-VI</div>
      <br></br>
    <h1> Some really interesting text</h1>
</body>
</html>

Example output,

> cat file.xml | ./tb.py 
All tags:
{http://www.w3.org/1999/xhtml}html


{http://www.w3.org/1999/xhtml}head


{http://www.w3.org/1999/xhtml}body

HERE IS A LOT OF TEXT, THAT IS NOT INTERESTING 

{http://www.w3.org/1999/xhtml}br
None
{http://www.w3.org/1999/xhtml}div
None
{http://www.w3.org/1999/xhtml}div
11/222-AA
{http://www.w3.org/1999/xhtml}h2
This is the title
{http://www.w3.org/1999/xhtml}p
Here is some multiline desc-
{http://www.w3.org/1999/xhtml}br
None
{http://www.w3.org/1999/xhtml}br
None
{http://www.w3.org/1999/xhtml}br
None
{http://www.w3.org/1999/xhtml}div
IV-VI
{http://www.w3.org/1999/xhtml}br
None
{http://www.w3.org/1999/xhtml}h1
Some really interesting text
Only div:
None
IV-VI

But a lot of HTML is difficult to parse as strict XML, so this may prove hard to implement for your case.


Need Your Help

Linearize non-linear model considering a partial set of states in Matlab\Simulink

matlab controls simulink states linearization

I've a non-linear model with 9 states. For the linearization procedure, it considers 8 states because one state is not visible from a point of view of linearization, in base of assumptions .

Library agnostic way to mock/fake http requests in Clojure?

http testing clojure mocking

I'm looking for a project similar to mockwebserver in Clojure that allows you to programmatically define HTTP endpoints and their response by matching (parts of) the of the HTTP request (for testing