#953 Python Code to Find "shall" and "will" for Specified Topic in a Process/Requirements Doc

#953 Python Code to Find "shall" and "will" for Specified Topic in a Process/Requirements Doc

#953 I have a document with process and/or technical requirements strewn throughout the document. I want to write Python code to parse the document and locate all sentences and/or paragraphs that contain requirements for a specified topic. Requirements statements are typically identified with a "shall" or possibly a "will". Provide the python code to search a document for a specified topic and returns all matches showing the requirements text in-context (i.e. full sentence or applicable paragraph).

Be Serious:

Certainly! To achieve the task you're describing, you can create a Python script that reads the document and uses regular expressions to locate sentences or paragraphs containing the specified keywords and requirement identifiers like "shall" or "will."

Here's a simple code snippet to help you get started. This code will read a document (e.g., a .txt file), search for the specified topic, and return sentences containing the words "shall" or "will."

import re

def find_requirements(file_path, topic):

    requirements = []

    with open(file_path, 'r') as file:

        text = file.read()

        

        # Splitting the document into sentences using a simple regex pattern

        sentences = re.split(r' *[\.\?!][\'"\)\]]* *', text)

        # Searching for sentences containing the specified topic and "shall" or "will"

        for sentence in sentences:

            if topic.lower() in sentence.lower() and ('shall' in sentence or 'will' in sentence):

                requirements.append(sentence)

    return requirements

file_path = 'path/to/your/document.txt'

topic = 'desired_topic'

requirements = find_requirements(file_path, topic)

for requirement in requirements:

    print(requirement)

Make sure to replace 'path/to/your/document.txt' with the actual path to the document you want to parse, and 'desired_topic' with the topic you're interested in.

This code assumes that the document is plain text. If you're working with another file format like a Word document or a PDF, you'll need to use an appropriate library to read the content, such as python-docx for Word documents or PyPDF2 for PDFs.

Additionally, please note that the sentence splitting here is quite simplistic and might not handle all cases perfectly. For a more robust solution, you might consider using a natural language processing library like nltk to tokenize the text into sentences.

So the workflow would be: PDF to Text, NLTK to tokenize the text into sentences. Then the provided search code to return all occurrences of topic shalls and/or wills, in-context. Would NLTK be able to identify paragraph boundaries and bullet lists?

Be Serious:

Yes, your described workflow is correct: converting the PDF to text, using NLTK to tokenize the text into sentences, and then using the search code to find the relevant requirements related to the specified topic.

NLTK provides tokenization utilities to split text into sentences and words, but it doesn't directly offer functionality to identify paragraph boundaries or bullet lists. Paragraphs and lists are often format-specific and might require custom logic to identify, depending on how they are represented in the text.

However, you can create some logic to identify paragraph boundaries based on the text format. For example, you could use empty lines or specific indentation as indicators of paragraph boundaries.

Here's an example of how you might modify the code to handle paragraphs, assuming paragraphs are separated by empty lines:

from nltk.tokenize import sent_tokenize

import re

def find_requirements(file_path, topic):

    requirements = []

    with open(file_path, 'r') as file:

        text = file.read()

        # Splitting the document into paragraphs using empty lines

        paragraphs = text.split('\n\n')

        for paragraph in paragraphs:

            # Tokenizing the paragraph into sentences

            sentences = sent_tokenize(paragraph)

            # Searching for sentences containing the specified topic and "shall" or "will"

            for sentence in sentences:

                if topic.lower() in sentence.lower() and ('shall' in sentence or 'will' in sentence):

                    requirements.append(sentence)

    return requirements

# ...

# Rest of the code

For bullet lists, the handling would depend on how they are formatted in the text. If bullet lists are consistently formatted with a specific character (e.g., - or *), you could detect them using a regular expression pattern and then process them as needed.

Keep in mind that the accuracy of these techniques may vary depending on the specific formatting of the document you're working with. It may require some tuning and experimentation to achieve the desired results.

From <https://chat.openai.com/?model=gpt-4-code-interpreter>