The following section will cover basic concepts and usage patterns in
After reading this section, you should be able to:
Partitioning a document with the
Understand how documents are structured in
Convert a document to a dictionary and/or save it as a JSON.
The example documents in this section come from the
directory in the
Before running the code in this make sure you’ve installed the
and all dependencies using the instructions in the Quick Start section.
Partitioning a document
In this section, we’ll cut right to the chase and get to the most important part of the library: partitioning a document.
The goal of document partitioning is to read in a source document, split the document into sections, categorize those sections,
and extract the text associated with those sections. Depending on the document type, unstructured uses different methods for
partitioning a document. We’ll cover those in a later section. For now, we’ll use the simplest API in the library,
partition function. The
partition function will detect the filetype of the source document and route it to the appropriate
partitioning function. You can try out the partition function by running the cell below.
from unstructured.partition.auto import partition elements = partition(filename="example-10k.html")
You can also pass in a file as a file-like object using the following workflow:
with open("example-10k.html", "rb") as f: elements = partition(file=f)
partition function uses libmagic for filetype detection. If
not present and the user passes a filename,
partition falls back to detecting the filetype using the file extension.
libmagic is required if you’d like to pass a file-like object to
We highly recommend installing
libmagic and you may observe different file detection behaviors
libmagic is not installed`.
When we partition a document, the output is a list of document
These element objects represent different components of the source document. Currently, the
unstructured library supports the following element types:
Other element types that we will add in the future include tables and figures.
Different partitioning functions use different methods for determining the element type and extracting the associated content.
Document elements have a
str representation. You can print them using the snippet below.
elements = partition(filename="example-10k.html") for element in elements[:5]: print(element) print("\n")
One helpful aspect of document elements is that they allow you to cut a document down to the elements that you need for your particular use case. For example, if you’re training a summarization model you may only want to include narrative text for model training. You’ll notice that the output above includes a lot of titles and other content that may not be suitable for a summarization model. The following code shows how you can limit your output to only narrative text with at least two sentences. As you can see, the output now only contains narrative text.
from unstructured.documents.elements import NarrativeText from unstructured.partition.text_type import sentence_count for element in elements[:100]: if isinstance(element, NarrativeText) and sentence_count(element.text) > 2: print(element) print("\n")
unstructured library includes helper functions for
reading and writing a list of
Element objects to and
from JSON. You can use the following workflow for
serializing and deserializing an
from unstructured.documents.elements import ElementMetadata, Text, Title, FigureCaption from unstructured.staging.base import elements_to_json, elements_from_json filename = "my-elements.json" metadata = ElementMetadata(filename="fake-file.txt") elements = [ FigureCaption(text="caption", metadata=metadata, element_id="1"), Title(text="title", metadata=metadata, element_id="2"), Text(text="title", metadata=metadata, element_id="3"), ] elements_to_json(elements, filename=filename) new_elements = elements_from_json(filename=filename) # alternatively, one can also serialize/deserialize to/from a string with: serialized_elements_json = elements_to_json(elements) new_elements = elements_from_json(text=serialized_elements_json)
Converting elements to a dictionary or JSON
The final step in the process for most users is to convert the output to JSON.
You can convert a list of document elements to a list of dictionaries using the
The workflow for using
convert_to_dict appears below.
from unstructured.staging.base import convert_to_dict convert_to_dict(elements)
unstructured library also includes utilities for saving a list of elements to JSON and reading
a list of elements from JSON, as seen in the snippet below
from unstructured.staging.base import elements_to_json, elements_from_json filename = "outputs.json" elements_to_json(elements, filename=filename) elements = elements_from_json(filename=filename)
Wrapping it all up
To conclude, the basic workflow for reading in a document and converting it to a JSON in
looks like the following:
from unstructured.partition.auto import partition from unstructured.staging.base import elements_to_json input_filename = "example-10k.html" output_filename = "outputs.json" elements = partition(filename=input_filename) elements_to_json(elements, filename=output_filename)