Integrations
Integrate your model development pipeline with your favorite machine learning frameworks and libraries,
and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of
staging bricks,
which take a list of Element
objects as input and return formatted dictionaries as output.
Integration with Argilla
You can convert a list of Text
elements to an Argilla Dataset
using the stage_for_argilla staging brick. Specify the type of dataset to be generated using the argilla_task
parameter. Valid values are "text_classification"
, "token_classification"
, and "text2text"
. Follow the link for more details on usage.
Integration with Baseplate
Baseplate is a backend optimized for use with LLMs that has an easy to use spreadsheet
interface. The unstructured
library offers a staging brick to convert a list of Element
objects into the
rows format required by the Baseplate API. See the
stage_for_baseplate documentation for
information on how to stage elements for ingestion into Baseplate.
Integration with Datasaur
You can format a list of Text
elements as input to token based tasks in Datasaur using the stage_for_datasaur staging brick. You will obtain a list of dictionaries indexed by the keys "text"
with the content of the element, and "entities"
with an empty list. Follow the link to learn how to customise your entities and for more details on usage.
Integration with Hugging Face
You can prepare Text
elements for processing in Hugging Face Transformers
pipelines by splitting the elements into chunks that fit into the model’s attention window using the stage_for_transformers staging brick. You can customise the transformation by defining
the buffer
and window_size
, the split_function
and the chunk_separator
. if you need to operate on
text directly instead of unstructured
Text
objects, use the chunk_by_attention_window helper function. Follow the links for more details on usage.
Integration with Labelbox
You can format your outputs for use with LabelBox using the stage_for_label_box staging brick. LabelBox accepts cloud-hosted data and does not support importing text directly. With this integration you can stage the data files in the output_directory
to be uploaded to a cloud storage service (such as S3 buckets) and get a config of type List[Dict[str, Any]]
that can be written to a .json
file and imported into LabelBox. Follow the link to see how to generate the config.json
file that can be used with LabelBox, how to upload the staged data files to an S3 bucket, and for more details on usage.
Integration with Label Studio
You can format your outputs for upload to Label Studio using the stage_for_label_studio staging brick. After running stage_for_label_studio
, you can write the results
to a JSON folder that is ready to be included in a new Label Studio project. You can also include pre-annotations and predictions
as part of your upload.
Check our example notebook to format and upload the risk section from an SEC filing to Label Studio for a sentiment analysis labeling task here . Follow the link for more details on usage, and check Label Studio docs for a full list of options for labels and annotations.
Integration with LangChain
Our integration with LangChain makes it incredibly easy to combine language models with your data, no matter what form it is in. The Unstructured.io File Loader extracts the text from a variety of unstructured text files using our unstructured
library. It is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for more LlamaHub examples.
To use Unstructured.io File Loader
you will need to have LlamaIndex 🦙 (GPT Index) installed in your environment. Just pip install llama-index
and then pass in a Path
to a local file. Optionally, you may specify split_documents if you want each element generated by unstructured
to be placed in a separate document. Here is a simple example on how to use it:
from pathlib import Path
from llama_index import download_loader
UnstructuredReader = download_loader("UnstructuredReader")
loader = UnstructuredReader()
documents = loader.load_data(file=Path('./10k_filing.html'))
Integration with Pandas
You can convert a list of Element
objects to a Pandas dataframe with columns for
the text from each element and their types such as NarrativeText
or Title
using the convert_to_dataframe staging brick. Follow the link for more details on usage.
Integration with Prodigy
You can format your JSON or CSV outputs for use with Prodigy using the stage_for_prodigy and stage_csv_for_prodigy staging bricks. After running stage_for_prodigy
|
stage_csv_for_prodigy
, you can write the results to a .json
| .jsonl
or a .csv
file that is ready to be used with Prodigy. Follow the links for more details on usage.
Integration with Weaviate
Weaviate is an open-source vector database that allows you to store data objects and vector embeddings
from a variety of ML models. Storing text and embeddings in a vector database such as Weaviate is a key component of the
emerging LLM tech stack.
See the stage_for_weaviate docs for details
on how to upload unstructured
outputs to Weaviate. An example notebook is also available
here.