Integrate your model development pipeline with your favorite machine learning frameworks and libraries,
and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of
which take a list of
Element objects as input and return formatted dictionaries as output.
Integration with Argilla
You can convert a list of
Text elements to an Argilla
Dataset using the stage_for_argilla staging brick. Specify the type of dataset to be generated using the
argilla_task parameter. Valid values are
"text2text". Follow the link for more details on usage.
Integration with Baseplate
Baseplate is a backend optimized for use with LLMs that has an easy to use spreadsheet
unstructured library offers a staging brick to convert a list of
Element objects into the
rows format required by the Baseplate API. See the
stage_for_baseplate documentation for
information on how to stage elements for ingestion into Baseplate.
Integration with Datasaur
You can format a list of
Text elements as input to token based tasks in Datasaur using the stage_for_datasaur staging brick. You will obtain a list of dictionaries indexed by the keys
"text" with the content of the element, and
"entities" with an empty list. Follow the link to learn how to customise your entities and for more details on usage.
Integration with Hugging Face
You can prepare
Text elements for processing in Hugging Face Transformers
pipelines by splitting the elements into chunks that fit into the model’s attention window using the stage_for_transformers staging brick. You can customise the transformation by defining
split_function and the
chunk_separator. if you need to operate on
text directly instead of
Text objects, use the chunk_by_attention_window helper function. Follow the links for more details on usage.
Integration with Labelbox
You can format your outputs for use with LabelBox using the stage_for_label_box staging brick. LabelBox accepts cloud-hosted data and does not support importing text directly. With this integration you can stage the data files in the
output_directory to be uploaded to a cloud storage service (such as S3 buckets) and get a config of type
List[Dict[str, Any]] that can be written to a
.json file and imported into LabelBox. Follow the link to see how to generate the
config.json file that can be used with LabelBox, how to upload the staged data files to an S3 bucket, and for more details on usage.
Integration with Label Studio
You can format your outputs for upload to Label Studio using the stage_for_label_studio staging brick. After running
stage_for_label_studio, you can write the results
to a JSON folder that is ready to be included in a new Label Studio project. You can also include pre-annotations and predictions
as part of your upload.
Check our example notebook to format and upload the risk section from an SEC filing to Label Studio for a sentiment analysis labeling task here . Follow the link for more details on usage, and check Label Studio docs for a full list of options for labels and annotations.
Integration with LangChain
Our integration with LangChain makes it incredibly easy to combine language models with your data, no matter what form it is in. The Unstructured.io File Loader extracts the text from a variety of unstructured text files using our
unstructured library. It is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for more LlamaHub examples.
Unstructured.io File Loader you will need to have LlamaIndex 🦙 (GPT Index) installed in your environment. Just
pip install llama-index and then pass in a
Path to a local file. Optionally, you may specify split_documents if you want each element generated by
unstructured to be placed in a separate document. Here is a simple example on how to use it:
from pathlib import Path from llama_index import download_loader UnstructuredReader = download_loader("UnstructuredReader") loader = UnstructuredReader() documents = loader.load_data(file=Path('./10k_filing.html'))
Integration with Pandas
You can convert a list of
Element objects to a Pandas dataframe with columns for
the text from each element and their types such as
Title using the convert_to_dataframe staging brick. Follow the link for more details on usage.
Integration with Prodigy
You can format your JSON or CSV outputs for use with Prodigy using the stage_for_prodigy and stage_csv_for_prodigy staging bricks. After running
stage_csv_for_prodigy, you can write the results to a
.jsonl or a
.csv file that is ready to be used with Prodigy. Follow the links for more details on usage.
Integration with Weaviate
Weaviate is an open-source vector database that allows you to store data objects and vector embeddings
from a variety of ML models. Storing text and embeddings in a vector database such as Weaviate is a key component of the
emerging LLM tech stack.
See the stage_for_weaviate docs for details
on how to upload
unstructured outputs to Weaviate. An example notebook is also available