Unstructured content is everywhere

RAG app builders say they spend over 20% of their dev time on data processing and indexing.* They'd rather work on the more creative parts of the GenAI stack, like model building and app design.

Unfortunately, processing and indexing unstructured data is a must. Enterprises that want to power chatbots and internal tools need to use the terabytes of unstructured data stored in different places and formats across their organization.

To build next-gen RAG apps, developers need an easy way to turn their unstructured data into structured, actionable insights that machines can read and learn from.

‍

Introducing Pryon Ingestion SDK

This is why we’re pleased to announce the availability of Pryon Ingestion SDK. This SDK enables developers to ingest unstructured, multimodal content and convert it into knowledge-rich chunks at scale.

Pryon Ingestion Engine excels at converting unstructured content into computer-readable JSON. Pryon Ingestion SDK enables developers to take full advantage of the Ingestion Engine to power their own data ingestion needs. It uses advanced natural language processing (NLP) techniques to extract valuable insights from unstructured documents, wherever they may live and in numerous formats.  The Ingestion SDK uses advanced NLP techniques to extract insights from unstructured documents, in any format and location.

Our differentiation lies in our accuracy, scalability, and built-in enterprise readiness.

Pryon Ingestion Engine is ideal for enterprise RAG, offering:

• Native connectors to a host of common enterprise repositories, including SharePoint, Google Drive, Box, Salesforce, ServiceNow, Zendesk, and Documentum, as well as continuous ingestion
• The ability to read content like a human by cleaning and transforming data through a series of techniques, including component labeling, content normalization, and advanced OCR and handwritten text recognition (HTR)
• Metadata extraction, so nothing gets lost in translation
• Stringent security through read-only access
• High scale with support for millions of documents per collection

Pryon Ingestion SDK demonstrates extreme speed-to-value, allowing you to go from documents to parsed content in minutes. Additionally, Pryon Ingestion SDK is readily scalable for enterprise deployments, including configurability, access controls, metadata, content updating and management, robust infrastructure, and various other platform capabilities.

‍

‍

How to use it

Users typically use Pryon Ingestion SDK in a RAG pipeline. They construct these pipelines by translating text outputs from the Pryon Ingestion SDK to embeddings then storing these embeddings in a vector database. The RAG pipelines then pair the content in the vector database with downstream language models, as shown in the diagram below. This may include users taking advantage of some (e.g., certain document elements) or all fields in the parse content output.

To get started, developers can download the SDK and begin making function calls. The SDK is based on two primary function calls:

• Parse content: returns a JSON from an input document
• Parse to collection: returns a JSON from an input document and persists the ingested content to a collection for continuing to retrieve the JSON output
  ‍

‍

With this setup, you can easily extract insights from unstructured data and use them to power your AI applications. You can also use Pryon Ingestion SDK to preprocess text data for downstream NLP tasks such as sentiment analysis or named entity recognition.

‍

Conclusion

Pryon Ingestion SDK helps customers unlock the value of unstructured data for AI application development. With our advanced technology and built-in enterprise readiness, you can transform your messy and unorganized information into valuable insights that machines can understand and learn from.

Try Pryon Ingestion SDK today and see how it can transform the way you process and analyze data to power your AI applications.

* Pryon-conducted survey of RAG developers, June 2024

‍