Example gallery¶

Explore different Ordeq example pipelines.

Integration Docker

This example demonstrates how to integrate Ordeq with Docker to run pipelines in isolated container environments.

graph TB

    integration_docker.pipeline:hello_world@{shape: subroutine, label: "hello_world"}

    class integration_docker.pipeline:hello_world view
    classDef view fill:#00C853,color:#FFF

View example

Integration Marimo

Marimo is an open-source reactive Python notebook. An improved version of Jupyter notebooks, Marimo allows you to build data applications with interactive widgets and reactive cells.

View example

Integration Streamlit

To run the Streamlit application:

graph TB
    integration_streamlit.catalog:checkbox --> integration_streamlit.pipeline:display_values
    integration_streamlit.catalog:slider --> integration_streamlit.pipeline:display_values

    integration_streamlit.pipeline:display_values@{shape: subroutine, label: "display_values"}
    integration_streamlit.catalog:checkbox@{shape: rect, label: "checkbox"}
    integration_streamlit.catalog:slider@{shape: rect, label: "slider"}

    class integration_streamlit.pipeline:display_values view
    class integration_streamlit.catalog:checkbox,integration_streamlit.catalog:slider io0
    classDef io fill:#FFD43B
    classDef view fill:#00C853,color:#FFF
    classDef io0 fill:#66c2a5

View example

Orchestration Kubeflow

This example demonstrates how to integrate Ordeq with Kubeflow Pipelines for orchestrating machine learning workflows in Kubernetes clusters.

graph TB
    ml_pipeline.catalog:csv_remote --> ml_pipeline.preprocessing:create_dataset
    ml_pipeline.preprocessing:create_dataset --> ml_pipeline.catalog:csv_local
    ml_pipeline.catalog:csv_local --> ml_pipeline.preprocessing:normalize_dataset
    ml_pipeline.catalog:scaler --> ml_pipeline.preprocessing:normalize_dataset
    ml_pipeline.preprocessing:normalize_dataset --> ml_pipeline.catalog:csv_normalized
    ml_pipeline.catalog:csv_normalized --> ml_pipeline.train:train_model
    ml_pipeline.catalog:n_neighbors --> ml_pipeline.train:train_model
    ml_pipeline.catalog:seed --> ml_pipeline.train:train_model
    ml_pipeline.train:train_model --> ml_pipeline.catalog:knn_model

    ml_pipeline.preprocessing:create_dataset@{shape: rounded, label: "create_dataset"}
    ml_pipeline.preprocessing:normalize_dataset@{shape: rounded, label: "normalize_dataset"}
    ml_pipeline.catalog:csv_local@{shape: rect, label: "csv_local"}
    ml_pipeline.train:train_model@{shape: rounded, label: "train_model"}
    ml_pipeline.catalog:csv_normalized@{shape: rect, label: "csv_normalized"}
    ml_pipeline.catalog:csv_remote@{shape: rect, label: "csv_remote"}
    ml_pipeline.catalog:knn_model@{shape: rect, label: "knn_model"}
    ml_pipeline.catalog:n_neighbors@{shape: rect, label: "n_neighbors"}
    ml_pipeline.catalog:scaler@{shape: rect, label: "scaler"}
    ml_pipeline.catalog:seed@{shape: rect, label: "seed"}

    class ml_pipeline.preprocessing:create_dataset,ml_pipeline.preprocessing:normalize_dataset,ml_pipeline.train:train_model node
    class ml_pipeline.catalog:csv_local,ml_pipeline.catalog:csv_normalized,ml_pipeline.catalog:csv_remote io1
    class ml_pipeline.catalog:knn_model io2
    class ml_pipeline.catalog:n_neighbors,ml_pipeline.catalog:scaler,ml_pipeline.catalog:seed io0
    classDef node fill:#008AD7,color:#FFF
    classDef io fill:#FFD43B
    classDef io0 fill:#66c2a5
    classDef io1 fill:#fc8d62
    classDef io2 fill:#8da0cb

View example

Rag Pipeline Scaffold

This example demonstrates how to create a Retrieval-Augmented Generation (RAG) pipeline scaffold using Ordeq.

graph TB
    rag_pipeline.catalog:pdf_documents --> rag_pipeline.rag.indexer:create_vector_index
    rag_pipeline.catalog:llm_vision_retrieval_model --> rag_pipeline.rag.indexer:create_vector_index
    rag_pipeline.rag.indexer:create_vector_index --> rag_pipeline.catalog:index
    rag_pipeline.catalog:policies --> rag_pipeline.rag.policies:generate_questions
    rag_pipeline.rag.policies:generate_questions --> rag_pipeline.catalog:questions
    rag_pipeline.catalog:index --> rag_pipeline.rag.retrieval:retrieve
    rag_pipeline.catalog:questions --> rag_pipeline.rag.retrieval:retrieve
    rag_pipeline.catalog:llm_vision_retrieval_model --> rag_pipeline.rag.retrieval:retrieve
    rag_pipeline.rag.retrieval:retrieve --> rag_pipeline.catalog:retrieved_pages
    rag_pipeline.catalog:retrieved_pages --> rag_pipeline.rag.retrieval:filter_relevant
    rag_pipeline.catalog:llm_model --> rag_pipeline.rag.retrieval:filter_relevant
    rag_pipeline.rag.retrieval:filter_relevant --> rag_pipeline.catalog:relevant_pages
    rag_pipeline.catalog:questions --> rag_pipeline.rag.question_answering:question_answering
    rag_pipeline.catalog:relevant_pages --> rag_pipeline.rag.question_answering:question_answering
    rag_pipeline.catalog:llm_model --> rag_pipeline.rag.question_answering:question_answering
    rag_pipeline.rag.question_answering:question_answering --> rag_pipeline.catalog:llm_answers
    rag_pipeline.catalog:llm_answers --> rag_pipeline.rag.evaluation:evaluate_answers
    rag_pipeline.catalog:llm_model --> rag_pipeline.rag.evaluation:evaluate_answers
    rag_pipeline.rag.evaluation:evaluate_answers --> rag_pipeline.catalog:metrics
    rag_pipeline.catalog:llm_answers --> rag_pipeline.rag.annotation:annotate_documents
    rag_pipeline.catalog:pdf_documents --> rag_pipeline.rag.annotation:annotate_documents
    rag_pipeline.rag.annotation:annotate_documents --> rag_pipeline.catalog:pdfs_documents_annotated

    rag_pipeline.rag.indexer:create_vector_index@{shape: rounded, label: "create_vector_index"}
    rag_pipeline.rag.policies:generate_questions@{shape: rounded, label: "generate_questions"}
    rag_pipeline.rag.retrieval:retrieve@{shape: rounded, label: "retrieve"}
    rag_pipeline.rag.retrieval:filter_relevant@{shape: rounded, label: "filter_relevant"}
    rag_pipeline.catalog:retrieved_pages@{shape: rect, label: "retrieved_pages"}
    rag_pipeline.rag.question_answering:question_answering@{shape: rounded, label: "question_answering"}
    rag_pipeline.rag.evaluation:evaluate_answers@{shape: rounded, label: "evaluate_answers"}
    rag_pipeline.rag.annotation:annotate_documents@{shape: rounded, label: "annotate_documents"}
    rag_pipeline.catalog:index@{shape: rect, label: "index"}
    rag_pipeline.catalog:llm_answers@{shape: rect, label: "llm_answers"}
    rag_pipeline.catalog:llm_model@{shape: rect, label: "llm_model"}
    rag_pipeline.catalog:llm_vision_retrieval_model@{shape: rect, label: "llm_vision_retrieval_model"}
    rag_pipeline.catalog:metrics@{shape: rect, label: "metrics"}
    rag_pipeline.catalog:pdf_documents@{shape: rect, label: "pdf_documents"}
    rag_pipeline.catalog:pdfs_documents_annotated@{shape: rect, label: "pdfs_documents_annotated"}
    rag_pipeline.catalog:policies@{shape: rect, label: "policies"}
    rag_pipeline.catalog:questions@{shape: rect, label: "questions"}
    rag_pipeline.catalog:relevant_pages@{shape: rect, label: "relevant_pages"}

    class rag_pipeline.rag.indexer:create_vector_index,rag_pipeline.rag.policies:generate_questions,rag_pipeline.rag.retrieval:retrieve,rag_pipeline.rag.retrieval:filter_relevant,rag_pipeline.rag.question_answering:question_answering,rag_pipeline.rag.evaluation:evaluate_answers,rag_pipeline.rag.annotation:annotate_documents node
    class rag_pipeline.catalog:retrieved_pages,rag_pipeline.catalog:llm_answers,rag_pipeline.catalog:questions,rag_pipeline.catalog:relevant_pages io1
    class rag_pipeline.catalog:index io0
    class rag_pipeline.catalog:llm_model,rag_pipeline.catalog:llm_vision_retrieval_model io5
    class rag_pipeline.catalog:metrics io3
    class rag_pipeline.catalog:pdf_documents,rag_pipeline.catalog:pdfs_documents_annotated io4
    class rag_pipeline.catalog:policies io2
    classDef node fill:#008AD7,color:#FFF
    classDef io fill:#FFD43B
    classDef io0 fill:#66c2a5
    classDef io1 fill:#fc8d62
    classDef io2 fill:#8da0cb
    classDef io3 fill:#e78ac3
    classDef io4 fill:#a6d854
    classDef io5 fill:#ffd92f

View example

Starter Kedro Spaceflights

This project contains the Kedro spaceflights starter, migrated to Ordeq.

graph TB
    kedro_spaceflights.catalog:companies --> kedro_spaceflights.pipeline:preprocess_companies
    kedro_spaceflights.pipeline:preprocess_companies --> kedro_spaceflights.catalog:preprocessed_companies
    kedro_spaceflights.catalog:shuttles --> kedro_spaceflights.pipeline:preprocess_shuttles
    kedro_spaceflights.pipeline:preprocess_shuttles --> kedro_spaceflights.catalog:preprocessed_shuttles

    kedro_spaceflights.pipeline:preprocess_companies@{shape: rounded, label: "preprocess_companies"}
    kedro_spaceflights.pipeline:preprocess_shuttles@{shape: rounded, label: "preprocess_shuttles"}
    kedro_spaceflights.catalog:companies@{shape: rect, label: "companies"}
    kedro_spaceflights.catalog:preprocessed_companies@{shape: rect, label: "preprocessed_companies"}
    kedro_spaceflights.catalog:preprocessed_shuttles@{shape: rect, label: "preprocessed_shuttles"}
    kedro_spaceflights.catalog:shuttles@{shape: rect, label: "shuttles"}

    class kedro_spaceflights.pipeline:preprocess_companies,kedro_spaceflights.pipeline:preprocess_shuttles node
    class kedro_spaceflights.catalog:companies io0
    class kedro_spaceflights.catalog:preprocessed_companies,kedro_spaceflights.catalog:preprocessed_shuttles io2
    class kedro_spaceflights.catalog:shuttles io1
    classDef node fill:#008AD7,color:#FFF
    classDef io fill:#FFD43B
    classDef io0 fill:#66c2a5
    classDef io1 fill:#fc8d62
    classDef io2 fill:#8da0cb

View example

Starter Nested Subpipelines

This example demonstrates how to create and use nested sub-pipelines in Ordeq.
```
graph TB
```
View example

Starter Package

This example demonstrates how to create a Python package using Ordeq.

graph TB
    starter_package.catalog:user --> starter_package.pipeline:parse_users
    starter_package.pipeline:parse_users --> starter_package.catalog:yaml

    starter_package.pipeline:parse_users@{shape: rounded, label: "parse_users"}
    starter_package.catalog:user@{shape: rect, label: "user"}
    starter_package.catalog:yaml@{shape: rect, label: "yaml"}

    class starter_package.pipeline:parse_users node
    class starter_package.catalog:user io0
    class starter_package.catalog:yaml io1
    classDef node fill:#008AD7,color:#FFF
    classDef io fill:#FFD43B
    classDef io0 fill:#66c2a5
    classDef io1 fill:#fc8d62

View example

Starter Single File

This is a simple example of an Ordeq pipeline defined in a single file.

graph TB
    single_file:user --> single_file:parse_users
    single_file:parse_users --> single_file:yaml

    single_file:parse_users@{shape: rounded, label: "parse_users"}
    single_file:user@{shape: rect, label: "user"}
    single_file:yaml@{shape: rect, label: "yaml"}

    class single_file:parse_users node
    class single_file:user io0
    class single_file:yaml io1
    classDef node fill:#008AD7,color:#FFF
    classDef io fill:#FFD43B
    classDef io0 fill:#66c2a5
    classDef io1 fill:#fc8d62

View example

Starter Subpipelines

This example demonstrates how to create and use sub-pipelines in Ordeq.
```
graph TB
```
View example

Testing Nodes

This example demonstrates how to test nodes.

graph TB
    testing_nodes.catalog:txs --> testing_nodes.pipeline:join_txs_and_clients
    testing_nodes.catalog:clients --> testing_nodes.pipeline:join_txs_and_clients
    testing_nodes.catalog:date --> testing_nodes.pipeline:join_txs_and_clients
    testing_nodes.pipeline:join_txs_and_clients --> testing_nodes.catalog:txs_and_clients
    testing_nodes.catalog:txs_and_clients --> testing_nodes.pipeline:aggregate_txs
    testing_nodes.pipeline:aggregate_txs --> testing_nodes.catalog:aggregated_txs

    testing_nodes.pipeline:join_txs_and_clients@{shape: rounded, label: "join_txs_and_clients"}
    testing_nodes.pipeline:aggregate_txs@{shape: rounded, label: "aggregate_txs"}
    testing_nodes.catalog:txs_and_clients@{shape: rect, label: "txs_and_clients"}
    testing_nodes.catalog:aggregated_txs@{shape: rect, label: "aggregated_txs"}
    testing_nodes.catalog:clients@{shape: rect, label: "clients"}
    testing_nodes.catalog:date@{shape: rect, label: "date"}
    testing_nodes.catalog:txs@{shape: rect, label: "txs"}

    class testing_nodes.pipeline:join_txs_and_clients,testing_nodes.pipeline:aggregate_txs node
    class testing_nodes.catalog:txs_and_clients,testing_nodes.catalog:aggregated_txs,testing_nodes.catalog:clients,testing_nodes.catalog:txs io1
    class testing_nodes.catalog:date io0
    classDef node fill:#008AD7,color:#FFF
    classDef io fill:#FFD43B
    classDef io0 fill:#66c2a5
    classDef io1 fill:#fc8d62

View example