Provenance-Aware Fine-grained Containerized Workflows and Orchestration
Date Issued
May 1, 2026
Author(s)
Anderson, Ty Julian
The University of Tennessee
Advisor(s)
Michela Taufer
Additional Advisor(s)
Kin Ng Lugo
Jack Marquez
Abstract
Reproducibility in computational science requires precise control over execution environments, workflow structure, and data lineage. Prior work introduced a provenance-aware fine-grained containerization framework built on Apptainer that decomposes scientific workflows into individual data and application containers and captures runtime provenance. However, the system was limited to single-node execution, static SIF-based data containers, and manually orchestrated workflows. This thesis extends fine-grained containerized environments to support arbitrary multi-layer Directed Acyclic Graph (DAG) workflows and distributed execution via the Pegasus Workflow Management System. We augment the Apptainer plugin to construct and execute DAG-based workflows locally using Kahn’s topological sorting algorithm and develop a Jupyter-based orchestration layer that translates containerized DAG specifications into Pegasus workflows for distributed execution. To address static storage limitations, we introduce IMG-based data containers that support dynamic resizing during runtime. Provenance capture is decoupled into a reusable library, enabling consistent metadata generation under both local and distributed execution. We evaluate the extended system by measuring zero-copy throughput for SIF and IMG containers, comparing Pegasus job wall time for a bioinformatics variant-calling workflow under containerized and non-containerized execution, and analyzing the data-transfer-to-computation ratio. Results demonstrate that fine-grained containerized workflows can scale to distributed infrastructures while preserving provenance guarantees with manageable performance overhead.
Degree
Master of Science
Major
Computer Science
File(s)![Thumbnail Image]()
Name
ANDERSON-PRIMARY-2026.pdf
Size
1.87 MB
Format
Adobe PDF
Checksum (MD5)
208ac6d0bf6b652839fadec5233bdbc7