Project flow¶
LaminDB allows tracking data lineage on the entire project level.
Here, we walk through exemplified app uploads, pipelines & notebooks following Schmidt et al., 2022.
A CRISPR screen reading out a phenotypic endpoint on T cells is paired with scRNA-seq to generate insights into IFN-γ production.
These insights get linked back to the original data through the steps taken in the project to provide context for interpretation & future decision making.
More specifically: Why should I care about data flow?
Data flow tracks data sources & transformations to trace biological insights, verify experimental outcomes, meet regulatory standards, increase the robustness of research and optimize the feedback loop of team-wide learning iterations.
While tracking data flow is easier when it’s governed by deterministic pipelines, it becomes hard when it’s governed by interactive human-driven analyses.
LaminDB interfaces workflow mangers for the former and embraces the latter.
# !pip install 'lamindb[jupyter,bionty,aws]'
!lamin init --storage ./mydata
Show code cell output
→ connected lamindb: testuser1/mydata
Import lamindb:
import lamindb as ln
from IPython.display import Image, display
→ connected lamindb: testuser1/mydata
Steps¶
In the following, we walk through exemplified steps covering different types of transforms (Transform).
Note
The full notebooks are in this repository.
App upload of phenotypic data 
¶
Register data through app upload from wetlab by testuser1:
# This function mimics the upload of artifacts via the UI
# In reality, you simply drag and drop files into the UI
def mock_upload_crispra_result_app():
ln.setup.login("testuser1")
transform = ln.Transform(name="Upload GWS CRISPRa result", type="upload")
ln.track(transform=transform)
output_path = ln.core.datasets.schmidt22_crispra_gws_IFNG(ln.settings.storage.root)
output_file = ln.Artifact(
output_path, description="Raw data of schmidt22 crispra GWS"
)
output_file.save()
mock_upload_crispra_result_app()
Show code cell output
→ created Transform('SJz9qqjO'), started new Run('F2FIMZSW') at 2024-10-04 09:15:08 UTC
Hit identification in notebook 
¶
Access, transform & register data in drylab by testuser2 in notebook hit-identification.
Show code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
import nbproject_test
from pathlib import Path
cwd = Path.cwd()
nbproject_test.execute_notebooks(
cwd / "project-flow-scripts/hit-identification.ipynb", write=True
)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/hit-identification.ipynb
Scheduled: ['hit-identification']
hit-identification
✓ (3.724s)
Total time: 3.725s
Inspect data flow:
artifact = ln.Artifact.get(description="hits from schmidt22 crispra GWS")
artifact.view_lineage()
Sequencer upload 
¶
Upload files from sequencer via script chromium_10x_upload.py:
!python project-flow-scripts/chromium_10x_upload.py
Show code cell output
→ connected lamindb: testuser1/mydata
→ created Transform('qCJPkOuZ'), started new Run('eyObGxbv') at 2024-10-04 09:15:15 UTC
→ finished Run('eyObGxbv') after 0:00:00.348624 at 2024-10-04 09:15:15 UTC
scRNA-seq bioinformatics pipeline 
¶
Process uploaded files using a script or workflow manager: Pipelines – workflow managers and obtain 3 output files in a directory filtered_feature_bc_matrix/:
!python project-flow-scripts/cellranger.py
Show code cell output
→ connected lamindb: testuser1/mydata
→ created Transform('GQcDqRDD'), started new Run('qU2UHjq4') at 2024-10-04 09:15:17 UTC
!python project-flow-scripts/postprocess_cellranger.py
Show code cell output
→ connected lamindb: testuser1/mydata
→ created Transform('YqmbO6oM'), started new Run('KFE2R8FT') at 2024-10-04 09:15:19 UTC
Inspect data flow:
output_file = ln.Artifact.get(description="perturbseq counts")
output_file.view_lineage()
Integrate scRNA-seq & phenotypic data ¶
Integrate data in notebook integrated-analysis.
Show code cell content
# the following mimics the integrated analysis notebook
# In reality, you would execute inside the notebook
nbproject_test.execute_notebooks(
cwd / "project-flow-scripts/integrated-analysis.ipynb", write=True
)
Executing notebooks in /home/runner/work/lamin-usecases/lamin-usecases/docs/project-flow-scripts/integrated-analysis.ipynb
Scheduled: ['integrated-analysis']
integrated-analysis
✓ (4.136s)
Total time: 4.137s
Review results¶
Let’s load one of the plots:
# track the current notebook as transform
ln.track("1LCd8kco9lZU0000")
→ notebook imports: ipython==8.28.0 lamindb==0.76.11 nbproject_test==0.5.1
→ created Transform('1LCd8kco'), started new Run('Y9SjBskD') at 2024-10-04 09:15:24 UTC
artifact = ln.Artifact.get(key__contains="figures/matrixplot")
artifact.cache()
Show code cell output
PosixUPath('/home/runner/work/lamin-usecases/lamin-usecases/docs/mydata/.lamindb/KVAFTCMaWctzMmrR0000.png')
display(Image(filename=artifact.path))
We see that the image artifact is tracked as an input of the current notebook. The input is highlighted, the notebook follows at the bottom:
artifact.view_lineage()
Alternatively, we can also look at the sequence of transforms:
transform = ln.Transform.search("Project flow").first()
transform.predecessors.df()
| uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 6 | lB3IyPLQSmvt0000 | None | True | Perform single cell analysis, integrate with C... | integrated-analysis.ipynb | None | notebook | None | None | None | None | None | 2024-10-04 09:15:23.302140+00:00 | 2 |
transform.view_lineage()
Understand runs¶
We tracked pipeline and notebook runs through track(), which stores a Transform and a Run record within a global context.
Artifact objects are the inputs and outputs of runs.
What if I don’t want a global context?
Sometimes, we don’t want to create a global run context but manually pass a run when creating an artifact:
run = ln.Run(transform=transform)
ln.Artifact(filepath, run=run)
When does an artifact appear as a run input?
When accessing an artifact via cache(), load() or open(), two things happen:
The current run gets added to
artifact.input_ofThe transform of that artifact gets added as a parent of the current transform
You can then switch off auto-tracking of run inputs if you set ln.settings.track_run_inputs = False: Can I disable tracking run inputs?
You can also track run inputs on a case by case basis via is_run_input=True, e.g., here:
artifact.load(is_run_input=True)
Query by provenance¶
We can query or search for the notebook that created the artifact:
transform = ln.Transform.search("GWS CRIPSRa analysis").first()
And then find all the artifacts created by that notebook:
ln.Artifact.filter(transform=transform).df()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 2 | mvk1Lgie8jwbZm4E0000 | None | True | hits from schmidt22 crispra GWS | None | .parquet | dataset | 18368 | y_EDGTFsymBWJ0G0aUWHQg | None | None | md5 | DataFrame | 1 | True | 1 | 2 | 2 | 2024-10-04 09:15:13.080815+00:00 | 2 |
Which transform ingested a given artifact?
artifact = ln.Artifact.filter().first()
artifact.transform
Transform(uid='SJz9qqjOOdpp0000', is_latest=True, name='Upload GWS CRISPRa result', type='upload', created_by_id=1, created_at=2024-10-04 09:15:08 UTC)
And which user?
artifact.created_by
User(uid='DzTjkKse', handle='testuser1', name='Test User1', created_at=2024-10-04 09:15:07 UTC)
Which transforms were created by a given user?
users = ln.User.lookup()
ln.Transform.filter(created_by=users.testuser1).df()
| uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 1 | SJz9qqjOOdpp0000 | None | True | Upload GWS CRISPRa result | None | None | upload | None | None | None | None | None | 2024-10-04 09:15:08.880477+00:00 | 1 |
| 3 | qCJPkOuZAi9q0000 | None | True | chromium_10x_upload.py | chromium_10x_upload.py | None | script | import lamindb as ln\n\nln.setup.login("testus... | nXWdh475QhVKuoAfToWZTw | None | None | None | 2024-10-04 09:15:15.012980+00:00 | 1 |
| 7 | 1LCd8kco9lZU0000 | None | True | Project flow | project-flow.ipynb | None | notebook | None | None | None | None | None | 2024-10-04 09:15:24.821473+00:00 | 1 |
Which notebooks were created by a given user?
ln.Transform.filter(created_by=users.testuser1, type="notebook").df()
| uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 7 | 1LCd8kco9lZU0000 | None | True | Project flow | project-flow.ipynb | None | notebook | None | None | None | None | None | 2024-10-04 09:15:24.821473+00:00 | 1 |
We can also view all recent additions to the entire database:
ln.view()
Show code cell output
Artifact
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 11 | KVAFTCMaWctzMmrR0000 | None | True | None | figures/matrixplot_fig2_score-wgs-hits-per-clu... | .png | None | 28814 | Uqpe3rI2qCa6KHvUIxadLw | None | None | md5 | None | 1 | True | 1 | 6 | 6 | 2024-10-04 09:15:24.039658+00:00 | 2 |
| 10 | XvBizK2ebGQTU3TW0000 | None | True | None | figures/umap_fig1_score-wgs-hits.png | .png | None | 118999 | JMFvnvCcQzIdtXM8Y12MKg | None | None | md5 | None | 1 | True | 1 | 6 | 6 | 2024-10-04 09:15:23.869838+00:00 | 2 |
| 9 | H1jqdG1MsG0FRbVj0000 | None | True | perturbseq counts | schmidt22_perturbseq.h5ad | .h5ad | None | 20659936 | la7EvqEUMDlug9-rpw-udA | None | None | md5 | AnnData | 1 | False | 1 | 5 | 5 | 2024-10-04 09:15:20.076271+00:00 | 2 |
| 8 | Yx89knwsxeXwN9ll0000 | None | True | None | perturbseq/filtered_feature_bc_matrix/matrix.m... | .mtx.gz | None | 6 | oZhdXqd1-u8EEKtPQjN_6w | None | None | md5 | None | 1 | False | 1 | 4 | 4 | 2024-10-04 09:15:17.622517+00:00 | 2 |
| 7 | FJjEugzX66UMBnNh0000 | None | True | None | perturbseq/filtered_feature_bc_matrix/barcodes... | .tsv.gz | None | 6 | tE7XJKhdYEyzwdfXYXCibA | None | None | md5 | None | 1 | False | 1 | 4 | 4 | 2024-10-04 09:15:17.622094+00:00 | 2 |
| 6 | Sj7rXQDmUStw8xgt0000 | None | True | None | perturbseq/filtered_feature_bc_matrix/features... | .tsv.gz | None | 6 | f8PQV2cY_9UJyjQBakQC5A | None | None | md5 | None | 1 | False | 1 | 4 | 4 | 2024-10-04 09:15:17.621497+00:00 | 2 |
| 4 | UTZIERQ5Mho87NQC0000 | None | True | None | fastq/perturbseq_R2_001.fastq.gz | .fastq.gz | None | 6 | opydXo9qm158owl1UGRFoA | None | None | md5 | None | 1 | False | 1 | 3 | 3 | 2024-10-04 09:15:15.355145+00:00 | 1 |
Run
| uid | started_at | finished_at | is_consecutive | reference | reference_type | transform_id | report_id | environment_id | parent_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||
| 1 | F2FIMZSWpqymHBxxibDl | 2024-10-04 09:15:08.882447+00:00 | NaT | True | None | None | 1 | None | NaN | None | 2024-10-04 09:15:08.882474+00:00 | 1 |
| 2 | MDYWIKXydcG0vgv7g0GF | 2024-10-04 09:15:12.694039+00:00 | NaT | True | None | None | 2 | None | NaN | None | 2024-10-04 09:15:12.694068+00:00 | 2 |
| 3 | eyObGxbvX9lHE36keM0V | 2024-10-04 09:15:15.015558+00:00 | 2024-10-04 09:15:15.364182+00:00 | True | None | None | 3 | None | 5.0 | None | 2024-10-04 09:15:15.015583+00:00 | 1 |
| 4 | qU2UHjq4TizpzDTxkRrK | 2024-10-04 09:15:17.264810+00:00 | NaT | None | None | None | 4 | None | NaN | None | 2024-10-04 09:15:17.264837+00:00 | 2 |
| 5 | KFE2R8FTOuKgIKno5m5d | 2024-10-04 09:15:19.010373+00:00 | NaT | None | None | None | 5 | None | NaN | None | 2024-10-04 09:15:19.010401+00:00 | 2 |
| 6 | J6rppvcyz5YyX4zh3HgT | 2024-10-04 09:15:23.305348+00:00 | NaT | True | None | None | 6 | None | NaN | None | 2024-10-04 09:15:23.305378+00:00 | 2 |
| 7 | Y9SjBskDmXUB2e1Q5SNb | 2024-10-04 09:15:24.824413+00:00 | NaT | True | None | None | 7 | None | NaN | None | 2024-10-04 09:15:24.824446+00:00 | 1 |
Storage
| uid | root | description | type | region | instance_uid | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| 1 | a6e5pidrnEoK | /home/runner/work/lamin-usecases/lamin-usecase... | None | local | None | 54ZGqgkROOFf | None | 2024-10-04 09:15:07.225745+00:00 | 1 |
Transform
| uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 7 | 1LCd8kco9lZU0000 | None | True | Project flow | project-flow.ipynb | None | notebook | None | None | None | None | None | 2024-10-04 09:15:24.821473+00:00 | 1 |
| 6 | lB3IyPLQSmvt0000 | None | True | Perform single cell analysis, integrate with C... | integrated-analysis.ipynb | None | notebook | None | None | None | None | None | 2024-10-04 09:15:23.302140+00:00 | 2 |
| 5 | YqmbO6oMXjRj0000 | None | True | postprocess_cellranger.py | postprocess_cellranger.py | None | script | None | None | None | None | None | 2024-10-04 09:15:19.007744+00:00 | 2 |
| 4 | GQcDqRDDFqiT0000 | 7.2.0 | True | Cell Ranger | None | None | pipeline | None | None | https://www.10xgenomics.com/support/software/c... | None | None | 2024-10-04 09:15:17.262884+00:00 | 2 |
| 3 | qCJPkOuZAi9q0000 | None | True | chromium_10x_upload.py | chromium_10x_upload.py | None | script | import lamindb as ln\n\nln.setup.login("testus... | nXWdh475QhVKuoAfToWZTw | None | None | None | 2024-10-04 09:15:15.012980+00:00 | 1 |
| 2 | T0T28btuB0PG0000 | None | True | GWS CRIPSRa analysis | hit-identification.ipynb | None | notebook | None | None | None | None | None | 2024-10-04 09:15:12.691448+00:00 | 2 |
| 1 | SJz9qqjOOdpp0000 | None | True | Upload GWS CRISPRa result | None | None | upload | None | None | None | None | None | 2024-10-04 09:15:08.880477+00:00 | 1 |
User
| uid | handle | name | created_at | |
|---|---|---|---|---|
| id | ||||
| 2 | bKeW4T6E | testuser2 | Test User2 | 2024-10-04 09:15:12.684762+00:00 |
| 1 | DzTjkKse | testuser1 | Test User1 | 2024-10-04 09:15:07.222505+00:00 |
Show code cell content
!lamin login testuser1
!rm -r ./mydata
!lamin delete --force mydata
✓ logged in with email testuser1@lamin.ai (uid: DzTjkKse)
• deleting instance testuser1/mydata