Keep artifacts local in a cloud instance¶
If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.
Let us first create a cloud instance that woul store artifacts exclusively on S3.
!lamin login testuser1
!lamin init --storage s3://lamindb-ci/keep-artifacts-local
Let’s import lamindb and track the current notebook run.
# !pip install 'lamindb[jupyter]'
import lamindb as ln
ln.track("l9lFf83aPwRc0000")
Toggling setting “keep artifacts local”¶
You checkmark the “Keep artifacts local” box on the instance settings tab.
For the current test instance, we toggle it through the following private attribute.
ln.setup.settings.instance._keep_artifacts_local = True
Create a local storage location¶
Call the following for a – potentially pre-existing – root path and a unique host identifier.
ln.Storage(root="./our_local_storage", host="abc-institute-drive1").save()
Now, you have two storage locations: one in the S3 bucket, and the other locally.
ln.Storage.df()
Use a local storage location¶
If you save an artifact in keep-artifacts-local mode, by default, it’s stored in local storage.
original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, key="example_datasets/file1.fcs").save()
local_path = artifact.path # local storage path
local_path
You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./our_local_storage/:
assert artifact.path.exists()
assert artifact.path.as_posix().startswith(ln.settings.local_storage.root.as_posix())
ln.settings.local_storage.root.view_tree()
Pre-existing artifacts¶
Assume you already have a file in your local storage location:
file_in_local_storage = ln.core.datasets.file_bam()
file_in_local_storage.rename("./our_local_storage/output.bam")
ln.UPath("our_local_storage/").view_tree()
When registering an artifact for it, it remains where it is.
my_existing_file = ln.Artifact("./our_local_storage/output.bam").save()
ln.UPath("our_local_storage/").view_tree()
The storage path of the artifact matches the pre-existing file:
my_existing_file.path
Switching between local storage locations¶
You might have several local storage locations. Here is how you can switch between them.
ln.Storage(root="./our_local_storage2", host="abc-institute-drive1").save()
ln.settings.local_storage = "./our_local_storage2" # switch to the new storage location
Ingest a file into the new local storage location.
filepath = ln.core.datasets.file_fastq()
artifact3 = ln.Artifact(filepath, key="example_datasets/file.fastq.gz").save()
Inspect where all the files are.
ln.Artifact.df(include=["storage__root", "storage__region"])
Upload a local artifact to the cloud¶
If you’d like to upload an artifact to the cloud storage location to more easily share it or view it through web applications, you pass upload=True to the save() method.
artifact.save(upload=True)
→ moved local artifact to cache: /home/runner/.cache/lamindb/GfNVEE0Lq0L7PreH0000.fcs
Artifact(uid='GfNVEE0Lq0L7PreH0000', is_latest=True, key='example_datasets/file1.fcs', suffix='.fcs', size=19330507, hash='rCPvmZB19xs4zHZ7p_-Wrg', branch_id=1, space_id=1, storage_id=1, run_id=1, created_by_id=1, created_at=2025-07-15 14:33:49 UTC)
You now see the artifact in the S3 bucket:
ln.settings.storage.root.view_tree()
2 sub-directories & 3 files with suffixes '.txt', '.fcs', '.db'
s3://lamindb-ci/keep-artifacts-local
└── .lamindb/
├── GfNVEE0Lq0L7PreH0000.fcs
├── lamin.db
├── storage_uid.txt
└── _exclusion/
And it’s no longer present in local storage:
assert artifact.path.exists()
assert not local_path.exists()
assert artifact.path.as_posix().startswith(ln.settings.storage.root.as_posix())
ln.settings.local_storage.root.view_tree()
1 sub-directory & 2 files with suffixes '.fastq.gz', '.txt'
/home/runner/work/lamindb/lamindb/docs/faq/our_local_storage2
└── .lamindb/
├── storage_uid.txt
└── pEtoFuPhcD4uB15j0000.fastq.gz
Upload directly to the cloud¶
You can also directly upload via upload=True:
filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, key="example_datasets/mini.csv").save(upload=True)
artifact2.path
S3QueryPath('s3://lamindb-ci/keep-artifacts-local/.lamindb/ixWcR40aCe4cPkoT0000.csv')
Now we have two files on S3:
ln.Artifact.df(include="storage__root")
| uid | key | storage__root | |
|---|---|---|---|
| id | |||
| 4 | ixWcR40aCe4cPkoT0000 | example_datasets/mini.csv | s3://lamindb-ci/keep-artifacts-local |
| 3 | pEtoFuPhcD4uB15j0000 | example_datasets/file.fastq.gz | /home/runner/work/lamindb/lamindb/docs/faq/our... |
| 2 | 62ajjZBAouzDKgGG0000 | output.bam | /home/runner/work/lamindb/lamindb/docs/faq/our... |
| 1 | GfNVEE0Lq0L7PreH0000 | example_datasets/file1.fcs | s3://lamindb-ci/keep-artifacts-local |
Update storage description¶
You can add a description to the storage location by using the description field.
storage_record = ln.Storage.get(root__endswith="our_local_storage")
storage_record.description = "Our shared directory for project X"
storage_record.save()
ln.Storage.df()
| uid | root | description | type | region | instance_uid | space_id | run_id | created_at | created_by_id | _aux | branch_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||
| 3 | RoO6nDiiE2cg | /home/runner/work/lamindb/lamindb/docs/faq/our... | None | local | abc-institute-drive1 | 6uGWmLpZlNoJ | 1 | 1.0 | 2025-07-15 14:33:49.519000+00:00 | 1 | None | 1 |
| 2 | 1RAegxTyoUqC | /home/runner/work/lamindb/lamindb/docs/faq/our... | Our shared directory for project X | local | abc-institute-drive1 | 6uGWmLpZlNoJ | 1 | 1.0 | 2025-07-15 14:33:48.348000+00:00 | 1 | None | 1 |
| 1 | HyMNGTxYEmWH | s3://lamindb-ci/keep-artifacts-local | None | s3 | us-west-1 | 6uGWmLpZlNoJ | 1 | NaN | 2025-07-15 14:33:38.592000+00:00 | 1 | None | 1 |
Delete the test instance¶
Delete the artifacts:
artifact.delete(permanent=True)
artifact2.delete(permanent=True)
artifact3.delete(permanent=True)
my_existing_file.delete(permanent=True, storage=False)
→ a file/folder remains here: /home/runner/work/lamindb/lamindb/docs/faq/our_local_storage/output.bam
Delete the instance:
ln.setup.delete("keep-artifacts-local", force=True)