Running a Custom Python Script on One or More Materials

The platform can run an arbitrary Python script against materials from an account's collection, with no simulation engine involved. The script, its dependencies and any data files it reads are uploaded to the account's object storage folder; a workflow fetches them onto the compute node next to the material, and whatever the script prints comes back as the result.

This page describes the flow through the custom_python_calculation.ipynb notebook, which performs the upload, assembles the workflow, and creates one job per material. For a script that needs neither a material from the collection nor data files of its own, the general Python flavor/template described in Python MLFF is sufficient on its own.

1. Prepare the script

The script runs on the compute node in a working directory that holds the material and the uploaded files side by side, so every path in it is relative. Nothing in the script is specific to the platform: it reads files, computes, and prints.

1.1. Read the material

The job's material is written to material.json before the script starts, in the representation the platform stores it in — a lattice object holding the cell lengths and angles, and a basis object holding the elements and their coordinates. A script that reports the number of atoms and the elements present reads it as follows:

import json

material = json.load(open("material.json"))
elements = [element["value"] for element in material["basis"]["elements"]]

print(json.dumps({"n_atoms": len(elements), "elements": sorted(set(elements))}))

This is the default script in the notebook, and it serves as the starting point for a custom one.

1.2. Print the results

The standard output of the script is the result. The notebook parses each line that is a JSON object and renders the parsed values as a table, one row per material, so a script that prints a single JSON object per run needs no further formatting. Lines that are not JSON are shown as printed, which makes print() usable for progress messages and debugging.

1.3. Add data files

A script that reads its own data — a table of radii, a set of parameters, a pseudopotential — takes those files from the uploads folder of the JupyterLite session. The files are placed there first, by dragging them into the file browser, and named in the USER_ASSET_FILES parameter. The notebook then uploads them next to the script, and the workflow fetches them into the same working directory, where the script opens them by name.

A data file may be of any type and any size — a table, a pseudopotential, a model checkpoint — because data files go straight to object storage rather than through the platform. Section 6 covers what that means for the script itself.

1.4. Declare dependencies

Packages the script imports are listed in the USER_REQUIREMENTS parameter, in the form used by requirements.txt — for example ["numpy<2"]. The list is installed into a Python virtual environment on the compute node before the script starts. An empty list is valid, and is what the default script uses, since it imports nothing beyond the standard library.

Shared virtual environment

Virtual environments are shared across jobs and users. As long as the contents of the requirement list are unchanged, the same environment is reused, so the first job pays for pip install and later ones start immediately. Versions that need to be exact should be pinned explicitly.

2. Run the notebook

The notebook lives in the JupyterLite session that ships with the platform, and it runs entirely in the browser until the moment it submits the jobs.

2.1. Open the notebook

First, open a JupyterLite session from the account page. Then, in the file browser, open made, then workflows, then custom_python_calculation.ipynb.

2.2. Set the parameters

The parameters sit in the cell under 1.2. Set parameters and configurations for the workflow and job, and the script sits on its own in the cell under 1.3. Set the script to run, so the two are edited independently.

MATERIAL_NAMES lists the materials, one job per entry. Each name is looked up in the uploads folder first and in the standard materials set that ships with the notebook environment as a fallback, then saved to the account's collection if an identical material is not there already. USER_ASSET_FILES and USER_REQUIREMENTS are described in Section 1 above. MY_WORKFLOW_NAME names the workflow, and save_to_collection decides whether it is kept. The cluster, queue and processor count are set further down the same cell.

2.3. Run all cells

Click Run > Run All. The notebook authenticates in the browser, uploads the script and the data files, assembles the workflow, creates one job per material, submits them, and polls until each one finishes. A run against a small material takes a few minutes, most of it spent waiting for the queue and, on the first run with a given requirement list, for pip install.

2.4. View the results

The last cell prints the standard output of each job and renders the parsed JSON values as a table. The same output is available in the Job Viewer: the Workflow tab exposes the standard output of the script unit, and the Files tab lists the uploaded files, the material, and everything the script wrote.

3. Anatomy of the workflow

The notebook does not build a workflow from nothing. It takes the Custom Python Script workflow from the standard workflow set and fills in the three things that differ per run. The same workflow is held in the Workflows Bank, where it can be inspected in the web interface. It consists of four units:

• I/O unit, object storage: fetches the uploaded script and data files into the working directory. The notebook sets its input list from the upload it has just performed.
• I/O unit, material: fetches the job's material through the API, by the id the job carries.
• Assignment unit: takes the first material and assigns it to a global variable named MATERIAL.
• Execution unit: runs a short runner script that writes MATERIAL to material.json and executes the uploaded script in the same process. The notebook sets the runner and the requirement list on this unit.

The runner is what keeps the user's script untouched: the script is fetched as a file rather than embedded in the workflow, so text in it that resembles a template placeholder is never rendered.

4. Reuse the saved workflow

With save_to_collection left at its default, the workflow is saved to the account's collection under the name given by MY_WORKFLOW_NAME. It points at the uploaded files, so it can be selected in the Jobs Designer like any other workflow and run against a different material without opening the notebook again.

Re-running the notebook overwrites the uploaded files in place, under the same names in the same folder. An edited script therefore reaches a saved workflow without any change to the workflow itself.

5. The shell twin: run any node-side application

A sibling notebook, custom_shell_calculation.ipynb, carries the same flow with a shell script in place of the Python one. A shell script runs in a module-capable shell, so it can invoke any application installed on the compute node. Its default example builds Quantum ESPRESSO inputs from material.json and runs an SCF and a band structure step for silicon with a pseudopotential from the platform's library. A pseudopotential of the user's own goes up through the same upload call as any data file — placed in the uploads folder, listed in USER_ASSET_FILES, with the script's PSEUDO_DIR, PSEUDO_FILES and MASSES set at the top of the script — and Quantum ESPRESSO's own log then names the uploaded file as the one it read. Such a file can also be registered as a first-class pseudopotential through the web interface, as described in Upload a custom pseudopotential.

6. File size

A file that travels inside the request carrying it is bounded by the size of that request: roughly 75 MB through the web interface and 37 MB through the API, the same bound for text and binary alike. The notebook sends only the script that way. Everything named in USER_ASSET_FILES takes the other route — the platform signs a short-lived upload URL and the file goes straight to object storage, with no size limit — so a pseudopotential and a several-hundred-megabyte model checkpoint are listed exactly the same way.

The script is given one material per job, as material.json. A calculation over a set of materials at once is a different shape of workflow, and is not what this notebook builds.

7. Links

The pages below cover the platform features this notebook builds on.

• Dropbox — the object storage folder the files are uploaded to.
• Python — the application the execution unit uses.
• JupyterLite — the in-browser notebook environment.
• File endpoint — the API the notebook uploads through, usable directly for the same purpose.
• api-examples — the repository the notebook is maintained in.