Introduction to managed workflows

Plan availability

Qiskit Functions and managed workflows are beta features available only to Premium, Flex, and On-Prem (through the IBM Quantum® Platform API) plan users.

Managed workflows run your custom Python code on classical compute that IBM Quantum Platform manages for you. Use managed workflows to run hybrid quantum-classical workflows, large-scale diagonalization, and long-running parallel tasks without managing compute infrastructure or storage.

When to use a managed workflow

Use a managed workflow when no pre-built Catalog Function covers your task, or when you want to package your own method as a reusable function. Managed workflows are a good fit when you want to do the following:

Run long jobs in the cloud. A managed workflow keeps running when you close your laptop, and you can reconnect to the job later.
Use large-memory and GPU nodes. Choose the memory, CPU, and GPU resources that the workload needs.
Run tasks in parallel. Start many single-node jobs at the same time.

You provide the workflow code, its dependencies, and the compute sizes that it supports. Manage workflows run the workflow on IBM®-managed compute in a pre-built environment with your packages installed, and provides cloud storage for the files that the workflow reads and writes.

How managed workflows work

You upload a workflow one time and run as many jobs as you need. Every job works in the following ways:

The job starts in a new environment with an empty local disk. Managed workflows shut down the environment after the job ends.
The job runs on one of the compute sizes that you define when you upload the workflow. You can choose a different size for each run.
The job can read and write the workflow's file storage. Files in file storage remain after the job ends.

Because each job starts with an empty local disk, a later job can find only the files that an earlier job saved to file storage.

Lifecycle of a run

Each job runs asynchronously for up to 24 hours. You can have up to 50 active jobs at the same time if your instance has enough allocation.

From your Python session, you call upload() one time to upload the workflow's code, packages, and compute profiles. No job exists yet. Each call to run(args) creates a job that moves through these stages: Queue, where it waits for allocation with status QUEUED; Start, where the managed workflow creates the environment with status INITIALIZING; Running, where your code can optionally set stages with update_status(): Mapping (RUNNING: MAPPING), Optimizing (RUNNING: OPTIMIZING_FOR_HARDWARE), Waiting for QPU (RUNNING: WAITING_FOR_QPU), Executing QPU (RUNNING: EXECUTING_QPU), and Post-processing (RUNNING: POST_PROCESSING); and Result, where the environment shuts down with status DONE, ERROR, or CANCELED. While running, the job reads and writes workflow file storage, which exists outside the job, is shared by all runs, and persists before and after each job. From your session, you can read status() from Queue onward, logs() from Running onward, and result() after the job reaches Result. All three remain available after the job ends.

The following table describes each stage in the diagram and the job status that you see during that stage:

Stage	What happens	Job status
Upload	You upload the workflow's code, dependencies, and sizes. The workflow rejects an invalid upload before any job runs.	None (no job yet)
Queue	You call `run()`. The workflow checks whether your instance has enough allocation, and schedules the job when compute is available.	`QUEUED`
Start	The workflow creates an environment with the compute profile that you selected.	`INITIALIZING`
Run	Your workflow's entrypoint runs. The entrypoint can report its progress with `update_status()`, and you can read its logs with `logs()` while it runs.	`RUNNING`, or a stage that you set, such as `RUNNING: WAITING_FOR_QPU`
Result	The job ends, and the workflow shuts down its environment, while keeping the result and logs.	`DONE`, `ERROR`, or `CANCELED`

What a running job can access

Each job runs in an isolated environment. The following diagram and table show the resources that a running job can use.

You call run() with arguments from your Python session and receive status, logs, and the result. Inside the managed workflow, each run executes in an isolated job environment with access to its compute (CPU, RAM, and optionally GPU), your code and packages, and its arguments. The job produces outputs with save_result(), logs, and update_status(), and submits QPU jobs that run on your instance. File storage and your instance, which provides access and cost management, are shared across all managed workflow jobs and persist after the job ends. The job has no internet access.

Resource	What the job can use
Compute	The CPU, memory, and GPUs of the compute profile that you selected for the job.
Code and packages	The files that you uploaded with the workflow, and the supported packages that are installed in the environment.
Arguments	The arguments that you passed to `run()`. Read them with `get_arguments()`.
QPU access	The credentials and instance that you used to call `run()`. QPU jobs that the workflow submits run on your instance and count against your usage.
Outputs	A result that the job saves with `save_result()`, logs that the job prints, and status updates that the job sets with `update_status()`.
File storage	Files that all of your runs of the workflow share. Files remain in storage after the job ends.
Internet	No access. A job cannot make outbound connections. Declare packages when you upload the workflow, and provide data as arguments or files.

Security and isolation

Managed workflows isolate each job from other jobs and limits the privileges of the code that runs in it. The access boundaries in the preceding table are part of this isolation. Managed workflows also apply the following protections:

Job isolation. No other job runs in your job's environment. This includes other users' jobs and your own jobs that run at the same time.
Results, logs, and storage. Only the user who ran a job can read its results and logs.
Credentials. Managed workflows encrypt the credentials that a job uses, including your API token, when it stores them. Your workflow code can read these credentials while the job runs.
Privileges. Managed workflows remove administrator-level privileges from the environment before your entrypoint starts.

Compare with on-premises HPC

Managed workflows differ from an on-premises HPC cluster in how you get compute time, set up software, and submit jobs. The following table compares common tasks.

Task	On-premises HPC cluster	Managed workflows
Get compute time	Request an allocation or a reservation from the cluster administrator.	Use the allocation of your IBM Quantum instance. The workflow queues each job when you call `run()`.
Access the system	Connect to a login node with SSH.	Use the Python client on your local computer. Managed workflows do not provide a login node.
Install software	Build software, load modules, or ask an administrator to install packages.	Use the supported packages that are installed in every environment, or upload wheels for other packages.
Submit a job	Write a batch script and submit it to the scheduler.	Call `run()` with the job arguments.
Request resources	Request nodes, CPUs, memory, and GPUs.	Select one of the compute sizes that are defined for the workflow. Each job runs on a single node.
Run a parameter sweep	Submit a job array.	Call `run()` one time for each set of arguments.
Store shared files	Write to a shared scratch file system.	Write to the file storage for the workflow.
Limit run time	Set a wall-time limit.	The workflow cancels a job after 24 hours.
Checkpoint long jobs	Write checkpoints to scratch storage.	Save intermediate results to file storage.
Rerun a failed job	Resubmit the job.	Call `run()` again. The workflow does not retry failed jobs automatically.
Monitor jobs	Check the queue and log files from the login node.	Check the status, logs, and result of each job from your local Python session.

Next steps

Recommendations

Install the client packages and authenticate.
Deploy your first managed workflow to upload a workflow, run it, and get its result.

¿Le ha resultado útil esta página?

Informe de un error, de una errata o solicite contenido en GitHub.