kunefe

Documentation about kunefe.

Submodules

Package Contents

Classes

Kunefe

Submit jobs to SLURM cluster.
class kunefe.Kunefe(username: str, hostname: str, port: int)[source]

Submit jobs to SLURM cluster.

username

login name of the user.

Type:

str

hostname

hostname or the IP address of the remote system.

Type:

str

port

SSH port to be used by the clients.

Type:

int

password

password to connect to the remote.

Type:

int

ssh_client

ssh client to connect and run commands on the remote system.

sftp_client

sftp client to copy files from and to a remote system.

set_password() → str[source]

Sets user password. The password is not echoed.

Returns:

password of the user

Return type:

password (str)

set_ssh_client() → paramiko.SSHClient[source]

Creates an SSH client and connects to remote system.

Returns:

ssh client

Return type:

ssh_client (paramiko.SSHClient)

connect_remote() → None[source]

Creates an ssh and sftp clients, prompts for user password and connects to the remote host.

Returns:

None

create_remote_folder(remote_folder: str) → None[source]

Create a folder in the remote system.

Parameters:

remote_folder (str) – path of the folder to be created on the remote system.

Returns:

None

get_files(remote_folder: str, local_folder: str = './') → None[source]

Get files from the remote system.

Parameters:

  • remote_folder (str) – path of the remote folder to copy the files from.

  • local_folder (str, optional) – path of the host folder to copy the files to. Defaults to “./”.

Returns:

None

put_files(remote_folder: str = '~', local_folder: str = './', verbose: bool = False) → None[source]

Copy files to the remote system.

Parameters:

  • remote_folder (str) – path on a remote system to copy the files to.

  • local_folder (str) – path to copy the files from.

  • verbose (bool) – show verbose info when copying.

Returns:

None

submit_job(job_file: str, verbose: bool = False) → Tuple[int, str, str, str][source]

Submit job to SLURM cluster.

Parameters:

  • job_file (str) – full path of the job script to be submitted.

  • verbose (bool) – run in verbose mode.

Returns:

job_id, stdin, stdout, stderr

Return type:

tuple(int, str, str, str)

build_apptainer_image(docker_image: str, sif_file_name: str = 'app.sif') → bool[source]

Builds an Apptainer image from a Docker image.

Parameters:

  • docker_image (str) – docker image name to be used to build an apptainer image.

  • sif_file_name (str, optional) – name of the apptainer image (sif) to be built. Defaults to ‘app.sif’.

Returns:

True if the apptainer image was successfully built. Otherwise, returns False.

Return type:

bool

install_apptainer_on_remote(install_path: str = '~') → bool[source]

Installs Apptainer on a remote system in unprivileged mode.

Parameters:

install_path (str) – path to install apptainer binary.

Returns:

True if the apptainer installation was successful. Otherwise, returns False.

Return type:

bool

check_local_command_exists(command: str) → bool[source]

Check whether command is on PATH and marked as executable.

Parameters:

command (str) – a command to be checked.

Returns:

True if command exists. Otherwise False.

Return type:

bool

Examples

>>> from kunefe import Kunefe
>>> kunefe = Kunefe(username="xenon", hostname="localhost", port=10022)
>>> kunefe.check_local_command_exists(command='ls')
True
>>> kunefe.check_local_command_exists(command='thisdoesnotexist')
False
check_required_tools(command_list: list[str]) → bool[source]

Check whether all required commands are available.

Parameters:

command_list (list[str]) – a list of tools to be checked.

Returns:

True if all the tools exist. Otherwise False.

Return type:

bool

Examples

>>> from kunefe import Kunefe
>>> kunefe = Kunefe(username="xenon", hostname="localhost", port=10022)
>>> kunefe.check_required_tools(['ls', 'rsync', 'cp'])
True
>>> kunefe.check_required_tools(['docker', 'rsync', 'bsxcommand'])
False
generate_job_script(job_name: str, sif_file_path: str, command: str, env_vars: str, job_time: str, job_file_path: str = './', template_name: str = 'generic') → None[source]

Generate a batch script file for job submission.

Parameters:

  • job_name (str) – name of the job to be used when submitting.

  • sif_file_path (str) – path of the Apptainer image.

  • command (str) – a command to be executed using Apptainer image.

  • env_vars (str) – environment variables to be used when submitting the job.

  • job_time (str) – time limit for the job.

  • job_file_path (str, optional) – path to save the generated batch job script. Defaults to ‘./’.

  • template_name (str, optional) – name of the template to be used. Defaults to ‘generic’.

Returns:

None

run_remote_command(command: str, timeout: int = 5, flush: bool = False, show_stdout: bool = False) → list[str][source]

Run a command on a remote system.

Parameters:

  • command (str) – command to be executed on the remote system.

  • timeout (int, optional) – time to wait before considering the command as failed. Defaults to 5.

  • flush (bool, optional) – flush the output. Defaults to False.

  • show_stdout (bool, optional) – prints the stdout. Defaults to False.

Returns:

stdin, stdout, stderr

Return type:

list[str]

watch_slurm_queue(sleep_time: float = 5.0) → None[source]

Watches the SLURM job queue.

Parameters:

sleep_time (float, optional) – time to wait before refreshing the queue status. Defaults to 5.0.

Returns:

None

cleanup() → None[source]

Destructor method to clean things up.

Returns:

None