Skip to main content

Shen Lab Handbook

6 Computing Resources

Shen Lab uses several levels of computing resources for research: local workstations, NTU high-performance computing resources, and national-scale computing resources such as Singapore NSCC. The right resource depends on the size of the data, the software needed, the expected runtime, and whether the work requires interactive use or batch jobs.

This chapter gives a practical overview. Specific server names, account policies, queues, storage paths, and access procedures may change, so always follow the latest lab, NTU, or NSCC instructions.

6.1 Choosing A Resource

Use the smallest suitable resource for the task.

General guidance:

  • use your laptop for writing, lightweight analysis, coding, and small test runs;
  • use a lab workstation for interactive analysis, visualization, GPU work, or medium-sized jobs;
  • use NTU HPC for larger jobs, repeated batch analyses, and workloads that need more CPU, memory, or storage;
  • use Singapore NSCC for national-scale or collaboration-scale workloads that exceed lab or NTU resources.

Before running a large job, test the workflow on a small subset of data.

6.2 Lab Workstations

Lab workstations are shared computing machines managed by the lab. They may be used for data analysis, software development, visualization, machine learning, or instrument-related workflows.

6.2.1 Access

Before using a workstation, confirm:

  • whether you are allowed to use the machine;
  • login method and account information;
  • whether remote access is enabled;
  • storage location for your project;
  • whether GPU, memory, or local disk use is shared with others;
  • backup expectations.

Do not create accounts, install system software, change network settings, or modify shared configuration without permission.

6.2.2 Good Practice

When using a lab workstation:

  • keep project files in the agreed folder;
  • avoid storing important files only on the desktop or downloads folder;
  • do not fill local disks with temporary files;
  • clean up intermediate files after large analyses;
  • document software versions and commands;
  • avoid interrupting other users’ jobs;
  • ask before rebooting a shared machine.

If you need long-running jobs, discuss with the workstation owner or project team first.

6.3 NTU HPC

NTU HPC resources are suitable for larger computational jobs that require more CPU, memory, storage, or batch scheduling than a local workstation can provide.

Typical use cases:

  • large-scale omics data processing;
  • parameter sweeps;
  • repeated statistical analyses;
  • workflow pipelines;
  • jobs requiring many CPU cores or large memory;
  • analyses that should run without depending on a personal laptop.

6.3.1 Access And Accounts

Access requirements may depend on NTU status, PI approval, project allocation, and current HPC policy. New members should confirm:

  • whether they are eligible for an NTU HPC account;
  • which project or group allocation should be used;
  • login method and authentication;
  • storage locations and quotas;
  • available software modules or containers;
  • queue or partition rules;
  • job runtime and resource limits.

If you cannot log in or do not know which allocation to use, ask your mentor or the PI.

6.3.2 Batch Jobs

HPC systems usually run jobs through a scheduler. Do not run heavy jobs directly on login nodes.

Good practice:

  • test commands on a small input first;
  • request only the resources you need;
  • write clear job scripts;
  • save logs for troubleshooting;
  • monitor failed jobs;
  • clean temporary files;
  • document the exact command and environment used.

If a job fails, collect the job script, job ID, log file, and error message before asking for help.

6.4 Singapore NSCC

Singapore NSCC provides national high-performance computing resources. It may be useful for very large analyses, collaborative projects, GPU workloads, or workflows that exceed lab and NTU capacity.

Potential use cases:

  • large-scale multi-omics processing;
  • compute-intensive simulations;
  • machine-learning or deep-learning workloads;
  • large batch pipelines;
  • collaboration projects requiring national-scale computing.

6.4.1 Before Applying Or Using NSCC

Before using NSCC, discuss with the PI:

  • why lab workstation or NTU HPC is not sufficient;
  • expected computing resources;
  • storage requirements;
  • software environment;
  • project timeline;
  • data sensitivity and transfer plan;
  • collaborators who need access.

NSCC access and project allocation should be handled carefully because resource use, data movement, and account management may involve formal procedures.

6.5 Data Storage

Computing resources and data storage should be planned together.

Important questions:

  • where is the raw data stored;
  • where will intermediate files be written;
  • where will final results be saved;
  • how large are the files;
  • who needs access;
  • what needs backup;
  • what should be deleted after analysis.

Do not move sensitive or unpublished data to an external system without PI approval.

6.6 Software Environments

For reproducible computing, record the software environment used for each project.

Useful options include:

  • R package records such as renv;
  • Python environment.yml or requirements.txt;
  • Conda or Mamba environments;
  • containers when appropriate;
  • workflow-management tools for larger pipelines.

When using shared resources, avoid changing global software environments. Use project-specific environments whenever possible.

6.7 Asking For Help

When asking for help with computing resources, include:

  • which resource you are using;
  • your username or account type, if appropriate;
  • the project folder or working directory;
  • the command or job script;
  • the job ID, if using HPC;
  • the exact error message;
  • expected runtime, memory, CPU, or GPU needs;
  • what you already tried.

Clear information makes it much easier for lab members, IT support, or HPC support to help.