Skip to main content

Shen Lab Handbook

4 Python

Python is another major programming language used in Shen Lab. We use Python for data processing, automation, machine learning, bioinformatics workflows, API access, visualization, and building reproducible analysis pipelines. This chapter introduces the recommended setup for Python, editors, environments, and daily project practice.

4.1 What To Install

New members who will use Python should install:

  • Python;
  • a Python environment manager;
  • Visual Studio Code or another suitable editor;
  • JupyterLab or Jupyter Notebook, when interactive analysis is useful;
  • Git, so code can be version controlled and shared;
  • common scientific Python packages.

For most lab members, the recommended approach is to use a managed Python environment instead of installing packages into the system Python.

4.2 Installing Python

Python can be installed in several ways. The best choice depends on your operating system, project needs, and whether you are working locally or on a lab server.

4.2.2 macOS

Recommended options:

  • install Miniforge or Miniconda for scientific Python environments;
  • or install Python through Homebrew if you are comfortable using the command line.

After installation, open Terminal and check:

python3 --version

If you use Conda or Miniforge, also check:

conda --version

4.2.3 Windows

Recommended options:

  • install Miniforge or Miniconda;
  • or install Python from the official Python website.

During installation, make sure Python is available from the command line if the installer provides that option.

After installation, open PowerShell and check:

python --version

or:

py --version

4.2.4 Linux

On Linux workstations or personal computers, Python may already be installed. You can check:

python3 --version

On shared servers, do not change the system Python. Use a project environment, Conda environment, module system, or instructions provided by the server administrator.

4.3 Environment Management

Python projects should use isolated environments. This prevents package conflicts and makes analyses easier to reproduce.

Recommended tools include:

  • conda or mamba for scientific environments;
  • venv for lightweight Python-only environments;
  • pip for installing Python packages;
  • requirements.txt, environment.yml, or another lock file to record dependencies.

4.3.1 Conda Environment

Example:

conda create -n project-name python=3.11
conda activate project-name

Install packages:

conda install numpy pandas scipy matplotlib seaborn scikit-learn jupyterlab

Export the environment:

conda env export > environment.yml

4.3.2 venv Environment

Example:

python3 -m venv .venv
source .venv/bin/activate

On Windows PowerShell:

.venv\Scripts\Activate.ps1

Install packages:

pip install numpy pandas scipy matplotlib seaborn scikit-learn jupyterlab

Record dependencies:

pip freeze > requirements.txt

4.4 Editors And Notebooks

4.4.1 Visual Studio Code

Visual Studio Code is a good default editor for Python projects. Recommended extensions include:

  • Python;
  • Jupyter;
  • GitLens, if you want more Git integration;
  • Ruff or another linter/formatter, if used by the project.

After opening a project folder in VS Code, select the correct Python interpreter or environment. This is important because different projects may use different environments.

4.4.2 JupyterLab

JupyterLab is useful for exploration, visualization, and interactive analysis. It should be used carefully for reproducible work.

Good practice:

  • use notebooks for exploration and communication;
  • move stable workflows into scripts or packages when possible;
  • restart the kernel and run all cells before sharing a notebook;
  • avoid hidden state from out-of-order execution;
  • keep large outputs out of version control unless they are needed.

Start JupyterLab:

jupyter lab

4.5 Common Packages

Commonly used Python packages include:

  • numpy for numerical computing;
  • pandas for tabular data;
  • scipy for scientific computing;
  • matplotlib and seaborn for visualization;
  • scikit-learn for machine learning;
  • statsmodels for statistical modeling;
  • jupyterlab for notebooks;
  • pyarrow for efficient table formats;
  • openpyxl for Excel files;
  • requests for APIs;
  • pytest for testing.

Project-specific omics or mass spectrometry analyses may require additional packages. Follow the project README or ask your mentor before installing many packages into an existing environment.

4.6 Project Organization

Use one folder per project and keep code, data, results, and documentation organized.

Example structure:

project-name/
  data/
  notebooks/
  scripts/
  src/
  results/
  figures/
  docs/
  environment.yml
  README.md

Good practice:

  • keep raw data unchanged;
  • place reusable code in src/ or a package-like structure;
  • use scripts/ for command-line workflows;
  • use notebooks/ for exploration;
  • write a README explaining how to run the analysis;
  • save results and figures in clear output folders;
  • avoid absolute paths that only work on your computer.

4.7 Working With Data

When using Python for lab data analysis:

  • preserve raw data;
  • document each processing step;
  • keep sample metadata close to the analysis;
  • check missing values, duplicates, and inconsistent identifiers;
  • make plots from code rather than manual editing;
  • save processed data only when the processing script is also saved;
  • keep sensitive or unpublished data in approved storage locations.

For omics data, pay careful attention to feature identifiers, sample names, batch variables, normalization, quality control, and file provenance.

4.8 Reproducibility

Python analyses should be rerunnable by another lab member.

Recommended habits:

  • use a project-specific environment;
  • record dependencies in environment.yml or requirements.txt;
  • set random seeds when using stochastic methods;
  • use relative paths within the project;
  • write functions for repeated steps;
  • keep parameters in code or configuration files;
  • restart the Python session and rerun the analysis before sharing final results;
  • include enough comments for non-obvious analysis choices.

For important analyses, consider adding simple tests or checks, especially when code is reused across projects.

4.9 Getting Help

If you encounter a Python problem, collect:

  • your operating system;
  • Python version;
  • environment name and package versions;
  • the exact error message;
  • the command or code that produced the error;
  • a small example that reproduces the problem, if possible;
  • what you already tried.

Then ask your mentor, a relevant lab member, or the PI. A clear and reproducible error report saves everyone time.