In [ ]:
from IPython.core.display import HTML

def _set_css_style(css_file_path):
   """
   Read the custom CSS file and load it into Jupyter.
   Pass the file path to the CSS file.
   """

   styles = open(css_file_path, "r").read()
   s = '<style>%s</style>' % styles     
   return HTML(s)

_set_css_style('rise.css')

File systems and the command line¶

print view
notebook

  • Distinguish absolute and relative paths
  • Identify your home directory on macOS, Windows, and Linux
  • Use core shell commands (pwd, ls, cd, mkdir, cp, mv, rm) to navigate and manage files
  • Apply a sensible directory layout for research projects
  • Execute shell commands inside a Jupyter notebook using ! or %%bash

About file systems¶

  • A hierarchical “tree” of directories and files beginning at the root (/ or a drive letter)
  • Each file has a path that tells the computer how to reach it
  • File extensions hint at type, but the OS ultimately stores bytes -- be explicit

Home directories across operating systems¶

Platform Typical home path Example Notes
macOS / Linux /Users/<user> or /home/<user> /Users/alice Forward‐slash / separator
Windows 10+ C:\Users\<user> C:\Users\Bob Back‑slash \ separator
HPC cluster /home/<user> /home/ab1234 Access via SSH

Within WSL, main Windows folders are reached through /mnt/c/<path>, assuming the default C drive

Absolute vs relative paths¶

  • Absolute – always starts from the root or drive: /Users/alice/course-filesystem-lab/data/README.md
  • Relative – interpreted from your current working directory: data/README.md
  • . = this directory  .. = parent directory
In [ ]:
# Where am I?  (print working directory)
!pwd
In [ ]:
%%bash
# Doing the same thing with magic commands
pwd
In [ ]:
# Note that %% commands need to be at the top!
%%bash
pwd
In [ ]:
# What is here?  (list files, including hidden dotfiles)
!ls -a

Command‑line navigation¶

Try these yourself in the notebook or an external terminal. Use TAB completion!

Action Command Handy options
Show location pwd –
List directory ls -l long, -a all, -h human‑readable sizes
Change directory cd <path> cd ~, cd ..
Make directory mkdir <name> -p create parents
Copy / move cp, mv -i interactive prompt
Delete rm, rm -r -i, never rm -rf / 😱
In [ ]:
%%bash
# Create an example project tree and explore it
mkdir -p demo_project/data demo_project/results
echo "example" > demo_project/data/dummy.txt
cd demo_project
pwd
echo "--- contents ---"
ls -R
cd ..

A reusable project layout¶

my-analysis/
├── data/        ← raw & intermediate datasets (never edit raw)
├── notebooks/   ← exploratory notebooks
├── scripts/     ← reusable Python/bash scripts
├── results/     ← figures & tables generated by scripts/notebooks
└── env/         ← environment & dependency files

Keeping code, raw data, and generated results separate makes projects reproducible (more on this later)

Activity¶

  1. In your home directory, create a folder course-filesystem-lab
  2. Inside it, make the sub‑folders data/, interim/, and results/
  3. Download the file http://mscbio2025-2025.github.io/files/Spellman.csv into data/
  4. Copy it into interim/ preserving the original
  5. Use what you’ve learned (pwd, ls, cp, etc.) to verify the layout

Remote file systems, data transfer, and permissions¶

What is SSH?¶

  • Secure Shell protocol for encrypted remote login
  • Same file system layout as local Linux/macOS machines
ssh <username>@<hostname>

Generating and using SSH keys (could differ on different servers)¶

SSH keys unable passwordless login

# Generate a new Ed25519 key‑pair
ssh-keygen -t ed25519 -C "student@laptop"

# Copy the public key to the cluster
ssh-copy-id student@hpc.example.edu

Copying files – scp¶

  • Syntax: scp [options] source destination
  • Copies entire files; no delta‑transfer
  • Flags: -r recursive, -p preserve times/permissions, -i identity file, -C compression

Examples:

# Copy local notebook to remote home dir
scp analysis.ipynb student@hpc.example.edu:~/

# Copy an entire folder from remote to local
scp -r student@hpc.example.edu:~/results ./remote_results

Synchronizing efficiently – rsync¶

  • Transfers only changed blocks
  • Retains permissions; can delete extraneous files (--delete)
  • Works over SSH with -e ssh

Examples:

# Push local results → remote
rsync -avh --progress results/ student@hpc.example.edu:project/results/

# Mirror remote scratch directory locally
rsync -avzh --delete student@hpc.example.edu:scratch/run42/ ./run42/

File permissions¶

  • Three roles: user (u), group (g), other (o)
  • Three bits: read (r), write (w), execute (x)
  • Symbolic form: drwxr-x---
  • Numeric (octal) form: 750

Useful commands:

ls -l file.txt
chmod u+x script.sh      # add execute for user
chmod g-w notes.txt      # remove write for group
chmod 640 data.tsv       # rw for user, r for group, – for others

More on the command line¶

du - disk usage of files/directores

In [ ]:
%%bash
du -s         # -s option to "summarize" usage for all files  
du -sh        # -h option for "human-readable" format  
du -sh files  # check usage in a particular directory

df - usage of full disk

In [ ]:
%%bash
df -h .

Finding files, directories, and information¶

locate find a file system wide
find search directory tree
which print location of a command
man print manual page of a command

In [ ]:
%%bash
man find

Command line history¶

history show commands previously issued

up arrow cycle through previous commands

Ctrl-R search through history for command

.bash_history file that stores the history

Other useful shortcuts¶

tab -- autocomplete

Ctrl-D -- EOF/logout/exit

Ctrl-A -- go to beginning of line

Ctrl-E -- go to end of line

alias new=cmd -- make a nickname for a command (from the command line directly, not in Jupyter)

Further resources¶

  • Software‑Carpentry Unix Shell lesson
  • Printable cheat‑sheet: “Twenty most useful Unix commands” (link in Canvas).
  • Optional: install Oh My Zsh (macOS/Linux) or Git Bash (Windows) for improved prompts.

Next time¶

  • Environment variables
  • Simple shell scripts
  • Text manipulation on the command line