Unix Scripts - BlueHive CLI

Command-line tools for starting Slurm-aware SSH, VNC, and development environments from macOS.

Features

• Automatic Hostname Mapping: Supports bluehive, bluehive3, and bhward clusters with automatic hostname resolution
• Automatic SSH Config Setup: Cluster entry points resolve their own connection settings and export standard SSH aliases so terminals and apps can share the same OpenSSH connection
• First-run Setup: Prompts for the remote account and storage root, then optionally saves them
• Reusable Login Connection: Reuses an OpenSSH control master, so a running connection does not request the login password again
• Slurm-aware Remote Access: Starts VS Code/Cursor tunnels, Dropbear SSHD, or VNC inside separate Slurm jobs
• Container SSH and SFTP: blhc3 opens Fish inside the VNC job's Apptainer environment while preserving its Slurm cgroup
• Persistent Apptainer Environment: Gives each user a writable fakeroot sandbox with apt, MATLAB, CUDA, Chrome, ChatGPT, uv, pixi, and compilers
• Container Batch Jobs: sbatch job.sh from the container preserves the script's #SBATCH directives and runs the complete Bash script in the environment
• Self-deploying Remote Files: Copies missing tools and checksum-versioned VNC files under REMOTE_SHARED_ROOT
• Real-time Output: Shows job state, allocated node, startup stage, and failure logs

Supported Clusters

• bluehive: bluehive.circ.rochester.edu
• bluehive3: bluehive3.circ.rochester.edu
• bhward: bhward.circ.rochester.edu

All scripts automatically map cluster names to their full hostnames.

Installation and Setup

1. Clone Repository

For zsh:

git clone https://github.com/Sigurd-git/Unix-scripts.git
cd Unix-scripts
printf 'export PATH="%s:$PATH"\n' "$PWD" >> ~/.zshrc
source ~/.zshrc

For bash:

git clone https://github.com/Sigurd-git/Unix-scripts.git
cd Unix-scripts
printf 'export PATH="%s:$PATH"\n' "$PWD" >> ~/.bashrc
source ~/.bashrc

2. Start

./remote_vnc.sh

The first command asks for:

• the BlueHive username;
• a writable remote root, defaulting to /scratch/<username>;
• an optional password for the first SSH connection;
• whether to save the profile and whether to include the password.

Saved profiles use ~/.config/unix-scripts/config with mode 0600. Passwords are optional. When no password is saved, OpenSSH asks for it as needed. The fixed VNC SSH port is derived from the username and saved automatically. Existing user_password.txt files continue to work and take precedence, so current installations do not need migration.

After connecting, the scripts automatically maintain a marked cluster block at the start of ~/.ssh/config. Existing content is preserved and backed up before changes. ssh bluehive3, ssh blhc3, and ssh bluehive_compute3 then use the same ControlPath as the script's login and VNC connections. Apps using system OpenSSH can select these hosts and reuse the connection; clients with their own SSH implementation do not share OpenSSH control sockets. The script commands still connect with explicit options and do not require these aliases to exist.

To refresh aliases for an already-running VNC job without restarting it:

./sync_ssh_config.sh -a bluehive3

Usage

For first-time cluster setup, see ADMIN_INIT.md or ADMIN_INIT.en.md.

Command Line Interface

# Open a login-node shell without an SSH config entry
./cluster_ssh.sh --cluster bluehive3

# Start a VS Code tunnel on bluehive3 with default parameters
./tunnel.sh

# Start a VS Code tunnel on bhward with custom parameters
./tunnel.sh -a bhward -p doppelbock -c 16 -g 1 -m 256 -t 12

# Start a VS Code tunnel on bluehive3 explicitly
./tunnel.sh -a bluehive3 -p preempt -c 16 -g 1 -m 256 -t 12

# Start a Cursor tunnel instead of the default VS Code tunnel
./tunnel.sh -a bluehive3 --tool cursor -p doppelbock -c 16 -g 1 -m 256 -t 12

# Start a Dropbear SSHD job and save its managed connection state
./remote_sshd.sh -a bluehive3 -p doppelbock -c 16 -g 1 -m 256 -t 24

# Enter that Dropbear allocation
blhc3 --service sshd

# Deploy or repair remote tools manually
./deploy_remote_tools.sh -a bluehive3 --all

# Copy only the VNC release files; this does not build a SIF on the login node
./deploy_remote_tools.sh -a bluehive3 --vnc

# Start VNC, OpenCodex, Codex app-server, and a Slurm-bound SSH shell
./remote_vnc.sh -a bluehive3 -p doppelbock -c 8 -g 0 -m 64 -t 120

# Use a named persistent environment
./remote_vnc.sh --env analysis --restart

# Keep the original read-only VNC image
./remote_vnc.sh --immutable --restart

# Reuse the running VNC job without opening a viewer
./remote_vnc.sh --no-open

# Enter the running VNC job's Apptainer environment
blhc3

Remote VNC and Slurm-bound SSH

remote_vnc.sh starts an independent Slurm job containing an Apptainer XFCE desktop. The default default environment is a persistent writable sandbox. The job remains separate from the my_sshd job created by remote_sshd.sh.

The VNC job starts two SSH services:

• The Mac connects with blhc3 to a public-key-only sshd running inside the read-only Apptainer rootfs.
• The container and the Bluehive Host Terminal use a private localhost sshd to return to the compute host.

Both services inherit the VNC job's Slurm environment and batch cgroup. The container has its installed software, allocated GPU, /gpfs/fs1, /gpfs/fs2, /scratch, and SFTP. Run bluehive-host-shell for host programs and modules.

The sandbox maps the current user to root only inside the container when bh-env admin is running. It cannot change the host, read files the account cannot normally read, request unallocated GPUs, or bypass Slurm limits.

For a mutable environment, the job also starts OpenCodex and Codex app-server inside one job-scoped Apptainer service instance. The direct container SSH entry, VNC desktop, proxy, and daemon share the same persistent home and network. Host operations remain available through bluehive-host-shell.

Start or reuse VNC

Start with the defaults (8 CPUs, no GPU, 64 GiB, 120 hours):

./remote_vnc.sh

After the SSH and VNC checks pass, the script opens TigerVNC in full screen when it is installed under /Applications; otherwise it uses macOS Screen Sharing. Full screen lets TigerVNC forward the Mac Command key to the remote desktop automatically. The viewer asks the server to match the current monitor instead of scaling a larger framebuffer and uses lossless full-color Tight encoding for sharp text and interface elements. Keep the job and tunnel running without opening a viewer with:

./remote_vnc.sh --no-open

The initial framebuffer is 2560x1440. TigerVNC dynamically changes it to the current full-screen viewport after authentication. --geometry controls the initial size and remains the fallback for clients without remote-resize support. Select another fallback when submitting a new job, for example:

./remote_vnc.sh --geometry 2880x1800 --restart

An existing managed VNC job is reused. Resource arguments only affect a new job. Use --restart when resource values must change or when the script finds an older VNC job:

./remote_vnc.sh --restart -p doppelbock -c 8 -g 0 -m 32 -t 4

During startup, the script reports stages such as CHECKING_IMAGE, BUILDING_IMAGE, BUILDING_ENVIRONMENT, PROVISIONING_ENVIRONMENT, STARTING_VNC, STARTING_SSH, and READY. The first mutable launch installs the development environment and can take substantially longer than later launches. Existing valid generations are reused.

macOS-style desktop and clipboard

The VNC session installs a checksum-pinned WhiteSur light theme into the persistent container home, keeps the existing XFCE panel launchers, enlarges the lower launcher panel, moves the window controls to the left, and applies an included high-resolution background. If the theme download is unavailable, the session still starts with Greybird. The profile is applied once per profile version so later user customization is retained. Reapply it from a VNC terminal with:

remote-vnc-desktop-profile apply --force

TigerVNC normally maps the left Mac Command key to X11 Alt and the right Command key to Super. The VNC server normalizes the incoming left Command key to Super, then the desktop profile adds an application-aware bridge for common macOS shortcuts. In Chrome, ChatGPT, MATLAB, Thunar, and other GUI applications, Command+C/V/A/X/Z/Shift+Z/F/S/O/N/T/W/R/L/P invoke the corresponding Linux Control shortcut. In XFCE Terminal, Command+C/V/A/F/N/T/W instead invoke Control+Shift+C/V/A/F/N/T/W. Actions such as Command+Z and Command+S stay inactive in terminals so they cannot suspend a foreground process or enable terminal flow control.

The existing desktop shortcuts remain available: Command+Space opens the application finder, Command+Tab switches windows, Command+M minimizes, Command+Arrow maximizes or tiles, Command+Return opens a terminal, and Command+Shift+3 or 4 takes screenshots. macOS reserves some system shortcuts, including Command+Tab and Command+Space. The launcher opens TigerVNC in full screen with system-key forwarding enabled, so the Command key is captured automatically. Press Control+Option to return those shortcuts to macOS; press Control+Option+G to capture them again. The window title includes keyboard grabbed while capture is active. The first capture may require enabling TigerVNC in System Settings → Privacy & Security → Accessibility.

TigerVNC uses RFB 3.8 with VncAuth and listens only on compute-node loopback. The script forwards it through the existing public-key SSH connection. TigerVNC is the recommended Mac client because its text clipboard works with this Linux server and it forwards Command as Super. The launcher enables both clipboard directions, fits the remote framebuffer to the current monitor, uses lossless full-color encoding, and never reads the saved VNC password. Lossless mode consumes more bandwidth than JPEG mode but avoids compression blur. File and image clipboard formats are outside the standard VNC text clipboard protocol.

Remote root and file layout

The script reads REMOTE_SHARED_ROOT from the saved remote profile. The root is selected in this order:

1. -r PATH or --root PATH
2. REMOTE_SHARED_ROOT from the saved profile
3. /scratch/<username> for a new profile
4. /scratch/snormanh_lab/shared for a legacy profile with a blank third line

The profile stores the fixed container SSH port. A newly allocated VNC job reuses that port, so clients no longer need a port update after every job restart. Missing values are derived consistently from the remote username. The external SSH server host key is generated once under the private remote user directory and reused by later Slurm jobs. Each client verifies and stores that key on its first launch; restarting a job on the same node no longer changes the SSH port or server identity.

The first VNC launch also creates a dedicated client key at ~/.local/share/unix-scripts/id_ed25519. It does not depend on an existing personal SSH key. Set REMOTE_VNC_IDENTITY_FILE to use another identity.

Override it for one run with:

./remote_vnc.sh --root /path/you/can/write --no-open

The writable root contains immutable release files and private per-user state:

${REMOTE_SHARED_ROOT}/remote-vnc/
├── releases/<bundle_sha256>/
│   ├── remote_vnc_job.sh
│   ├── start_vnc.sh
│   ├── configure_desktop.sh
│   ├── bluehive-aurora.svg
│   ├── start_opencodex.sh
│   ├── build_vnc_image.sh
│   ├── prepare_environment.sh
│   ├── provision_environment.sh
│   ├── environment_common.sh
│   ├── bh-env.sh
│   ├── environment-packages.txt
│   ├── matlab-products.txt
│   ├── ubuntu-vnc-xfce-g3_24.04.def
│   └── bundle.sha256
└── users/${USER}/
    ├── state/
    │   ├── connection.env
    │   ├── vnc-password.txt
    │   ├── remote-ssh/
    │   │   ├── server-key
    │   │   └── server-key.pub
    │   └── jobs/<slurm_job_id>/
    ├── logs/
    ├── images/
    ├── bin/bh-env
    └── environments/<name>/
        ├── current -> generations/<generation>/
        ├── generations/<generation>/rootfs/
        ├── home/
        └── checkpoints/

The release directory name is the SHA-256 of bundle.sha256. Existing releases are reused only when the manifest hash, every listed file hash, and the file list all match. A mismatched directory is reported and left unchanged.

User directories use mode 0700. VNC passwords, SSH keys, logs, and job state are not stored in another user's directory.

deploy_remote_tools.sh --vnc and --all copy release and build files but do not build a SIF on the login node.

Image selection and automatic build

The preferred read-only image is:

/scratch/snormanh_lab/shared/remote-vnc/images/ubuntu-vnc-xfce-g3_24.04.sif

The job selects an image in this order:

1. Use the shared SIF when it is readable and its SHA-256 is 2848904237e27cba487a14af1613d40ea4ab02d3d3808d36daa49cb831a3ffaf.
2. Use the private SIF under users/${USER}/images/ when its sidecar checksum and runtime checks pass.
3. Build a private SIF inside the current VNC Slurm job.

The Apptainer definition pins the Linux amd64 source manifest instead of the mutable 24.04 tag. Base-image build cache and temporary data stay in SLURM_TMPDIR; a per-user flock prevents concurrent jobs from building the same image. The image preparation deadline is 30 minutes. Git stores the definition and checksums, while .gitignore excludes SIF binaries.

The selected SIF is the base for the named writable sandbox. Environment build files stay under the private user directory because the complete MATLAB and CUDA installation can exceed a compute node's local temporary storage.

Use a different shared image path for another cluster or to test the private build path:

REMOTE_VNC_SHARED_IMAGE=/path/to/shared-vnc.sif ./remote_vnc.sh --no-open

Set the VNC password

The first run under a new remote root creates an eight-character password and prints its file path. The script never prints the password itself. To replace it, log in to the cluster, copy the printed path into password_file, and enter the new password without terminal echo:

ssh -t bluehive3 bash
password_file=/path/printed/by/remote_vnc.sh
umask 077
read -rsp "New VNC password: " vnc_password
printf '\n'
printf '%s\n' "$vnc_password" > "$password_file"
chmod 600 "$password_file"
unset vnc_password
exit

Use eight characters or fewer because VNC authentication only uses the first eight. Restart the VNC job so it regenerates TigerVNC's password file:

./remote_vnc.sh --restart

Compute shell and OpenCodex

After startup, connect directly to the mutable Apptainer environment:

blhc3

The login shell is Fish. Return to a Bash shell on the allocated compute host when modules or host-only programs are needed:

bluehive-host-shell

The host backchannel, container sshd, and their child processes stay in the same Slurm batch cgroup. SFTP also opens the persistent container home at /home/$USER.

The VNC desktop itself already runs in this environment. Its desktop contains launchers for an environment terminal, an admin terminal, Google Chrome, ChatGPT, and MATLAB R2025b. Bluehive Host Terminal remains available for host modules and programs. The environment terminal and MATLAB launchers always enter the current environment generation.

The environment includes Ubuntu development tools, Fish, GCC/G++, GFortran, CMake, Ninja, Git, SSH, screen, tmux, Node.js, npm, OpenCodex 2.39.0, Codex CLI 0.150.1, uv, pixi, CUDA 12.5 development components, Google Chrome, the ChatGPT Linux desktop app, VNC/XFCE, MathWorks Package Manager (mpm), Noto CJK fonts for Chinese text, and MATLAB R2025b with:

• Signal Processing Toolbox
• Statistics and Machine Learning Toolbox
• Parallel Computing Toolbox
• Image Processing Toolbox
• Audio Toolbox
• DSP System Toolbox
• Optimization Toolbox
• Curve Fitting Toolbox
• Deep Learning Toolbox

The CUDA toolkit comes from the container. Apptainer --nv injects the driver and allocated GPU devices from the compute host. MATLAB uses the existing BlueHive network license file through its /gpfs/fs1 mount.

Install software and preserve changes

The SSH rootfs stays read-only. Open a short-lived writable fakeroot shell when system packages need to change:

bh-admin
apt-get update
apt-get install -y ffmpeg
exit
bh-env shell

Or run one command directly:

bh-admin -- apt-get install -y ffmpeg

bh-admin proxies to the host-side bh-env admin, holds the environment's mutation lock only for that command or shell, and releases it on exit. The familiar bh-env command is also proxied through the same backchannel.

bh-env shell opens a fresh container mount, so it sees packages installed by the preceding admin shell immediately. bh-env exec -- COMMAND, new batch jobs, and new Environment Terminal launchers do the same. Use ./remote_vnc.sh --restart when the base blhc3 shell, the existing VNC desktop, OpenCodex, and Codex app-server should all reload the changed rootfs.

Changes to the sandbox and its home directory persist across VNC jobs. Use a named environment when separate software stacks are useful:

./remote_vnc.sh --env experiment --restart
blhc3

--immutable starts the prior read-only SIF workflow, skips sandbox creation, and does not start the container OpenCodex service. Changing environment name or mode for an active VNC allocation requires --restart.

Submit batch scripts in the environment

From blhc3, submit a script directly:

sbatch analysis.sh

The generated submission wrapper copies every #SBATCH directive from analysis.sh. Slurm allocates the resources and establishes the working directory, then /bin/bash analysis.sh runs inside the current environment. Arguments are forwarded:

sbatch analysis.sh subject-01 --overwrite

The container sbatch command maps scripts under its persistent home, /bluehive-home, /gpfs, /scratch, or /host back to the corresponding host path, then calls bh-env --env NAME sbatch .... Use the explicit bh-env --env NAME sbatch ... proxy for another prepared environment.

Checkpoint, restore, and rebuild

These commands perform Apptainer image operations and must run inside a Slurm allocation:

# Save the current rootfs before a risky change
bh-env checkpoint before-upgrade

# List the checkpoint path, then restore it as a new current generation
bh-env restore 20260903T120000Z-before-upgrade.sif

# Build a clean generation from the verified VNC base image
bh-env rebuild

# Show versions, paths, and the selected generation
bh-env status

restore and rebuild switch the current symlink atomically. Proxied bh-env shells and new batch jobs use the new generation immediately. Restart the VNC job to move its long-running container sshd, desktop, and services to that generation. Old generations remain in the private environment directory for manual recovery.

Check that the shell is in the expected Slurm job and can see host resources:

blhc3 'printf "job=%s node=%s env=%s\n" "$SLURM_JOB_ID" "$(hostname -s)" "$BH_ENV_NAME"; cat /proc/self/cgroup; command -v matlab; nvidia-smi -L'
blhc3 "bluehive-host-shell 'type module; cat /proc/self/cgroup'"

OpenCodex and Codex app-server start automatically with every mutable VNC job. Check them directly from the container SSH shell:

blhc3 'ocx ready --json'
blhc3 'codex app-server daemon version'

The same managed SSH connection exposes the OpenCodex dashboard on the Mac:

http://127.0.0.1:10102

The job SSH server permits forwarding only to the active loopback VNC and OpenCodex ports. remote_vnc.sh checks the dashboard over this tunnel before reporting startup success.

The VNC desktop and service instance use the same sandbox and persistent HOME. The separate instance keeps the long-lived OpenCodex and app-server processes in one PID namespace. Direct container SSH sessions share their network and HOME, while host-side wrappers can still join the service instance.

The first mutable launch copies the existing host configuration into that environment's private persistent home. This includes the current OpenCodex configuration and authentication files, Codex config.toml, authentication, catalog settings, AGENTS.md, personal skills, plugin files, memories, and imported skill sources. Existing destination files are preserved. OpenCodex routing, Codex authentication, packages, logs, and app-server sockets remain private to the environment.

BlueHive $HOME/.codex is the canonical Codex session store. Every mutable or immutable container mounts the host sessions, archived_sessions, attachments, plans, shell_snapshots, visualizations, shared writer locks, history, and session index into its private CODEX_HOME, while CODEX_SQLITE_HOME points to the host state directory. New and updated sessions therefore appear immediately from both the host and container without copying. Existing container-only session files are hidden and are no longer used. The host terminal continues to use $HOME/.codex directly.

Startup verifies the Apptainer instance, OpenCodex proxy, and Codex app-server PIDs against the VNC job's batch cgroup before marking the job ready. Service state and logs are in users/${USER}/state/jobs/<slurm_job_id>/opencodex/. If either service exits, the managed VNC job fails and records the reason there.

Automatic Illustrator Bundle Sync

sync_ai_bundles.sh checks the BlueHive toydata/tmp directory for top-level folders containing an Illustrator file. For each new folder, it creates a two-way-safe Mutagen session with the matching folder under ~/Downloads. The existing final_paper_ai_linked sessions remain separate.

The launch agent runs the check every five minutes:

mkdir -p "$HOME/Library/Application Support/PaperAISync"
install -m 755 sync_ai_bundles.sh \
  "$HOME/Library/Application Support/PaperAISync/sync_ai_bundles.sh"
launchctl bootstrap "gui/$(id -u)" \
  "$HOME/Library/LaunchAgents/com.gliao2.paper-ai-bundles.plist"

Run an immediate check with:

./sync_ai_bundles.sh

Parameters

The Slurm launch scripts share these resource options:

• -a, --cluster CLUSTER: bluehive, bluehive3, or bhward
• -p, --partition PARTITION: Slurm partition
• -c, --cpus CPUS: CPU cores
• -g, --gpus GPUS: GPUs; use 0 for CPU-only jobs
• -m, --memory GIB: memory in GiB
• -t, --time HOURS: time limit in hours
• -w, --node NODE: request a specific compute node
• -r, --root PATH: override REMOTE_SHARED_ROOT

remote_vnc.sh also accepts:

• --env NAME: select or create a persistent environment; default default
• --geometry WIDTHxHEIGHT: set the initial/fallback VNC framebuffer; default 2560x1440; TigerVNC dynamically matches its full-screen viewport
• --immutable: run the original read-only image without preparing a sandbox
• --restart: replace the current VNC job
• --no-open: do not open a VNC viewer

tunnel.sh also accepts --tool code|cursor and -n to disable logging. Run any script with --help for its current defaults.

Security Features

• Saved profiles use mode 0600; saving the password is optional.
• Cluster SSH connections use explicit parameters. Generated SSH aliases share these settings with other OpenSSH clients.
• VNC listens on compute-node loopback and reaches the Mac through an SSH local forward.
• TigerVNC keeps connection blacklisting enabled and requires VncAuth; its send/receive text clipboard parameters are checked before a job is marked ready.
• The mutable VNC job's external SSH service runs inside the read-only Apptainer rootfs, accepts only the configured public key, and enables SFTP.
• Each user's VNC state directory uses mode 0700; the VNC password file uses mode 0600.
• The external SSH host key persists in each user's mode 0700 state directory; its private file uses mode 0600, and the key pair is checked before use.
• The Mac checks the persistent SSH host key before storing the VNC connection state.
• The script rejects a container SSH shell, host backchannel, OpenCodex proxy, or Codex app-server process outside the expected Slurm batch cgroup.
• Shared and private SIF files are checked before execution.
• Mutable environments and checkpoints remain under each user's mode 0700 directory.
• Fakeroot applies only to the Apptainer sandbox and retains the user's host permissions.