Hello, Candle on AWS!

Prerequisites

A Github Account
An AWS Account

Create Candle-EC2 IAM User

From AWS IAM Console >> Users >> Create User "Candle-EC2" >> Attach Policies Directly:

AmazonS3FullAccess

Launch GPU Enabled AWS EC2 Compute Instance

Open the AWS EC2 Console
Configure a Ubuntu 20.04 Deep Learning Base GPU AMI image + G5.##xlarge instance type (see here for charges and instance capcity)
Create new key pair and save .pem file
Allow SSH traffic
Launch Instance
From EC2 landing page >> Start Instance

Gotchas

You may need to request capacity increase from AWS to handle larger G5.##xlarge instances
Move .pem file from downloads to ~/.ssh using mv ~/Downloads/candle-key.pem ~/.ssh

Connect to EC2 via SSH

Install Remote - SSH from VSCode Extensions
Add new SSH Connection
From EC2 landing page >> Connect >> SSH client >> Copy commands to VSCode

# NB: edit path to .pem file as needed
chmod 400 ~/.ssh/candle-key.pem
ssh -i "~/.ssh/candle-key.pem" ubuntu@ec2-##-##.us-east-1.compute.amazonaws.com

Update .config file and validate format as follows

Host ec2-##-###.us-east-1.compute.amazonaws.com
  HostName ec2-##-###.us-east-1.compute.amazonaws.com
  IdentityFile ~/.ssh/candle-key.pem
  User ubuntu

Confirm remote host platform (Linux) and fingerprint (Continue)
Launch terminal on remote host

Gotchas

If you stop and restart your EC2 instance, you will need to update the IP address in your .config file

Verify EC2 CUDA/cuDNN

nvidia-smi --query-gpu=compute_cap --format=csv
nvcc --version

whereis cudnn.h

Install Rust via rustup

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable

# Set path
source "$HOME/.cargo/env"

Install AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Check install
aws --version

Configure Candle-EC2 IAM User

IAM Console >> Users >> Candle-EC2 >> Security Credentials >> Create Access Key "EC2-CLI"
From EC2 SSH terminal

aws configure
# Copy-Paste Candle-EC2 Access Key ID
# Copy-Paste Candle-EC2 Secret Access Key
# Default region name: us-east-1

Configure Candle

git clone https://github.com/huggingface/candle.git
cd candle

Build & Run Binaries

See all Candle example models here

Example:

# CPU build
cargo build --example falcon --release

# CUDA + cuDNN build
cargo build --example falcon --features cuda,cudnn --release

# Run binary
cd target/release/examples
./falcon --prompt "who invented the lightbulb"

Store Binaries in S3

From AWS S3 Console >> Create S3 Bucket i.e my-candle-binaries

# Copy model binary from EC2 to S3 
cd target/release/examples
aws s3 cp quantized s3://my-candle-binaries

⚠️ IMPORTANT: Terminate any AWS resources to prevent unexpected charges.

Candle Cookbook