On this tutorial, we implement a Colab-ready model of the AutoResearch framework originally proposed by Andrej Karpathy. We construct an automatic experimentation pipeline that clones the AutoResearch repository, prepares a light-weight coaching setting, and runs a baseline experiment to ascertain preliminary efficiency metrics. We then create an automatic analysis loop that programmatically edits the hyperparameters in prepare.py, runs new coaching iterations, evaluates the ensuing mannequin utilizing the validation bits-per-byte metric, and logs each experiment in a structured outcomes desk. By operating this workflow in Google Colab, we display how we will reproduce the core thought of autonomous machine studying analysis: iteratively modifying coaching configurations, evaluating efficiency, and preserving the perfect configurations, with out requiring specialised {hardware} or complicated infrastructure.
import os, sys, subprocess, json, re, random, shutil, time
from pathlib import Path
def pip_install(pkg):
subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", pkg])
for pkg in [
"numpy","pandas","pyarrow","requests",
"rustbpe","tiktoken","openai"
]:
strive:
__import__(pkg)
besides:
pip_install(pkg)
import pandas as pd
if not Path("autoresearch").exists():
subprocess.run(["git","clone","https://github.com/karpathy/autoresearch.git"])
os.chdir("autoresearch")
OPENAI_API_KEY=None
strive:
from google.colab import userdata
OPENAI_API_KEY = userdata.get("OPENAI_API_KEY")
besides:
OPENAI_API_KEY=os.environ.get("OPENAI_API_KEY")
if OPENAI_API_KEY:
os.environ["OPENAI_API_KEY"]=OPENAI_API_KEYWe start by importing the core Python libraries required for the automated analysis workflow. We set up all crucial dependencies and clone the autoresearch repository immediately from GitHub, making certain the setting consists of the unique coaching framework. We additionally configure entry to the OpenAI API key, if obtainable, permitting the system to optionally help LLM-assisted experimentation later within the pipeline.
prepare_path=Path("put together.py")
train_path=Path("prepare.py")
program_path=Path("program.md")
prepare_text=prepare_path.read_text()
train_text=train_path.read_text()
prepare_text=re.sub(r"MAX_SEQ_LEN = d+","MAX_SEQ_LEN = 512",prepare_text)
prepare_text=re.sub(r"TIME_BUDGET = d+","TIME_BUDGET = 120",prepare_text)
prepare_text=re.sub(r"EVAL_TOKENS = .*","EVAL_TOKENS = 4 * 65536",prepare_text)
train_text=re.sub(r"DEPTH = d+","DEPTH = 4",train_text)
train_text=re.sub(r"DEVICE_BATCH_SIZE = d+","DEVICE_BATCH_SIZE = 16",train_text)
train_text=re.sub(r"TOTAL_BATCH_SIZE = .*","TOTAL_BATCH_SIZE = 2**17",train_text)
train_text=re.sub(r'WINDOW_PATTERN = "SSSL"','WINDOW_PATTERN = "L"',train_text)
prepare_path.write_text(prepare_text)
train_path.write_text(train_text)
program_path.write_text("""
Aim:
Run autonomous analysis loop on Google Colab.
Guidelines:
Solely modify prepare.py hyperparameters.
Metric:
Decrease val_bpb is healthier.
""")
subprocess.run(["python","prepare.py","--num-shards","4","--download-workers","2"])We modify key configuration parameters contained in the repository to make the coaching workflow suitable with Google Colab {hardware}. We cut back the context size, coaching time finances, and analysis token counts so the experiments run inside restricted GPU sources. After making use of these patches, we put together the dataset shards required for coaching in order that the mannequin can instantly start experiments.
subprocess.run("python prepare.py > baseline.log 2>&1",shell=True)
def parse_run_log(log_path):
textual content=Path(log_path).read_text(errors="ignore")
def discover(p):
m=re.search(p,textual content,re.MULTILINE)
return float(m.group(1)) if m else None
return {
"val_bpb":discover(r"^val_bpb:s*([0-9.]+)"),
"training_seconds":discover(r"^training_seconds:s*([0-9.]+)"),
"peak_vram_mb":discover(r"^peak_vram_mb:s*([0-9.]+)"),
"num_steps":discover(r"^num_steps:s*([0-9.]+)")
}
baseline=parse_run_log("baseline.log")
results_path=Path("outcomes.tsv")
rows=[{
"commit":"baseline",
"val_bpb":baseline["val_bpb"] if baseline["val_bpb"] else 0,
"memory_gb":spherical((baseline["peak_vram_mb"] or 0)/1024,1),
"standing":"maintain",
"description":"baseline"
}]
pd.DataFrame(rows).to_csv(results_path,sep="t",index=False)
print("Baseline:",baseline)We execute the baseline coaching run to ascertain an preliminary efficiency reference for the mannequin. We implement a log-parsing perform that extracts key coaching metrics, together with validation bits-per-byte, coaching time, GPU reminiscence utilization, and optimization steps. We then retailer these baseline ends in a structured experiment desk so that every one future experiments might be in contrast in opposition to this beginning configuration.
TRAIN_FILE=Path("prepare.py")
BACKUP_FILE=Path("prepare.base.py")
if not BACKUP_FILE.exists():
shutil.copy2(TRAIN_FILE,BACKUP_FILE)
HP_KEYS=[
"WINDOW_PATTERN",
"TOTAL_BATCH_SIZE",
"EMBEDDING_LR",
"UNEMBEDDING_LR",
"MATRIX_LR",
"SCALAR_LR",
"WEIGHT_DECAY",
"ADAM_BETAS",
"WARMUP_RATIO",
"WARMDOWN_RATIO",
"FINAL_LR_FRAC",
"DEPTH",
"DEVICE_BATCH_SIZE"
]
def read_text(path):
return Path(path).read_text()
def write_text(path,textual content):
Path(path).write_text(textual content)
def extract_hparams(textual content):
vals={}
for ok in HP_KEYS:
m=re.search(rf"^{ok}s*=s*(.+?)$",textual content,re.MULTILINE)
if m:
vals[k]=m.group(1).strip()
return vals
def set_hparam(textual content,key,worth):
return re.sub(rf"^{key}s*=.*$",f"{key} = {worth}",textual content,flags=re.MULTILINE)
base_text=read_text(BACKUP_FILE)
base_hparams=extract_hparams(base_text)
SEARCH_SPACE={
"WINDOW_PATTERN":['"L"','"SSSL"'],
"TOTAL_BATCH_SIZE":["2**16","2**17","2**18"],
"EMBEDDING_LR":["0.2","0.4","0.6"],
"MATRIX_LR":["0.01","0.02","0.04"],
"SCALAR_LR":["0.3","0.5","0.7"],
"WEIGHT_DECAY":["0.05","0.1","0.2"],
"ADAM_BETAS":["(0.8,0.95)","(0.9,0.95)"],
"WARMUP_RATIO":["0.0","0.05","0.1"],
"WARMDOWN_RATIO":["0.3","0.5","0.7"],
"FINAL_LR_FRAC":["0.0","0.05"],
"DEPTH":["3","4","5","6"],
"DEVICE_BATCH_SIZE":["8","12","16","24"]
}
def sample_candidate():
keys=random.pattern(record(SEARCH_SPACE.keys()),random.selection([2,3,4]))
cand=dict(base_hparams)
adjustments={}
for ok in keys:
cand[k]=random.selection(SEARCH_SPACE[k])
adjustments[k]=cand[k]
return cand,adjustments
def apply_hparams(candidate):
textual content=read_text(BACKUP_FILE)
for ok,v in candidate.gadgets():
textual content=set_hparam(textual content,ok,v)
write_text(TRAIN_FILE,textual content)
def run_experiment(tag):
log=f"{tag}.log"
subprocess.run(f"python prepare.py > {log} 2>&1",shell=True)
metrics=parse_run_log(log)
metrics["log"]=log
return metricsWe construct the core utilities that allow automated hyperparameter experimentation. We extract the hyperparameters from prepare.py, outline the searchable parameter area, and implement capabilities that may programmatically edit these values. We additionally create mechanisms to generate candidate configurations, apply them to the coaching script, and run experiments whereas recording their outputs.
N_EXPERIMENTS=3
df=pd.read_csv(results_path,sep="t")
greatest=df["val_bpb"].substitute(0,999).min()
for i in vary(N_EXPERIMENTS):
tag=f"exp_{i+1}"
candidate,adjustments=sample_candidate()
apply_hparams(candidate)
metrics=run_experiment(tag)
if metrics["val_bpb"] and metrics["val_bpb"]We run the automated analysis loop that repeatedly proposes new hyperparameter configurations and evaluates their efficiency. For every experiment, we modify the coaching script, run the coaching course of, and evaluate the ensuing validation rating with the perfect configuration found up to now. We log all experiment outcomes, protect improved configurations, and export the perfect coaching script together with the experiment historical past for additional evaluation.
In conclusion, we constructed a whole automated analysis workflow that demonstrates how machines can iteratively discover mannequin configurations and enhance coaching efficiency with minimal handbook intervention. All through the tutorial, we ready the dataset, established a baseline experiment, and applied a search loop that proposes new hyperparameter configurations, runs experiments, and tracks outcomes throughout a number of trials. By sustaining experiment logs and robotically preserving improved configurations, we created a reproducible and extensible analysis course of that mirrors the workflow utilized in trendy machine studying experimentation. This strategy illustrates how we will mix automation, experimentation monitoring, and light-weight infrastructure to speed up mannequin growth and allow scalable analysis immediately from a cloud pocket book setting.
Take a look at Full Codes here. Additionally, be at liberty to comply with us on Twitter and don’t neglect to hitch our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
