[{"data":1,"prerenderedAt":2875},["ShallowReactive",2],{"casestudy-\u002Fcasestudies\u002Fsandboxed-evaluation":3},{"id":4,"title":5,"body":6,"date":2860,"description":2861,"extension":2862,"meta":2863,"navigation":174,"number":2864,"path":2865,"readingTime":2866,"seo":2867,"stem":2868,"topics":2869,"__hash__":2874},"casestudies\u002Fcasestudies\u002Fsandboxed-evaluation.md","Sandboxed evaluation",{"type":7,"value":8,"toc":2854},"minimark",[9,13,21,36,41,93,96,139,143,2693,2697,2703,2730,2736,2740,2760,2786,2818,2824,2850],[10,11,12],"p",{},"Run model-written Python inside a throwaway microVM over several turns, read each\nresult back, then delete the VM. This is the shape an eval harness or a coding\nagent wants: state carries across turns because it is the same guest, and nothing\nthe guest does reaches the machine driving it.",[10,14,15,16,20],{},"The VM is created with a 10 minute TTL. The script deletes it in a ",[17,18,19],"code",{},"finally",", so\nthe TTL is only the backstop for the run that dies before it gets there — a killed\nprocess, a lost network, a traceback in the harness itself.",[10,22,23,24,27,28,31,32,35],{},"Teardown is ",[17,25,26],{},"DELETE \u002Fvms\u002F{id}"," rather than ",[17,29,30],{},"POST \u002Fvms\u002F{id}\u002Fdestroy",". Both destroy\nthe guest and free the quota; the destroy leaves the record behind as ",[17,33,34],{},"gone",", which\nis what an operator wants and what a harness creating a VM per run does not.",[37,38,40],"h2",{"id":39},"what-it-needs","What it needs",[42,43,44,56,59,62,68,82],"ul",{},[45,46,47,48,51,52,55],"li",{},"A personal access token (Settings, or ",[17,49,50],{},"POST \u002Fme\u002Ftokens","), as ",[17,53,54],{},"DUMMIE_TOKEN",".",[45,57,58],{},"An SSH public key on your account. It is baked into the image at boot, so it\ncannot be added to a VM afterwards — set it before creating anything.",[45,60,61],{},"The matching private key readable by the script.",[45,63,64,65,55],{},"The generated SDK installed: ",[17,66,67],{},"just sdk-python && just sdk-install",[45,69,70,73,74,77,78,81],{},[17,71,72],{},"paramiko",", 3.2 or newer for ",[17,75,76],{},"PKey.from_path",". ",[17,79,80],{},"Dockerfile.sdk.dev"," pins 5.0.0.",[45,83,84,85,88,89,92],{},"An unencrypted private key. Nothing here prompts for a passphrase, and the\nkeyword that loads an encrypted one was renamed in paramiko 5.0 (",[17,86,87],{},"passphrase"," →\n",[17,90,91],{},"password","), so hardcoding it would break across the versions this runs on.",[10,94,95],{},"Two addresses are involved and only one of them comes from the API:",[42,97,98,108],{},[45,99,100,103,104,107],{},[17,101,102],{},"DUMMIE_BASE_URL"," — the control API, ",[17,105,106],{},"http:\u002F\u002Fcontrol:1323\u002Fapi\u002Fv1"," from the sdk\ncontainer.",[45,109,110,113,114,118,119,122,123,126,127,130,131,134,135,138],{},[17,111,112],{},"DUMMIE_SSH_HOST"," — the ",[115,116,117],"strong",{},"qemu host",", where ",[17,120,121],{},"proxy"," listens on port 22 and\nroutes the session to a guest by the key it authenticated with. ",[17,124,125],{},"10.68.0.2"," for\nthe local ",[17,128,129],{},"nix-vms"," host. ",[17,132,133],{},"GET \u002Fvms\u002Fhosts"," will not tell you this: ",[17,136,137],{},"hostname"," is\nwhat the host reported about itself, not necessarily anything you can dial.",[37,140,142],{"id":141},"the-script","The script",[144,145,151],"pre",{"className":146,"code":147,"filename":148,"language":149,"meta":150,"style":150},"language-python shiki shiki-themes github-light github-dark","#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"Run python in a throwaway VM across several turns, then delete it.\"\"\"\n\nimport os\nimport sys\nimport textwrap\nimport time\nfrom http import HTTPStatus\nfrom typing import NamedTuple\nfrom uuid import UUID\n\nimport paramiko\n\nfrom dummie import AuthenticatedClient\nfrom dummie.api.vms import (\n    delete_vms_id,\n    get_vms_hosts,\n    get_vms_id,\n    get_vms_kernels,\n    get_vms_osimages,\n    post_vms,\n)\nfrom dummie.models import ArtifactList, CreateVMReq, HostList, VmDTO\n\nBASE_URL = os.environ.get(\"DUMMIE_BASE_URL\", \"http:\u002F\u002Fcontrol:1323\u002Fapi\u002Fv1\")\nSSH_HOST = os.environ.get(\"DUMMIE_SSH_HOST\", \"10.68.0.2\")\nSSH_PORT = int(os.environ.get(\"DUMMIE_SSH_PORT\", \"22\"))\nSSH_KEY = os.environ.get(\"DUMMIE_SSH_KEY\", os.path.expanduser(\"~\u002F.ssh\u002Fid_ed25519\"))\nSSH_USER = \"ubuntu\"\n\nTTL_SECONDS = 600\nBOOT_TIMEOUT = 180\nSSH_TIMEOUT = 120\nTURN_TIMEOUT = 60\nDELETE_TIMEOUT = 60\nWORKDIR = \"\u002Fhome\u002Fubuntu\u002Feval\"\n\nTURNS = [\n    \"\"\"\n    from pathlib import Path\n    Path(\"primes.txt\").write_text(\"\\\\n\".join(\n        str(n) for n in range(2, 200)\n        if all(n % d for d in range(2, int(n**0.5) + 1))\n    ))\n    print(\"wrote\", Path(\"primes.txt\").stat().st_size, \"bytes\")\n    \"\"\",\n    \"\"\"\n    from pathlib import Path\n    primes = [int(line) for line in Path(\"primes.txt\").read_text().split()]\n    print(\"count:\", len(primes))\n    print(\"sum:\", sum(primes))\n    \"\"\",\n    \"\"\"\n    import platform\n    print(\"kernel:\", platform.release())\n    print(\"python:\", platform.python_version())\n    \"\"\",\n]\n\n\nclass Result(NamedTuple):\n    exit_status: int\n    stdout: str\n    stderr: str\n\n\ndef expect(result, want):\n    if not isinstance(result, want):\n        raise RuntimeError(f\"api call failed: {result!r}\")\n    return result\n\n\ndef create_vm(client: AuthenticatedClient) -> VmDTO:\n    hosts = expect(get_vms_hosts.sync(client=client), HostList)\n    kernels = expect(get_vms_kernels.sync(client=client), ArtifactList)\n    osimages = expect(get_vms_osimages.sync(client=client), ArtifactList)\n    if not (hosts.items and kernels.items and osimages.items):\n        raise RuntimeError(\"need a connected host, a kernel and an os image\")\n\n    req = CreateVMReq(\n        name=\"amazing-jepsen\",\n        client_id=hosts.items[0].id,\n        kernel_id=kernels.items[0].id,\n        osimage_id=osimages.items[0].id,\n        cpus=1,\n        memory_mib=512,\n        disk_size=\"2G\",\n        ttl_seconds=TTL_SECONDS,\n    )\n    return expect(post_vms.sync(client=client, body=req), VmDTO)\n\n\ndef wait_running(client: AuthenticatedClient, vm_id: str) -> VmDTO:\n    deadline = time.monotonic() + BOOT_TIMEOUT\n    status = \"unknown\"\n    while time.monotonic() \u003C deadline:\n        vm = expect(get_vms_id.sync(UUID(vm_id), client=client), VmDTO)\n        status = vm.status\n        if status == \"running\":\n            return vm\n        if status == \"failed\":\n            raise RuntimeError(f\"vm failed to boot: {vm.last_error}\")\n        time.sleep(2)\n    raise TimeoutError(f\"vm was still {status!r} after {BOOT_TIMEOUT}s\")\n\n\ndef delete_vm(client: AuthenticatedClient, vm_id: str) -> None:\n    \"\"\"Destroy the guest, drop the record, and wait for it to actually be gone.\n\n    The 202 means the destroy reached the host, not that it finished. What the\n    next run needs -- the quota, and the host's one proxy entry per key -- is only\n    released once the host confirms, so the wait is the useful part. A 204 is a VM\n    that had nothing on a host, and is already gone by the time it returns.\n    \"\"\"\n    res = delete_vms_id.sync_detailed(UUID(vm_id), client=client)\n    if res.status_code not in (HTTPStatus.ACCEPTED, HTTPStatus.NO_CONTENT):\n        raise RuntimeError(f\"could not delete the vm: {res.parsed!r}\")\n    if res.status_code == HTTPStatus.NO_CONTENT:\n        return\n\n    deadline = time.monotonic() + DELETE_TIMEOUT\n    while time.monotonic() \u003C deadline:\n        got = get_vms_id.sync_detailed(UUID(vm_id), client=client)\n        if got.status_code == HTTPStatus.NOT_FOUND:\n            return\n        time.sleep(2)\n    raise TimeoutError(f\"the vm record was still there after {DELETE_TIMEOUT}s\")\n\n\ndef connect_ssh() -> paramiko.SSHClient:\n    \"\"\"Connect once a command actually runs, not once the handshake succeeds.\n\n    Every step here is transient while a guest comes up. The proxy routes by\n    public key, so a session opened before its regenerated config lands\n    authenticates and then dies at the first channel -- which is why the probe\n    is part of the attempt rather than the first turn's problem.\n    \"\"\"\n    key = paramiko.PKey.from_path(SSH_KEY)\n    deadline = time.monotonic() + SSH_TIMEOUT\n    last: Exception | None = None\n    while time.monotonic() \u003C deadline:\n        client = paramiko.SSHClient()\n        # The guest's host key is new for every VM, so there is nothing to\n        # remember; no known_hosts is loaded, so AutoAdd stays in this process.\n        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())\n        try:\n            client.connect(\n                hostname=SSH_HOST,\n                port=SSH_PORT,\n                username=SSH_USER,\n                pkey=key,\n                look_for_keys=False,\n                allow_agent=False,\n                timeout=10,\n                banner_timeout=10,\n                auth_timeout=10,\n            )\n            _, stdout, _ = client.exec_command(\"true\", timeout=10)\n            if stdout.channel.recv_exit_status() != 0:\n                raise paramiko.SSHException(\"probe command did not run\")\n            return client\n        except (paramiko.SSHException, EOFError, OSError) as err:\n            last = err\n            client.close()\n            time.sleep(2)\n    raise TimeoutError(f\"no usable ssh to the guest after {SSH_TIMEOUT}s: {last!r}\")\n\n\ndef run_turn(ssh: paramiko.SSHClient, n: int, code: str) -> Result:\n    stdin, stdout, stderr = ssh.exec_command(\n        f\"mkdir -p {WORKDIR} && cd {WORKDIR} && python3 -\", timeout=TURN_TIMEOUT\n    )\n    stdin.write(textwrap.dedent(code))\n    stdin.channel.shutdown_write()\n\n    out = stdout.read().decode()\n    err = stderr.read().decode()\n    result = Result(stdout.channel.recv_exit_status(), out, err)\n\n    print(f\"--- turn {n} (exit {result.exit_status})\")\n    if out:\n        print(out, end=\"\")\n    if err:\n        print(err, end=\"\", file=sys.stderr)\n    return result\n\n\ndef main() -> int:\n    client = AuthenticatedClient(\n        base_url=BASE_URL,\n        token=os.environ[\"DUMMIE_TOKEN\"],\n        raise_on_unexpected_status=True,\n    )\n\n    started = time.monotonic()\n    vm = create_vm(client)\n    print(f\"created {vm.name} ({vm.id}), ttl {TTL_SECONDS}s\")\n    try:\n        vm = wait_running(client, vm.id)\n        booted = time.monotonic()\n        print(f\"running at {vm.ip} after {booted - started:.1f}s\")\n        # One connection for every turn, so a turn's cost is a channel rather\n        # than a handshake, and the guest sees a single session throughout.\n        ssh = connect_ssh()\n        print(\n            f\"ssh ready after a further {time.monotonic() - booted:.1f}s\"\n            f\" ({time.monotonic() - started:.1f}s since create)\"\n        )\n        try:\n            for n, code in enumerate(TURNS, start=1):\n                if run_turn(ssh, n, code).exit_status != 0:\n                    print(\"turn failed; stopping\", file=sys.stderr)\n                    break\n        finally:\n            ssh.close()\n    finally:\n        delete_vm(client, vm.id)\n        print(f\"deleted {vm.name}\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    raise SystemExit(main())\n","evaluate.py","python","",[17,152,153,162,169,176,187,195,203,211,225,238,252,257,265,270,283,296,302,308,314,320,326,332,338,351,356,379,399,424,445,456,461,472,483,494,505,515,526,531,542,548,554,566,572,584,590,596,605,610,615,621,627,633,640,645,651,657,663,670,676,681,686,705,714,723,731,736,741,753,767,801,810,815,820,831,852,870,887,908,922,927,938,951,968,983,998,1011,1024,1037,1049,1055,1078,1083,1088,1105,1122,1133,1147,1165,1176,1194,1203,1217,1243,1254,1290,1295,1300,1320,1326,1331,1337,1343,1349,1355,1360,1378,1406,1433,1449,1455,1460,1474,1485,1502,1519,1525,1534,1555,1560,1565,1576,1582,1587,1593,1599,1605,1611,1616,1631,1645,1665,1676,1687,1693,1699,1705,1713,1719,1731,1743,1755,1766,1779,1791,1804,1816,1828,1834,1859,1876,1890,1898,1924,1935,1941,1951,1984,1989,1994,2016,2027,2056,2061,2067,2073,2078,2089,2100,2111,2116,2151,2159,2178,2185,2209,2216,2221,2226,2241,2252,2264,2281,2294,2299,2304,2315,2326,2365,2373,2383,2393,2434,2440,2446,2457,2465,2491,2514,2520,2527,2557,2572,2591,2597,2605,2611,2619,2625,2647,2655,2660,2665,2682],{"__ignoreMap":150},[154,155,158],"span",{"class":156,"line":157},"line",1,[154,159,161],{"class":160},"sJ8bj","#!\u002Fusr\u002Fbin\u002Fenv python3\n",[154,163,165],{"class":156,"line":164},2,[154,166,168],{"class":167},"sZZnC","\"\"\"Run python in a throwaway VM across several turns, then delete it.\"\"\"\n",[154,170,172],{"class":156,"line":171},3,[154,173,175],{"emptyLinePlaceholder":174},true,"\n",[154,177,179,183],{"class":156,"line":178},4,[154,180,182],{"class":181},"szBVR","import",[154,184,186],{"class":185},"sVt8B"," os\n",[154,188,190,192],{"class":156,"line":189},5,[154,191,182],{"class":181},[154,193,194],{"class":185}," sys\n",[154,196,198,200],{"class":156,"line":197},6,[154,199,182],{"class":181},[154,201,202],{"class":185}," textwrap\n",[154,204,206,208],{"class":156,"line":205},7,[154,207,182],{"class":181},[154,209,210],{"class":185}," time\n",[154,212,214,217,220,222],{"class":156,"line":213},8,[154,215,216],{"class":181},"from",[154,218,219],{"class":185}," http ",[154,221,182],{"class":181},[154,223,224],{"class":185}," HTTPStatus\n",[154,226,228,230,233,235],{"class":156,"line":227},9,[154,229,216],{"class":181},[154,231,232],{"class":185}," typing ",[154,234,182],{"class":181},[154,236,237],{"class":185}," NamedTuple\n",[154,239,241,243,246,248],{"class":156,"line":240},10,[154,242,216],{"class":181},[154,244,245],{"class":185}," uuid ",[154,247,182],{"class":181},[154,249,251],{"class":250},"sj4cs"," UUID\n",[154,253,255],{"class":156,"line":254},11,[154,256,175],{"emptyLinePlaceholder":174},[154,258,260,262],{"class":156,"line":259},12,[154,261,182],{"class":181},[154,263,264],{"class":185}," paramiko\n",[154,266,268],{"class":156,"line":267},13,[154,269,175],{"emptyLinePlaceholder":174},[154,271,273,275,278,280],{"class":156,"line":272},14,[154,274,216],{"class":181},[154,276,277],{"class":185}," dummie ",[154,279,182],{"class":181},[154,281,282],{"class":185}," AuthenticatedClient\n",[154,284,286,288,291,293],{"class":156,"line":285},15,[154,287,216],{"class":181},[154,289,290],{"class":185}," dummie.api.vms ",[154,292,182],{"class":181},[154,294,295],{"class":185}," (\n",[154,297,299],{"class":156,"line":298},16,[154,300,301],{"class":185},"    delete_vms_id,\n",[154,303,305],{"class":156,"line":304},17,[154,306,307],{"class":185},"    get_vms_hosts,\n",[154,309,311],{"class":156,"line":310},18,[154,312,313],{"class":185},"    get_vms_id,\n",[154,315,317],{"class":156,"line":316},19,[154,318,319],{"class":185},"    get_vms_kernels,\n",[154,321,323],{"class":156,"line":322},20,[154,324,325],{"class":185},"    get_vms_osimages,\n",[154,327,329],{"class":156,"line":328},21,[154,330,331],{"class":185},"    post_vms,\n",[154,333,335],{"class":156,"line":334},22,[154,336,337],{"class":185},")\n",[154,339,341,343,346,348],{"class":156,"line":340},23,[154,342,216],{"class":181},[154,344,345],{"class":185}," dummie.models ",[154,347,182],{"class":181},[154,349,350],{"class":185}," ArtifactList, CreateVMReq, HostList, VmDTO\n",[154,352,354],{"class":156,"line":353},24,[154,355,175],{"emptyLinePlaceholder":174},[154,357,359,362,365,368,371,374,377],{"class":156,"line":358},25,[154,360,361],{"class":250},"BASE_URL",[154,363,364],{"class":181}," =",[154,366,367],{"class":185}," os.environ.get(",[154,369,370],{"class":167},"\"DUMMIE_BASE_URL\"",[154,372,373],{"class":185},", ",[154,375,376],{"class":167},"\"http:\u002F\u002Fcontrol:1323\u002Fapi\u002Fv1\"",[154,378,337],{"class":185},[154,380,382,385,387,389,392,394,397],{"class":156,"line":381},26,[154,383,384],{"class":250},"SSH_HOST",[154,386,364],{"class":181},[154,388,367],{"class":185},[154,390,391],{"class":167},"\"DUMMIE_SSH_HOST\"",[154,393,373],{"class":185},[154,395,396],{"class":167},"\"10.68.0.2\"",[154,398,337],{"class":185},[154,400,402,405,407,410,413,416,418,421],{"class":156,"line":401},27,[154,403,404],{"class":250},"SSH_PORT",[154,406,364],{"class":181},[154,408,409],{"class":250}," int",[154,411,412],{"class":185},"(os.environ.get(",[154,414,415],{"class":167},"\"DUMMIE_SSH_PORT\"",[154,417,373],{"class":185},[154,419,420],{"class":167},"\"22\"",[154,422,423],{"class":185},"))\n",[154,425,427,430,432,434,437,440,443],{"class":156,"line":426},28,[154,428,429],{"class":250},"SSH_KEY",[154,431,364],{"class":181},[154,433,367],{"class":185},[154,435,436],{"class":167},"\"DUMMIE_SSH_KEY\"",[154,438,439],{"class":185},", os.path.expanduser(",[154,441,442],{"class":167},"\"~\u002F.ssh\u002Fid_ed25519\"",[154,444,423],{"class":185},[154,446,448,451,453],{"class":156,"line":447},29,[154,449,450],{"class":250},"SSH_USER",[154,452,364],{"class":181},[154,454,455],{"class":167}," \"ubuntu\"\n",[154,457,459],{"class":156,"line":458},30,[154,460,175],{"emptyLinePlaceholder":174},[154,462,464,467,469],{"class":156,"line":463},31,[154,465,466],{"class":250},"TTL_SECONDS",[154,468,364],{"class":181},[154,470,471],{"class":250}," 600\n",[154,473,475,478,480],{"class":156,"line":474},32,[154,476,477],{"class":250},"BOOT_TIMEOUT",[154,479,364],{"class":181},[154,481,482],{"class":250}," 180\n",[154,484,486,489,491],{"class":156,"line":485},33,[154,487,488],{"class":250},"SSH_TIMEOUT",[154,490,364],{"class":181},[154,492,493],{"class":250}," 120\n",[154,495,497,500,502],{"class":156,"line":496},34,[154,498,499],{"class":250},"TURN_TIMEOUT",[154,501,364],{"class":181},[154,503,504],{"class":250}," 60\n",[154,506,508,511,513],{"class":156,"line":507},35,[154,509,510],{"class":250},"DELETE_TIMEOUT",[154,512,364],{"class":181},[154,514,504],{"class":250},[154,516,518,521,523],{"class":156,"line":517},36,[154,519,520],{"class":250},"WORKDIR",[154,522,364],{"class":181},[154,524,525],{"class":167}," \"\u002Fhome\u002Fubuntu\u002Feval\"\n",[154,527,529],{"class":156,"line":528},37,[154,530,175],{"emptyLinePlaceholder":174},[154,532,534,537,539],{"class":156,"line":533},38,[154,535,536],{"class":250},"TURNS",[154,538,364],{"class":181},[154,540,541],{"class":185}," [\n",[154,543,545],{"class":156,"line":544},39,[154,546,547],{"class":167},"    \"\"\"\n",[154,549,551],{"class":156,"line":550},40,[154,552,553],{"class":167},"    from pathlib import Path\n",[154,555,557,560,563],{"class":156,"line":556},41,[154,558,559],{"class":167},"    Path(\"primes.txt\").write_text(\"",[154,561,562],{"class":250},"\\\\",[154,564,565],{"class":167},"n\".join(\n",[154,567,569],{"class":156,"line":568},42,[154,570,571],{"class":167},"        str(n) for n in range(2, 200)\n",[154,573,575,578,581],{"class":156,"line":574},43,[154,576,577],{"class":167},"        if all(n ",[154,579,580],{"class":250},"% d",[154,582,583],{"class":167}," for d in range(2, int(n**0.5) + 1))\n",[154,585,587],{"class":156,"line":586},44,[154,588,589],{"class":167},"    ))\n",[154,591,593],{"class":156,"line":592},45,[154,594,595],{"class":167},"    print(\"wrote\", Path(\"primes.txt\").stat().st_size, \"bytes\")\n",[154,597,599,602],{"class":156,"line":598},46,[154,600,601],{"class":167},"    \"\"\"",[154,603,604],{"class":185},",\n",[154,606,608],{"class":156,"line":607},47,[154,609,547],{"class":167},[154,611,613],{"class":156,"line":612},48,[154,614,553],{"class":167},[154,616,618],{"class":156,"line":617},49,[154,619,620],{"class":167},"    primes = [int(line) for line in Path(\"primes.txt\").read_text().split()]\n",[154,622,624],{"class":156,"line":623},50,[154,625,626],{"class":167},"    print(\"count:\", len(primes))\n",[154,628,630],{"class":156,"line":629},51,[154,631,632],{"class":167},"    print(\"sum:\", sum(primes))\n",[154,634,636,638],{"class":156,"line":635},52,[154,637,601],{"class":167},[154,639,604],{"class":185},[154,641,643],{"class":156,"line":642},53,[154,644,547],{"class":167},[154,646,648],{"class":156,"line":647},54,[154,649,650],{"class":167},"    import platform\n",[154,652,654],{"class":156,"line":653},55,[154,655,656],{"class":167},"    print(\"kernel:\", platform.release())\n",[154,658,660],{"class":156,"line":659},56,[154,661,662],{"class":167},"    print(\"python:\", platform.python_version())\n",[154,664,666,668],{"class":156,"line":665},57,[154,667,601],{"class":167},[154,669,604],{"class":185},[154,671,673],{"class":156,"line":672},58,[154,674,675],{"class":185},"]\n",[154,677,679],{"class":156,"line":678},59,[154,680,175],{"emptyLinePlaceholder":174},[154,682,684],{"class":156,"line":683},60,[154,685,175],{"emptyLinePlaceholder":174},[154,687,689,692,696,699,702],{"class":156,"line":688},61,[154,690,691],{"class":181},"class",[154,693,695],{"class":694},"sScJk"," Result",[154,697,698],{"class":185},"(",[154,700,701],{"class":694},"NamedTuple",[154,703,704],{"class":185},"):\n",[154,706,708,711],{"class":156,"line":707},62,[154,709,710],{"class":185},"    exit_status: ",[154,712,713],{"class":250},"int\n",[154,715,717,720],{"class":156,"line":716},63,[154,718,719],{"class":185},"    stdout: ",[154,721,722],{"class":250},"str\n",[154,724,726,729],{"class":156,"line":725},64,[154,727,728],{"class":185},"    stderr: ",[154,730,722],{"class":250},[154,732,734],{"class":156,"line":733},65,[154,735,175],{"emptyLinePlaceholder":174},[154,737,739],{"class":156,"line":738},66,[154,740,175],{"emptyLinePlaceholder":174},[154,742,744,747,750],{"class":156,"line":743},67,[154,745,746],{"class":181},"def",[154,748,749],{"class":694}," expect",[154,751,752],{"class":185},"(result, want):\n",[154,754,756,759,762,765],{"class":156,"line":755},68,[154,757,758],{"class":181},"    if",[154,760,761],{"class":181}," not",[154,763,764],{"class":250}," isinstance",[154,766,752],{"class":185},[154,768,770,773,776,778,781,784,787,790,793,796,799],{"class":156,"line":769},69,[154,771,772],{"class":181},"        raise",[154,774,775],{"class":250}," RuntimeError",[154,777,698],{"class":185},[154,779,780],{"class":181},"f",[154,782,783],{"class":167},"\"api call failed: ",[154,785,786],{"class":250},"{",[154,788,789],{"class":185},"result",[154,791,792],{"class":181},"!r",[154,794,795],{"class":250},"}",[154,797,798],{"class":167},"\"",[154,800,337],{"class":185},[154,802,804,807],{"class":156,"line":803},70,[154,805,806],{"class":181},"    return",[154,808,809],{"class":185}," result\n",[154,811,813],{"class":156,"line":812},71,[154,814,175],{"emptyLinePlaceholder":174},[154,816,818],{"class":156,"line":817},72,[154,819,175],{"emptyLinePlaceholder":174},[154,821,823,825,828],{"class":156,"line":822},73,[154,824,746],{"class":181},[154,826,827],{"class":694}," create_vm",[154,829,830],{"class":185},"(client: AuthenticatedClient) -> VmDTO:\n",[154,832,834,837,840,843,847,849],{"class":156,"line":833},74,[154,835,836],{"class":185},"    hosts ",[154,838,839],{"class":181},"=",[154,841,842],{"class":185}," expect(get_vms_hosts.sync(",[154,844,846],{"class":845},"s4XuR","client",[154,848,839],{"class":181},[154,850,851],{"class":185},"client), HostList)\n",[154,853,855,858,860,863,865,867],{"class":156,"line":854},75,[154,856,857],{"class":185},"    kernels ",[154,859,839],{"class":181},[154,861,862],{"class":185}," expect(get_vms_kernels.sync(",[154,864,846],{"class":845},[154,866,839],{"class":181},[154,868,869],{"class":185},"client), ArtifactList)\n",[154,871,873,876,878,881,883,885],{"class":156,"line":872},76,[154,874,875],{"class":185},"    osimages ",[154,877,839],{"class":181},[154,879,880],{"class":185}," expect(get_vms_osimages.sync(",[154,882,846],{"class":845},[154,884,839],{"class":181},[154,886,869],{"class":185},[154,888,890,892,894,897,900,903,905],{"class":156,"line":889},77,[154,891,758],{"class":181},[154,893,761],{"class":181},[154,895,896],{"class":185}," (hosts.items ",[154,898,899],{"class":181},"and",[154,901,902],{"class":185}," kernels.items ",[154,904,899],{"class":181},[154,906,907],{"class":185}," osimages.items):\n",[154,909,911,913,915,917,920],{"class":156,"line":910},78,[154,912,772],{"class":181},[154,914,775],{"class":250},[154,916,698],{"class":185},[154,918,919],{"class":167},"\"need a connected host, a kernel and an os image\"",[154,921,337],{"class":185},[154,923,925],{"class":156,"line":924},79,[154,926,175],{"emptyLinePlaceholder":174},[154,928,930,933,935],{"class":156,"line":929},80,[154,931,932],{"class":185},"    req ",[154,934,839],{"class":181},[154,936,937],{"class":185}," CreateVMReq(\n",[154,939,941,944,946,949],{"class":156,"line":940},81,[154,942,943],{"class":845},"        name",[154,945,839],{"class":181},[154,947,948],{"class":167},"\"amazing-jepsen\"",[154,950,604],{"class":185},[154,952,954,957,959,962,965],{"class":156,"line":953},82,[154,955,956],{"class":845},"        client_id",[154,958,839],{"class":181},[154,960,961],{"class":185},"hosts.items[",[154,963,964],{"class":250},"0",[154,966,967],{"class":185},"].id,\n",[154,969,971,974,976,979,981],{"class":156,"line":970},83,[154,972,973],{"class":845},"        kernel_id",[154,975,839],{"class":181},[154,977,978],{"class":185},"kernels.items[",[154,980,964],{"class":250},[154,982,967],{"class":185},[154,984,986,989,991,994,996],{"class":156,"line":985},84,[154,987,988],{"class":845},"        osimage_id",[154,990,839],{"class":181},[154,992,993],{"class":185},"osimages.items[",[154,995,964],{"class":250},[154,997,967],{"class":185},[154,999,1001,1004,1006,1009],{"class":156,"line":1000},85,[154,1002,1003],{"class":845},"        cpus",[154,1005,839],{"class":181},[154,1007,1008],{"class":250},"1",[154,1010,604],{"class":185},[154,1012,1014,1017,1019,1022],{"class":156,"line":1013},86,[154,1015,1016],{"class":845},"        memory_mib",[154,1018,839],{"class":181},[154,1020,1021],{"class":250},"512",[154,1023,604],{"class":185},[154,1025,1027,1030,1032,1035],{"class":156,"line":1026},87,[154,1028,1029],{"class":845},"        disk_size",[154,1031,839],{"class":181},[154,1033,1034],{"class":167},"\"2G\"",[154,1036,604],{"class":185},[154,1038,1040,1043,1045,1047],{"class":156,"line":1039},88,[154,1041,1042],{"class":845},"        ttl_seconds",[154,1044,839],{"class":181},[154,1046,466],{"class":250},[154,1048,604],{"class":185},[154,1050,1052],{"class":156,"line":1051},89,[154,1053,1054],{"class":185},"    )\n",[154,1056,1058,1060,1063,1065,1067,1070,1073,1075],{"class":156,"line":1057},90,[154,1059,806],{"class":181},[154,1061,1062],{"class":185}," expect(post_vms.sync(",[154,1064,846],{"class":845},[154,1066,839],{"class":181},[154,1068,1069],{"class":185},"client, ",[154,1071,1072],{"class":845},"body",[154,1074,839],{"class":181},[154,1076,1077],{"class":185},"req), VmDTO)\n",[154,1079,1081],{"class":156,"line":1080},91,[154,1082,175],{"emptyLinePlaceholder":174},[154,1084,1086],{"class":156,"line":1085},92,[154,1087,175],{"emptyLinePlaceholder":174},[154,1089,1091,1093,1096,1099,1102],{"class":156,"line":1090},93,[154,1092,746],{"class":181},[154,1094,1095],{"class":694}," wait_running",[154,1097,1098],{"class":185},"(client: AuthenticatedClient, vm_id: ",[154,1100,1101],{"class":250},"str",[154,1103,1104],{"class":185},") -> VmDTO:\n",[154,1106,1108,1111,1113,1116,1119],{"class":156,"line":1107},94,[154,1109,1110],{"class":185},"    deadline ",[154,1112,839],{"class":181},[154,1114,1115],{"class":185}," time.monotonic() ",[154,1117,1118],{"class":181},"+",[154,1120,1121],{"class":250}," BOOT_TIMEOUT\n",[154,1123,1125,1128,1130],{"class":156,"line":1124},95,[154,1126,1127],{"class":185},"    status ",[154,1129,839],{"class":181},[154,1131,1132],{"class":167}," \"unknown\"\n",[154,1134,1136,1139,1141,1144],{"class":156,"line":1135},96,[154,1137,1138],{"class":181},"    while",[154,1140,1115],{"class":185},[154,1142,1143],{"class":181},"\u003C",[154,1145,1146],{"class":185}," deadline:\n",[154,1148,1150,1153,1155,1158,1160,1162],{"class":156,"line":1149},97,[154,1151,1152],{"class":185},"        vm ",[154,1154,839],{"class":181},[154,1156,1157],{"class":185}," expect(get_vms_id.sync(UUID(vm_id), ",[154,1159,846],{"class":845},[154,1161,839],{"class":181},[154,1163,1164],{"class":185},"client), VmDTO)\n",[154,1166,1168,1171,1173],{"class":156,"line":1167},98,[154,1169,1170],{"class":185},"        status ",[154,1172,839],{"class":181},[154,1174,1175],{"class":185}," vm.status\n",[154,1177,1179,1182,1185,1188,1191],{"class":156,"line":1178},99,[154,1180,1181],{"class":181},"        if",[154,1183,1184],{"class":185}," status ",[154,1186,1187],{"class":181},"==",[154,1189,1190],{"class":167}," \"running\"",[154,1192,1193],{"class":185},":\n",[154,1195,1197,1200],{"class":156,"line":1196},100,[154,1198,1199],{"class":181},"            return",[154,1201,1202],{"class":185}," vm\n",[154,1204,1206,1208,1210,1212,1215],{"class":156,"line":1205},101,[154,1207,1181],{"class":181},[154,1209,1184],{"class":185},[154,1211,1187],{"class":181},[154,1213,1214],{"class":167}," \"failed\"",[154,1216,1193],{"class":185},[154,1218,1220,1223,1225,1227,1229,1232,1234,1237,1239,1241],{"class":156,"line":1219},102,[154,1221,1222],{"class":181},"            raise",[154,1224,775],{"class":250},[154,1226,698],{"class":185},[154,1228,780],{"class":181},[154,1230,1231],{"class":167},"\"vm failed to boot: ",[154,1233,786],{"class":250},[154,1235,1236],{"class":185},"vm.last_error",[154,1238,795],{"class":250},[154,1240,798],{"class":167},[154,1242,337],{"class":185},[154,1244,1246,1249,1252],{"class":156,"line":1245},103,[154,1247,1248],{"class":185},"        time.sleep(",[154,1250,1251],{"class":250},"2",[154,1253,337],{"class":185},[154,1255,1257,1260,1263,1265,1267,1270,1272,1275,1277,1279,1282,1285,1288],{"class":156,"line":1256},104,[154,1258,1259],{"class":181},"    raise",[154,1261,1262],{"class":250}," TimeoutError",[154,1264,698],{"class":185},[154,1266,780],{"class":181},[154,1268,1269],{"class":167},"\"vm was still ",[154,1271,786],{"class":250},[154,1273,1274],{"class":185},"status",[154,1276,792],{"class":181},[154,1278,795],{"class":250},[154,1280,1281],{"class":167}," after ",[154,1283,1284],{"class":250},"{BOOT_TIMEOUT}",[154,1286,1287],{"class":167},"s\"",[154,1289,337],{"class":185},[154,1291,1293],{"class":156,"line":1292},105,[154,1294,175],{"emptyLinePlaceholder":174},[154,1296,1298],{"class":156,"line":1297},106,[154,1299,175],{"emptyLinePlaceholder":174},[154,1301,1303,1305,1308,1310,1312,1315,1318],{"class":156,"line":1302},107,[154,1304,746],{"class":181},[154,1306,1307],{"class":694}," delete_vm",[154,1309,1098],{"class":185},[154,1311,1101],{"class":250},[154,1313,1314],{"class":185},") -> ",[154,1316,1317],{"class":250},"None",[154,1319,1193],{"class":185},[154,1321,1323],{"class":156,"line":1322},108,[154,1324,1325],{"class":167},"    \"\"\"Destroy the guest, drop the record, and wait for it to actually be gone.\n",[154,1327,1329],{"class":156,"line":1328},109,[154,1330,175],{"emptyLinePlaceholder":174},[154,1332,1334],{"class":156,"line":1333},110,[154,1335,1336],{"class":167},"    The 202 means the destroy reached the host, not that it finished. What the\n",[154,1338,1340],{"class":156,"line":1339},111,[154,1341,1342],{"class":167},"    next run needs -- the quota, and the host's one proxy entry per key -- is only\n",[154,1344,1346],{"class":156,"line":1345},112,[154,1347,1348],{"class":167},"    released once the host confirms, so the wait is the useful part. A 204 is a VM\n",[154,1350,1352],{"class":156,"line":1351},113,[154,1353,1354],{"class":167},"    that had nothing on a host, and is already gone by the time it returns.\n",[154,1356,1358],{"class":156,"line":1357},114,[154,1359,547],{"class":167},[154,1361,1363,1366,1368,1371,1373,1375],{"class":156,"line":1362},115,[154,1364,1365],{"class":185},"    res ",[154,1367,839],{"class":181},[154,1369,1370],{"class":185}," delete_vms_id.sync_detailed(UUID(vm_id), ",[154,1372,846],{"class":845},[154,1374,839],{"class":181},[154,1376,1377],{"class":185},"client)\n",[154,1379,1381,1383,1386,1389,1392,1395,1398,1401,1404],{"class":156,"line":1380},116,[154,1382,758],{"class":181},[154,1384,1385],{"class":185}," res.status_code ",[154,1387,1388],{"class":181},"not",[154,1390,1391],{"class":181}," in",[154,1393,1394],{"class":185}," (HTTPStatus.",[154,1396,1397],{"class":250},"ACCEPTED",[154,1399,1400],{"class":185},", HTTPStatus.",[154,1402,1403],{"class":250},"NO_CONTENT",[154,1405,704],{"class":185},[154,1407,1409,1411,1413,1415,1417,1420,1422,1425,1427,1429,1431],{"class":156,"line":1408},117,[154,1410,772],{"class":181},[154,1412,775],{"class":250},[154,1414,698],{"class":185},[154,1416,780],{"class":181},[154,1418,1419],{"class":167},"\"could not delete the vm: ",[154,1421,786],{"class":250},[154,1423,1424],{"class":185},"res.parsed",[154,1426,792],{"class":181},[154,1428,795],{"class":250},[154,1430,798],{"class":167},[154,1432,337],{"class":185},[154,1434,1436,1438,1440,1442,1445,1447],{"class":156,"line":1435},118,[154,1437,758],{"class":181},[154,1439,1385],{"class":185},[154,1441,1187],{"class":181},[154,1443,1444],{"class":185}," HTTPStatus.",[154,1446,1403],{"class":250},[154,1448,1193],{"class":185},[154,1450,1452],{"class":156,"line":1451},119,[154,1453,1454],{"class":181},"        return\n",[154,1456,1458],{"class":156,"line":1457},120,[154,1459,175],{"emptyLinePlaceholder":174},[154,1461,1463,1465,1467,1469,1471],{"class":156,"line":1462},121,[154,1464,1110],{"class":185},[154,1466,839],{"class":181},[154,1468,1115],{"class":185},[154,1470,1118],{"class":181},[154,1472,1473],{"class":250}," DELETE_TIMEOUT\n",[154,1475,1477,1479,1481,1483],{"class":156,"line":1476},122,[154,1478,1138],{"class":181},[154,1480,1115],{"class":185},[154,1482,1143],{"class":181},[154,1484,1146],{"class":185},[154,1486,1488,1491,1493,1496,1498,1500],{"class":156,"line":1487},123,[154,1489,1490],{"class":185},"        got ",[154,1492,839],{"class":181},[154,1494,1495],{"class":185}," get_vms_id.sync_detailed(UUID(vm_id), ",[154,1497,846],{"class":845},[154,1499,839],{"class":181},[154,1501,1377],{"class":185},[154,1503,1505,1507,1510,1512,1514,1517],{"class":156,"line":1504},124,[154,1506,1181],{"class":181},[154,1508,1509],{"class":185}," got.status_code ",[154,1511,1187],{"class":181},[154,1513,1444],{"class":185},[154,1515,1516],{"class":250},"NOT_FOUND",[154,1518,1193],{"class":185},[154,1520,1522],{"class":156,"line":1521},125,[154,1523,1524],{"class":181},"            return\n",[154,1526,1528,1530,1532],{"class":156,"line":1527},126,[154,1529,1248],{"class":185},[154,1531,1251],{"class":250},[154,1533,337],{"class":185},[154,1535,1537,1539,1541,1543,1545,1548,1551,1553],{"class":156,"line":1536},127,[154,1538,1259],{"class":181},[154,1540,1262],{"class":250},[154,1542,698],{"class":185},[154,1544,780],{"class":181},[154,1546,1547],{"class":167},"\"the vm record was still there after ",[154,1549,1550],{"class":250},"{DELETE_TIMEOUT}",[154,1552,1287],{"class":167},[154,1554,337],{"class":185},[154,1556,1558],{"class":156,"line":1557},128,[154,1559,175],{"emptyLinePlaceholder":174},[154,1561,1563],{"class":156,"line":1562},129,[154,1564,175],{"emptyLinePlaceholder":174},[154,1566,1568,1570,1573],{"class":156,"line":1567},130,[154,1569,746],{"class":181},[154,1571,1572],{"class":694}," connect_ssh",[154,1574,1575],{"class":185},"() -> paramiko.SSHClient:\n",[154,1577,1579],{"class":156,"line":1578},131,[154,1580,1581],{"class":167},"    \"\"\"Connect once a command actually runs, not once the handshake succeeds.\n",[154,1583,1585],{"class":156,"line":1584},132,[154,1586,175],{"emptyLinePlaceholder":174},[154,1588,1590],{"class":156,"line":1589},133,[154,1591,1592],{"class":167},"    Every step here is transient while a guest comes up. The proxy routes by\n",[154,1594,1596],{"class":156,"line":1595},134,[154,1597,1598],{"class":167},"    public key, so a session opened before its regenerated config lands\n",[154,1600,1602],{"class":156,"line":1601},135,[154,1603,1604],{"class":167},"    authenticates and then dies at the first channel -- which is why the probe\n",[154,1606,1608],{"class":156,"line":1607},136,[154,1609,1610],{"class":167},"    is part of the attempt rather than the first turn's problem.\n",[154,1612,1614],{"class":156,"line":1613},137,[154,1615,547],{"class":167},[154,1617,1619,1622,1624,1627,1629],{"class":156,"line":1618},138,[154,1620,1621],{"class":185},"    key ",[154,1623,839],{"class":181},[154,1625,1626],{"class":185}," paramiko.PKey.from_path(",[154,1628,429],{"class":250},[154,1630,337],{"class":185},[154,1632,1634,1636,1638,1640,1642],{"class":156,"line":1633},139,[154,1635,1110],{"class":185},[154,1637,839],{"class":181},[154,1639,1115],{"class":185},[154,1641,1118],{"class":181},[154,1643,1644],{"class":250}," SSH_TIMEOUT\n",[154,1646,1648,1651,1654,1657,1660,1662],{"class":156,"line":1647},140,[154,1649,1650],{"class":185},"    last: ",[154,1652,1653],{"class":250},"Exception",[154,1655,1656],{"class":181}," |",[154,1658,1659],{"class":250}," None",[154,1661,364],{"class":181},[154,1663,1664],{"class":250}," None\n",[154,1666,1668,1670,1672,1674],{"class":156,"line":1667},141,[154,1669,1138],{"class":181},[154,1671,1115],{"class":185},[154,1673,1143],{"class":181},[154,1675,1146],{"class":185},[154,1677,1679,1682,1684],{"class":156,"line":1678},142,[154,1680,1681],{"class":185},"        client ",[154,1683,839],{"class":181},[154,1685,1686],{"class":185}," paramiko.SSHClient()\n",[154,1688,1690],{"class":156,"line":1689},143,[154,1691,1692],{"class":160},"        # The guest's host key is new for every VM, so there is nothing to\n",[154,1694,1696],{"class":156,"line":1695},144,[154,1697,1698],{"class":160},"        # remember; no known_hosts is loaded, so AutoAdd stays in this process.\n",[154,1700,1702],{"class":156,"line":1701},145,[154,1703,1704],{"class":185},"        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())\n",[154,1706,1708,1711],{"class":156,"line":1707},146,[154,1709,1710],{"class":181},"        try",[154,1712,1193],{"class":185},[154,1714,1716],{"class":156,"line":1715},147,[154,1717,1718],{"class":185},"            client.connect(\n",[154,1720,1722,1725,1727,1729],{"class":156,"line":1721},148,[154,1723,1724],{"class":845},"                hostname",[154,1726,839],{"class":181},[154,1728,384],{"class":250},[154,1730,604],{"class":185},[154,1732,1734,1737,1739,1741],{"class":156,"line":1733},149,[154,1735,1736],{"class":845},"                port",[154,1738,839],{"class":181},[154,1740,404],{"class":250},[154,1742,604],{"class":185},[154,1744,1746,1749,1751,1753],{"class":156,"line":1745},150,[154,1747,1748],{"class":845},"                username",[154,1750,839],{"class":181},[154,1752,450],{"class":250},[154,1754,604],{"class":185},[154,1756,1758,1761,1763],{"class":156,"line":1757},151,[154,1759,1760],{"class":845},"                pkey",[154,1762,839],{"class":181},[154,1764,1765],{"class":185},"key,\n",[154,1767,1769,1772,1774,1777],{"class":156,"line":1768},152,[154,1770,1771],{"class":845},"                look_for_keys",[154,1773,839],{"class":181},[154,1775,1776],{"class":250},"False",[154,1778,604],{"class":185},[154,1780,1782,1785,1787,1789],{"class":156,"line":1781},153,[154,1783,1784],{"class":845},"                allow_agent",[154,1786,839],{"class":181},[154,1788,1776],{"class":250},[154,1790,604],{"class":185},[154,1792,1794,1797,1799,1802],{"class":156,"line":1793},154,[154,1795,1796],{"class":845},"                timeout",[154,1798,839],{"class":181},[154,1800,1801],{"class":250},"10",[154,1803,604],{"class":185},[154,1805,1807,1810,1812,1814],{"class":156,"line":1806},155,[154,1808,1809],{"class":845},"                banner_timeout",[154,1811,839],{"class":181},[154,1813,1801],{"class":250},[154,1815,604],{"class":185},[154,1817,1819,1822,1824,1826],{"class":156,"line":1818},156,[154,1820,1821],{"class":845},"                auth_timeout",[154,1823,839],{"class":181},[154,1825,1801],{"class":250},[154,1827,604],{"class":185},[154,1829,1831],{"class":156,"line":1830},157,[154,1832,1833],{"class":185},"            )\n",[154,1835,1837,1840,1842,1845,1848,1850,1853,1855,1857],{"class":156,"line":1836},158,[154,1838,1839],{"class":185},"            _, stdout, _ ",[154,1841,839],{"class":181},[154,1843,1844],{"class":185}," client.exec_command(",[154,1846,1847],{"class":167},"\"true\"",[154,1849,373],{"class":185},[154,1851,1852],{"class":845},"timeout",[154,1854,839],{"class":181},[154,1856,1801],{"class":250},[154,1858,337],{"class":185},[154,1860,1862,1865,1868,1871,1874],{"class":156,"line":1861},159,[154,1863,1864],{"class":181},"            if",[154,1866,1867],{"class":185}," stdout.channel.recv_exit_status() ",[154,1869,1870],{"class":181},"!=",[154,1872,1873],{"class":250}," 0",[154,1875,1193],{"class":185},[154,1877,1879,1882,1885,1888],{"class":156,"line":1878},160,[154,1880,1881],{"class":181},"                raise",[154,1883,1884],{"class":185}," paramiko.SSHException(",[154,1886,1887],{"class":167},"\"probe command did not run\"",[154,1889,337],{"class":185},[154,1891,1893,1895],{"class":156,"line":1892},161,[154,1894,1199],{"class":181},[154,1896,1897],{"class":185}," client\n",[154,1899,1901,1904,1907,1910,1912,1915,1918,1921],{"class":156,"line":1900},162,[154,1902,1903],{"class":181},"        except",[154,1905,1906],{"class":185}," (paramiko.SSHException, ",[154,1908,1909],{"class":250},"EOFError",[154,1911,373],{"class":185},[154,1913,1914],{"class":250},"OSError",[154,1916,1917],{"class":185},") ",[154,1919,1920],{"class":181},"as",[154,1922,1923],{"class":185}," err:\n",[154,1925,1927,1930,1932],{"class":156,"line":1926},163,[154,1928,1929],{"class":185},"            last ",[154,1931,839],{"class":181},[154,1933,1934],{"class":185}," err\n",[154,1936,1938],{"class":156,"line":1937},164,[154,1939,1940],{"class":185},"            client.close()\n",[154,1942,1944,1947,1949],{"class":156,"line":1943},165,[154,1945,1946],{"class":185},"            time.sleep(",[154,1948,1251],{"class":250},[154,1950,337],{"class":185},[154,1952,1954,1956,1958,1960,1962,1965,1968,1971,1973,1976,1978,1980,1982],{"class":156,"line":1953},166,[154,1955,1259],{"class":181},[154,1957,1262],{"class":250},[154,1959,698],{"class":185},[154,1961,780],{"class":181},[154,1963,1964],{"class":167},"\"no usable ssh to the guest after ",[154,1966,1967],{"class":250},"{SSH_TIMEOUT}",[154,1969,1970],{"class":167},"s: ",[154,1972,786],{"class":250},[154,1974,1975],{"class":185},"last",[154,1977,792],{"class":181},[154,1979,795],{"class":250},[154,1981,798],{"class":167},[154,1983,337],{"class":185},[154,1985,1987],{"class":156,"line":1986},167,[154,1988,175],{"emptyLinePlaceholder":174},[154,1990,1992],{"class":156,"line":1991},168,[154,1993,175],{"emptyLinePlaceholder":174},[154,1995,1997,1999,2002,2005,2008,2011,2013],{"class":156,"line":1996},169,[154,1998,746],{"class":181},[154,2000,2001],{"class":694}," run_turn",[154,2003,2004],{"class":185},"(ssh: paramiko.SSHClient, n: ",[154,2006,2007],{"class":250},"int",[154,2009,2010],{"class":185},", code: ",[154,2012,1101],{"class":250},[154,2014,2015],{"class":185},") -> Result:\n",[154,2017,2019,2022,2024],{"class":156,"line":2018},170,[154,2020,2021],{"class":185},"    stdin, stdout, stderr ",[154,2023,839],{"class":181},[154,2025,2026],{"class":185}," ssh.exec_command(\n",[154,2028,2030,2033,2036,2039,2042,2044,2047,2049,2051,2053],{"class":156,"line":2029},171,[154,2031,2032],{"class":181},"        f",[154,2034,2035],{"class":167},"\"mkdir -p ",[154,2037,2038],{"class":250},"{WORKDIR}",[154,2040,2041],{"class":167}," && cd ",[154,2043,2038],{"class":250},[154,2045,2046],{"class":167}," && python3 -\"",[154,2048,373],{"class":185},[154,2050,1852],{"class":845},[154,2052,839],{"class":181},[154,2054,2055],{"class":250},"TURN_TIMEOUT\n",[154,2057,2059],{"class":156,"line":2058},172,[154,2060,1054],{"class":185},[154,2062,2064],{"class":156,"line":2063},173,[154,2065,2066],{"class":185},"    stdin.write(textwrap.dedent(code))\n",[154,2068,2070],{"class":156,"line":2069},174,[154,2071,2072],{"class":185},"    stdin.channel.shutdown_write()\n",[154,2074,2076],{"class":156,"line":2075},175,[154,2077,175],{"emptyLinePlaceholder":174},[154,2079,2081,2084,2086],{"class":156,"line":2080},176,[154,2082,2083],{"class":185},"    out ",[154,2085,839],{"class":181},[154,2087,2088],{"class":185}," stdout.read().decode()\n",[154,2090,2092,2095,2097],{"class":156,"line":2091},177,[154,2093,2094],{"class":185},"    err ",[154,2096,839],{"class":181},[154,2098,2099],{"class":185}," stderr.read().decode()\n",[154,2101,2103,2106,2108],{"class":156,"line":2102},178,[154,2104,2105],{"class":185},"    result ",[154,2107,839],{"class":181},[154,2109,2110],{"class":185}," Result(stdout.channel.recv_exit_status(), out, err)\n",[154,2112,2114],{"class":156,"line":2113},179,[154,2115,175],{"emptyLinePlaceholder":174},[154,2117,2119,2122,2124,2126,2129,2131,2134,2136,2139,2141,2144,2146,2149],{"class":156,"line":2118},180,[154,2120,2121],{"class":250},"    print",[154,2123,698],{"class":185},[154,2125,780],{"class":181},[154,2127,2128],{"class":167},"\"--- turn ",[154,2130,786],{"class":250},[154,2132,2133],{"class":185},"n",[154,2135,795],{"class":250},[154,2137,2138],{"class":167}," (exit ",[154,2140,786],{"class":250},[154,2142,2143],{"class":185},"result.exit_status",[154,2145,795],{"class":250},[154,2147,2148],{"class":167},")\"",[154,2150,337],{"class":185},[154,2152,2154,2156],{"class":156,"line":2153},181,[154,2155,758],{"class":181},[154,2157,2158],{"class":185}," out:\n",[154,2160,2162,2165,2168,2171,2173,2176],{"class":156,"line":2161},182,[154,2163,2164],{"class":250},"        print",[154,2166,2167],{"class":185},"(out, ",[154,2169,2170],{"class":845},"end",[154,2172,839],{"class":181},[154,2174,2175],{"class":167},"\"\"",[154,2177,337],{"class":185},[154,2179,2181,2183],{"class":156,"line":2180},183,[154,2182,758],{"class":181},[154,2184,1923],{"class":185},[154,2186,2188,2190,2193,2195,2197,2199,2201,2204,2206],{"class":156,"line":2187},184,[154,2189,2164],{"class":250},[154,2191,2192],{"class":185},"(err, ",[154,2194,2170],{"class":845},[154,2196,839],{"class":181},[154,2198,2175],{"class":167},[154,2200,373],{"class":185},[154,2202,2203],{"class":845},"file",[154,2205,839],{"class":181},[154,2207,2208],{"class":185},"sys.stderr)\n",[154,2210,2212,2214],{"class":156,"line":2211},185,[154,2213,806],{"class":181},[154,2215,809],{"class":185},[154,2217,2219],{"class":156,"line":2218},186,[154,2220,175],{"emptyLinePlaceholder":174},[154,2222,2224],{"class":156,"line":2223},187,[154,2225,175],{"emptyLinePlaceholder":174},[154,2227,2229,2231,2234,2237,2239],{"class":156,"line":2228},188,[154,2230,746],{"class":181},[154,2232,2233],{"class":694}," main",[154,2235,2236],{"class":185},"() -> ",[154,2238,2007],{"class":250},[154,2240,1193],{"class":185},[154,2242,2244,2247,2249],{"class":156,"line":2243},189,[154,2245,2246],{"class":185},"    client ",[154,2248,839],{"class":181},[154,2250,2251],{"class":185}," AuthenticatedClient(\n",[154,2253,2255,2258,2260,2262],{"class":156,"line":2254},190,[154,2256,2257],{"class":845},"        base_url",[154,2259,839],{"class":181},[154,2261,361],{"class":250},[154,2263,604],{"class":185},[154,2265,2267,2270,2272,2275,2278],{"class":156,"line":2266},191,[154,2268,2269],{"class":845},"        token",[154,2271,839],{"class":181},[154,2273,2274],{"class":185},"os.environ[",[154,2276,2277],{"class":167},"\"DUMMIE_TOKEN\"",[154,2279,2280],{"class":185},"],\n",[154,2282,2284,2287,2289,2292],{"class":156,"line":2283},192,[154,2285,2286],{"class":845},"        raise_on_unexpected_status",[154,2288,839],{"class":181},[154,2290,2291],{"class":250},"True",[154,2293,604],{"class":185},[154,2295,2297],{"class":156,"line":2296},193,[154,2298,1054],{"class":185},[154,2300,2302],{"class":156,"line":2301},194,[154,2303,175],{"emptyLinePlaceholder":174},[154,2305,2307,2310,2312],{"class":156,"line":2306},195,[154,2308,2309],{"class":185},"    started ",[154,2311,839],{"class":181},[154,2313,2314],{"class":185}," time.monotonic()\n",[154,2316,2318,2321,2323],{"class":156,"line":2317},196,[154,2319,2320],{"class":185},"    vm ",[154,2322,839],{"class":181},[154,2324,2325],{"class":185}," create_vm(client)\n",[154,2327,2329,2331,2333,2335,2338,2340,2343,2345,2348,2350,2353,2355,2358,2361,2363],{"class":156,"line":2328},197,[154,2330,2121],{"class":250},[154,2332,698],{"class":185},[154,2334,780],{"class":181},[154,2336,2337],{"class":167},"\"created ",[154,2339,786],{"class":250},[154,2341,2342],{"class":185},"vm.name",[154,2344,795],{"class":250},[154,2346,2347],{"class":167}," (",[154,2349,786],{"class":250},[154,2351,2352],{"class":185},"vm.id",[154,2354,795],{"class":250},[154,2356,2357],{"class":167},"), ttl ",[154,2359,2360],{"class":250},"{TTL_SECONDS}",[154,2362,1287],{"class":167},[154,2364,337],{"class":185},[154,2366,2368,2371],{"class":156,"line":2367},198,[154,2369,2370],{"class":181},"    try",[154,2372,1193],{"class":185},[154,2374,2376,2378,2380],{"class":156,"line":2375},199,[154,2377,1152],{"class":185},[154,2379,839],{"class":181},[154,2381,2382],{"class":185}," wait_running(client, vm.id)\n",[154,2384,2386,2389,2391],{"class":156,"line":2385},200,[154,2387,2388],{"class":185},"        booted ",[154,2390,839],{"class":181},[154,2392,2314],{"class":185},[154,2394,2396,2398,2400,2402,2405,2407,2410,2412,2414,2416,2419,2422,2425,2428,2430,2432],{"class":156,"line":2395},201,[154,2397,2164],{"class":250},[154,2399,698],{"class":185},[154,2401,780],{"class":181},[154,2403,2404],{"class":167},"\"running at ",[154,2406,786],{"class":250},[154,2408,2409],{"class":185},"vm.ip",[154,2411,795],{"class":250},[154,2413,1281],{"class":167},[154,2415,786],{"class":250},[154,2417,2418],{"class":185},"booted ",[154,2420,2421],{"class":181},"-",[154,2423,2424],{"class":185}," started",[154,2426,2427],{"class":181},":.1f",[154,2429,795],{"class":250},[154,2431,1287],{"class":167},[154,2433,337],{"class":185},[154,2435,2437],{"class":156,"line":2436},202,[154,2438,2439],{"class":160},"        # One connection for every turn, so a turn's cost is a channel rather\n",[154,2441,2443],{"class":156,"line":2442},203,[154,2444,2445],{"class":160},"        # than a handshake, and the guest sees a single session throughout.\n",[154,2447,2449,2452,2454],{"class":156,"line":2448},204,[154,2450,2451],{"class":185},"        ssh ",[154,2453,839],{"class":181},[154,2455,2456],{"class":185}," connect_ssh()\n",[154,2458,2460,2462],{"class":156,"line":2459},205,[154,2461,2164],{"class":250},[154,2463,2464],{"class":185},"(\n",[154,2466,2468,2471,2474,2476,2479,2481,2484,2486,2488],{"class":156,"line":2467},206,[154,2469,2470],{"class":181},"            f",[154,2472,2473],{"class":167},"\"ssh ready after a further ",[154,2475,786],{"class":250},[154,2477,2478],{"class":185},"time.monotonic() ",[154,2480,2421],{"class":181},[154,2482,2483],{"class":185}," booted",[154,2485,2427],{"class":181},[154,2487,795],{"class":250},[154,2489,2490],{"class":167},"s\"\n",[154,2492,2494,2496,2499,2501,2503,2505,2507,2509,2511],{"class":156,"line":2493},207,[154,2495,2470],{"class":181},[154,2497,2498],{"class":167},"\" (",[154,2500,786],{"class":250},[154,2502,2478],{"class":185},[154,2504,2421],{"class":181},[154,2506,2424],{"class":185},[154,2508,2427],{"class":181},[154,2510,795],{"class":250},[154,2512,2513],{"class":167},"s since create)\"\n",[154,2515,2517],{"class":156,"line":2516},208,[154,2518,2519],{"class":185},"        )\n",[154,2521,2523,2525],{"class":156,"line":2522},209,[154,2524,1710],{"class":181},[154,2526,1193],{"class":185},[154,2528,2530,2533,2536,2539,2542,2544,2546,2548,2551,2553,2555],{"class":156,"line":2529},210,[154,2531,2532],{"class":181},"            for",[154,2534,2535],{"class":185}," n, code ",[154,2537,2538],{"class":181},"in",[154,2540,2541],{"class":250}," enumerate",[154,2543,698],{"class":185},[154,2545,536],{"class":250},[154,2547,373],{"class":185},[154,2549,2550],{"class":845},"start",[154,2552,839],{"class":181},[154,2554,1008],{"class":250},[154,2556,704],{"class":185},[154,2558,2560,2563,2566,2568,2570],{"class":156,"line":2559},211,[154,2561,2562],{"class":181},"                if",[154,2564,2565],{"class":185}," run_turn(ssh, n, code).exit_status ",[154,2567,1870],{"class":181},[154,2569,1873],{"class":250},[154,2571,1193],{"class":185},[154,2573,2575,2578,2580,2583,2585,2587,2589],{"class":156,"line":2574},212,[154,2576,2577],{"class":250},"                    print",[154,2579,698],{"class":185},[154,2581,2582],{"class":167},"\"turn failed; stopping\"",[154,2584,373],{"class":185},[154,2586,2203],{"class":845},[154,2588,839],{"class":181},[154,2590,2208],{"class":185},[154,2592,2594],{"class":156,"line":2593},213,[154,2595,2596],{"class":181},"                    break\n",[154,2598,2600,2603],{"class":156,"line":2599},214,[154,2601,2602],{"class":181},"        finally",[154,2604,1193],{"class":185},[154,2606,2608],{"class":156,"line":2607},215,[154,2609,2610],{"class":185},"            ssh.close()\n",[154,2612,2614,2617],{"class":156,"line":2613},216,[154,2615,2616],{"class":181},"    finally",[154,2618,1193],{"class":185},[154,2620,2622],{"class":156,"line":2621},217,[154,2623,2624],{"class":185},"        delete_vm(client, vm.id)\n",[154,2626,2628,2630,2632,2634,2637,2639,2641,2643,2645],{"class":156,"line":2627},218,[154,2629,2164],{"class":250},[154,2631,698],{"class":185},[154,2633,780],{"class":181},[154,2635,2636],{"class":167},"\"deleted ",[154,2638,786],{"class":250},[154,2640,2342],{"class":185},[154,2642,795],{"class":250},[154,2644,798],{"class":167},[154,2646,337],{"class":185},[154,2648,2650,2652],{"class":156,"line":2649},219,[154,2651,806],{"class":181},[154,2653,2654],{"class":250}," 0\n",[154,2656,2658],{"class":156,"line":2657},220,[154,2659,175],{"emptyLinePlaceholder":174},[154,2661,2663],{"class":156,"line":2662},221,[154,2664,175],{"emptyLinePlaceholder":174},[154,2666,2668,2671,2674,2677,2680],{"class":156,"line":2667},222,[154,2669,2670],{"class":181},"if",[154,2672,2673],{"class":250}," __name__",[154,2675,2676],{"class":181}," ==",[154,2678,2679],{"class":167}," \"__main__\"",[154,2681,1193],{"class":185},[154,2683,2685,2687,2690],{"class":156,"line":2684},223,[154,2686,1259],{"class":181},[154,2688,2689],{"class":250}," SystemExit",[154,2691,2692],{"class":185},"(main())\n",[37,2694,2696],{"id":2695},"running-it","Running it",[10,2698,2699,2700,2702],{},"Save the block above as ",[17,2701,148],{}," somewhere the sdk container can see it —\nthe repo keeps no copy of its own, this page is the source — then:",[144,2704,2708],{"className":2705,"code":2706,"language":2707,"meta":150,"style":150},"language-sh shiki shiki-themes github-light github-dark","docker compose exec control-sdk python evaluate.py\n","sh",[17,2709,2710],{"__ignoreMap":150},[154,2711,2712,2715,2718,2721,2724,2727],{"class":156,"line":157},[154,2713,2714],{"class":694},"docker",[154,2716,2717],{"class":167}," compose",[154,2719,2720],{"class":167}," exec",[154,2722,2723],{"class":167}," control-sdk",[154,2725,2726],{"class":167}," python",[154,2728,2729],{"class":167}," evaluate.py\n",[10,2731,2732,2733,55],{},"Mount your private key into the container, or run it on the host with\n",[17,2734,2735],{},"DUMMIE_BASE_URL=http:\u002F\u002Flocalhost:1323\u002Fapi\u002Fv1",[37,2737,2739],{"id":2738},"things-that-will-bite-you","Things that will bite you",[10,2741,2742,2745,2746,2748,2749,2752,2753,2755,2756,2759],{},[115,2743,2744],{},"One VM per user per host."," ",[17,2747,121],{}," routes SSH by public key alone, and the\ncontrol plane writes one entry per VM keyed by its owner's key\n(",[17,2750,2751],{},"control\u002Fproxy_config.go:246-251","). Two live VMs owned by you on the same host\nproduce two entries with the same key, and ",[17,2754,121],{}," rejects the whole config as a\nduplicate rather than picking one (",[17,2757,2758],{},"backstage\u002Fproxy\u002Finternal\u002Fproxy\u002Fresolver.go:48-50",").\nSo this script deletes its VM, and waits for the record to go, before another run\ncreates one. Parallel evaluation needs either a host each or a routing key that is\nnot just the owner.",[10,2761,2762,2765,2766,2769,2770,2773,2774,2777,2778,2781,2782,2785],{},[115,2763,2764],{},"The guest has no egress."," A VM is created with an empty allowlist, so ",[17,2767,2768],{},"pip install"," and any other fetch is refused. Add what a turn genuinely needs with\n",[17,2771,2772],{},"POST \u002Fvms\u002F{id}\u002Ftargets",", which takes its own ",[17,2775,2776],{},"ttl_seconds"," — pass ",[17,2779,2780],{},"targets"," in the\ncreate call to have it in force before the guest is up, rather than as a follow-up\nthe boot can race. ",[17,2783,2784],{},"GET \u002Fvms\u002F{id}\u002Fdenied"," shows what a turn tried to reach.",[10,2787,2788,2794,2795,2798,2799,2802,2803,2806,2807,2810,2811,2814,2815,2817],{},[115,2789,2790,2793],{},[17,2791,2792],{},"202"," is not \"booted\"."," The create returns ",[17,2796,2797],{},"pending","; the host reports the real\noutcome over its own socket afterwards, which is what ",[17,2800,2801],{},"wait_running"," is polling\nfor. And ",[17,2804,2805],{},"running"," is the host's claim as of ",[17,2808,2809],{},"reported_at",", not a live check: it\nsays qemu started, not that sshd is listening or that the host's regenerated proxy\nconfig has landed. Both of those accept a connection before they can serve one, so\n",[17,2812,2813],{},"connect_ssh"," runs a throwaway command and only counts the session as ready once\nthat succeeds. Retrying the handshake alone is not enough — a session opened too\nearly authenticates fine and then dies with ",[17,2816,1909],{}," at the first channel.",[10,2819,2820,2823],{},[115,2821,2822],{},"Only a destroy or a delete frees quota."," Stopping a VM keeps its disk and its\nallowance, and the TTL keeps running while it is stopped.",[10,2825,2826,2745,2832,2834,2835,2838,2839,2842,2843,2846,2847,2849],{},[115,2827,2828,2829,2831],{},"A delete is ",[17,2830,2792],{}," too.",[17,2833,26],{}," sends the same destroy job to the\nhost, so the record survives until the host confirms — and if the destroy fails, so\ndoes the VM, carrying the reason in ",[17,2836,2837],{},"last_error",". That is why ",[17,2840,2841],{},"delete_vm"," polls for\nthe ",[17,2844,2845],{},"404"," rather than treating the ",[17,2848,2792],{}," as the end of the run.",[2851,2852,2853],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":150,"searchDepth":164,"depth":164,"links":2855},[2856,2857,2858,2859],{"id":39,"depth":164,"text":40},{"id":141,"depth":164,"text":142},{"id":2695,"depth":164,"text":2696},{"id":2738,"depth":164,"text":2739},"2026-08-19","Run model-written Python inside a throwaway microVM across several turns, read each result back, then delete the VM.","md",{},"001","\u002Fcasestudies\u002Fsandboxed-evaluation","8 min",{"title":5,"description":2861},"casestudies\u002Fsandboxed-evaluation",[2870,2871,2872,2873],"evals","agents","ssh","python sdk","wqfbaQP-klCt0jq6Ej3_P8160ywru5dZBHTHzBep63w",1787592506959]