LibreChat for Earlham College ITS
Earlham ITS & System Administration team | Primary maintainer | Fall 2025-present
I collaborate directly with Earlham ITS department and the System Administration team to build Earlham's chat platform, a custom LibreChat deployment for students. As its primary maintainer, I develop features and run the service, with occasional contributions from two teammates. My work gives students access to online and Earlham-hosted models, isolated coding workspaces, and API keys for their own tools without needing a GPU on their laptop.
chat.cluster.earlham.edu is an internal beta for Earlham students, requires Earlham SSO, and has about 150 daily users. Students are sharing it with each other and usage is growing gradually. A move to a domain such as chat.earlham.edu or ai.earlham.edu is expected when the platform is publicly announced (neither is the current address).
Local models students can use in their own tools
I built the Local model API keys page in LibreChat's account menu. Students can create a key, choose allowed models and an expiry, rename it, or revoke it. The compute runs on Earlham clusters, so their laptop doesn't need a GPU.
I added an administrator CLI, OpenCode configuration, and a desktop setup video. My supervisor, Porter Libby, and other users tested the key workflow and gave positive feedback on how easy it was to create and manage keys. Use from an external machine was also confirmed.
The system allows ten keys per user, with 300 requests per minute and four concurrent requests per key. Secrets are shown once; ownership records contain safe metadata instead of full keys. I added origin checks, browser security headers, and server-side ownership enforcement.
Workspaces, files, and smaller patches
On May 31, 2026, I added sandboxed workspaces for model-written commands and code. My work included file operations, project scanning, CodeGraph, upload import, and authenticated downloads and HTML previews.
I isolated users with separate filesystem identities and restricted the runner with no network, a read-only root filesystem, resource limits, and a 120-second command timeout. Workspaces have 15 GiB quotas, usage warnings, and 24-hour cleanup. Existing PyTorch and TensorFlow environments are mounted read-only, and project scans exclude .env files.
In September, I adopted upstream workspace support where applicable and reduced custom patching, retaining the isolated runner and file controls described in my deployment notes.
- Uploads: fixed a configuration issue in workspace import and verified it with a roughly 35 MB video.
- Private files: tested owner downloads and rejection of anonymous and other-user access.
- PDFs: added layout-aware extraction with anymd and scanned-document OCR with Tesseract inside private workspaces, then removed the older parser patch.
Making local inference fit the machines
After Porter told me about additional compute nodes and GPUs in August, I made a major inference update in September. I migrated from Ollama to native llama.cpp and removed redundant inference services. On October 9, I updated llama.cpp again.
- Loading: staged GGUF models and vision projectors on each node's local disk, preloaded Gemma 4 26B at startup, and kept loaded models warm between requests.
- Routing: ran separate servers on f4 and f5, each using two A5000 GPUs. Requests are balanced across replicas and retried when a replica fails. Each node keeps one model loaded; the router waits for active requests to finish before unloading an idle model to make room.
- Memory: kept model layers in VRAM. For the two 80B models, I moved q8 context caches to RAM and used a 48%/52% GPU split to leave room for compute buffers. Larger models use one concurrent slot; others use two.
- Conversation overhead: used LFM 2.5 8B for local chat titles and enabled native context compaction to reduce processing of accumulated history.
This work taught me to consider model weights, context caches, and concurrency together. A model loading successfully is only the first check; it also needs room to answer.
Upgrades I can recover from
I upgraded LibreChat to 0.8.8-rc4 and removed obsolete compatibility patches. After backing up MongoDB, I resolved a unique-message-index failure by removing 27 stale unfinished duplicates while preserving completed messages.
I created a core backup covering databases, user files, images, and deployment configuration, and tested database recovery in isolated containers. In September, I reduced the main runtime patch from 688 to 582 lines at that stage and checked attachment behavior across 3,712 cases.
I also maintain the $3 per-user daily online-model budget and pricing logic, configured a shared coding team with restrictions on expensive child models, and migrated four private agents to the newer permissions system. I replaced failing search with Firecrawl while preserving web fetch, and documented scoped deployment, verification, and rollback procedures.