github-clone

Public

Open-source GitHub clone: git repos, pull requests, issues, wiki, AI workspace, CLI and MCP. MIT.

Harakumo1 commitUpdated 11 days ago2 downloads

Loading the docs…

Loading the files…

System designdocs/SYSTEM-DESIGN.mdView source

System design

This document describes what the code in this repository does today: the data model, the main flows, who may do what, the limits, how failures are handled, security, and how far it scales. Where something is not built, it says so.

1. Data model

Data lives in four places:

MongoDB, through the Mongoose models in src/models/. Collection names below are Mongoose's defaults for each model name.
Repository disk: one git repository with a working tree per repository, at $NUCLEUS_REPOS_DIR/<owner>/<repo>.git (default ./repos).
S3-compatible bucket (optional): repos/<owner>/<repo>.tar.gz, one snapshot per repository, replaced on every write.
Clients: the browser's localStorage (AI chat history, architecture analyses, generated presentations), the CLI's ~/.nucleus/config.json (token, username, serverUrl) and each project's .nucleus.json (repo, serverUrl, baseCommit).

All models except OtpCode, AgentTeam and AgentDefinition use Mongoose timestamps, which adds createdAt and updatedAt (Date). OtpCode stores only createdAt. Every document also has _id (ObjectId).

users — `src/models/User.ts`

Field	Type	Notes
`username`	String	Required, unique, lowercase, 2 to 39 characters, letters, digits and single hyphens
`email`	String	Unique (sparse), lowercase. Optional in the schema; the sign-up route requires it
`passwordHash`	String	bcrypt, cost 12
`emailVerified`	Boolean	Default false. Set by a verify code or a password reset
`tokenVersion`	Number	Default 0. Bumped on password reset to revoke CLI and MCP tokens
`displayName`	String	Default empty
`bio`	String	Max 256
`avatar`, `location`, `website`, `company`	String	Default empty
`repos`	ObjectId[] → Repository	Repositories the user created or forked
`starredRepos`	ObjectId[] → Repository
`followers`, `following`	ObjectId[] → User

organizations — `src/models/Organization.ts`

Field	Type	Notes
`name`	String	Required, unique
`slug`	String	Required, unique, lowercase. Must not equal an existing username
`displayName`, `avatar`, `website`, `location`, `email`	String	Default empty
`description`	String	Max 500
`owner`	ObjectId → User	Required
`members`	Array of `{ user: ObjectId → User, role: 'owner' or 'admin' or 'member', joinedAt: Date }`
`repos`	ObjectId[] → Repository	No route writes this field

repositories — `src/models/Repository.ts`

Field	Type	Notes
`name`	String	Required, max 100
`slug`	String	Required, lowercase. Unique together with `owner`
`description`	String	Max 500
`owner`	ObjectId	Required. Points to a User or an Organization through `ownerType`
`ownerType`	'User' or 'Organization'	Default 'User'. Every route that creates a repository sets 'User'
`visibility`	'public' or 'private'	Default public
`defaultBranch`	String	Default 'main'
`language`	String	Most common file type in the last push through the upload route
`languages`	Array of `{ name: String, percentage: Number }`	Copied on fork
`collaborators`	Array of `{ user: ObjectId → User, role: 'admin' or 'write' or 'read', addedAt: Date }`	Default role write
`stars`	ObjectId[] → User
`starCount`, `forkCount`, `openIssuesCount`	Number	Counters kept by the star, fork and issue routes
`forkedFrom`	ObjectId → Repository	Default null
`topics`	String[]
`readme`, `license`	String	Default empty
`size`	Number	The schema comment says KB; the upload route stores the total character length of the pushed content
`secrets`	Array of `{ key: String, encryptedValue: String, addedAt: Date }`	AES-256-GCM ciphertext, see Security

Indexes: { owner, slug } unique, starCount, updatedAt, topics, visibility.

issues — `src/models/Issue.ts`

Field	Type	Notes
`title`	String	Required, max 256
`body`	String
`number`	Number	Required. Unique per repository, assigned as last number + 1
`repository`	ObjectId → Repository	Required
`author`	ObjectId → User	Required
`assignees`	ObjectId[] → User
`labels`	Array of `{ name: String, color: String }`
`status`	'open' or 'closed'	Default open
`comments`	Array of `{ author: ObjectId → User, body: String, createdAt: Date }`	Embedded

pullrequests — `src/models/PullRequest.ts`

Field	Type	Notes
`title`	String	Required, max 256
`body`	String
`number`	Number	Required, unique per repository
`repository`	ObjectId → Repository	Required
`author`	ObjectId → User	Required
`status`	'open' or 'closed' or 'merged'	Default open
`sourceBranch`	String	Required
`targetBranch`	String	Default 'main'
`assignees`, `reviewers`	ObjectId[] → User
`labels`	Array of `{ name, color }`
`comments`	Array of `{ author, body, createdAt }`	Embedded
`closedAt`, `mergedAt`	Date

discussions — `src/models/Discussion.ts`

Field	Type	Notes
`title`	String	Required, max 256
`body`	String
`number`	Number	Required, unique per repository
`repository`, `author`	ObjectId	Required
`category`	'announcements', 'ideas', 'qa', 'show-and-tell', 'general' or 'bugs'	Default general
`pinned`, `locked`, `answered`	Boolean	Default false. Only `pinned` is changed by a route
`upvotes`	ObjectId[] → User
`labels`	Array of `{ name, color }`
`comments`	Array of `{ author, body, upvotes: ObjectId[], isAnswer: Boolean, createdAt }`	Embedded

projects — `src/models/Project.ts`

Field	Type	Notes
`title`	String	Required, max 256
`description`	String
`repository`, `author`	ObjectId	Required
`columns`	String[]	Default To Do, In Progress, Done
`cards`	Array of `{ title: String, description: String, column: String, order: Number, assignee: ObjectId → User, linkedIssue: Number, createdAt: Date }`	Embedded

wikipages — `src/models/WikiPage.ts`

Field	Type	Notes
`title`	String	Required, max 256
`slug`	String	Required, from the title. Unique per repository
`content`	String	Markdown
`repository`, `author`	ObjectId	Required
`lastEditedBy`	ObjectId → User
`order`	Number	Position in the sidebar

actions — `src/models/Action.ts`

Field	Type	Notes
`repository`	ObjectId	Required
`owner`	ObjectId → User	Required, the creator
`name`	String	Required
`description`, `instruction`	String	`instruction` is added to the agent prompt
`trigger`	'on_push', 'on_pr', 'on_issue', 'on_schedule' or 'manual'	Required
`agentSlug`	String	Required, a slug from the agent catalog
`enabled`	Boolean	Default true
`schedule`	String	A cron expression, stored only
`lastRunAt`	Date
`runs`	Array of runs	Embedded, see below

A run is { status: 'running' or 'success' or 'failed' or 'cancelled', startedAt: Date, completedAt: Date, summary: String, triggeredBy: String, triggerDetail: String, agentOutput: String, tasksCreated: Number, issuesFound: Number }.

agentteams — `src/models/AgentTeam.ts`

Field	Type	Notes
`repoId`	ObjectId	Required. Unique together with `agentSlug`
`agentSlug`	String	Required
`status`	'active', 'paused' or 'idle'	Default idle. Set to active during a run
`hiredAt`	Date
`hiredBy`	ObjectId → User	Required
`config`	`{ model: 'claude' or 'grok', tokenBudget: Number, autoCommit: Boolean }`	Defaults claude, 100000, false. Only `model` is read by the server
`stats`	`{ tasksCompleted, filesCreated, tokensUsed, issuesFound: Number }`	`tokensUsed` is never incremented

agenttasks — `src/models/AgentTask.ts`

Field	Type	Notes
`repoId`	ObjectId	Required
`title`	String	Required
`description`	String
`status`	'todo', 'in_progress', 'review' or 'done'	Default todo
`priority`	'low', 'medium', 'high' or 'critical'	Default medium
`assignedAgent`, `createdByAgent`	String	Agent slugs
`branch`	String
`relatedFiles`	String[]

agentactivities — `src/models/AgentActivity.ts`

repoId (ObjectId, required), agentSlug (String, required), action (String, required, for example hired, created_task, opened_issue, executed, action_triggered), description (String, required), metadata (Mixed).

agentcomms — `src/models/AgentComms.ts`

repoId (ObjectId, required), fromAgent (String, required, an agent slug or user), toAgent (String, null means everyone), content (String, required), attachments (Array of { type: 'task' or 'file' or 'issue' or 'review', ref: String, title: String }).

otpcodes — `src/models/OtpCode.ts`

Field	Type	Notes
`user`	ObjectId → User	Required, indexed
`purpose`	'verify_email' or 'password_reset'	Required
`codeHash`	String	HMAC-SHA256 of user, purpose and code. The code itself is never stored
`sentTo`	String	Address the code went to
`attempts`	Number	Wrong guesses so far
`expiresAt`	Date	Required. A TTL index deletes the document when it passes

activities — `src/models/Activity.ts`

user (ObjectId → User, required), type ('create_repo', 'commit', 'star', 'fork', 'issue_open', 'issue_close', 'comment' or 'follow'), repository (ObjectId → Repository), metadata (Mixed). The model is registered, but no route writes or reads it yet.

agentdefinitions — `src/models/AgentDefinition.ts`

slug (unique), name, role, icon, color, description, capabilities (String[]), systemPrompt, category (all String) and tier ('free' or 'pro'). The schema exists, but nothing queries it: the eight agents (project manager, developer, QA engineer, security analyst, DevOps engineer, technical writer, code reviewer, UI/UX reviewer) are the AGENT_CATALOG constant in the same file, all tier: 'free'.

Note: Action, AgentTeam, AgentTask, AgentActivity and AgentComms declare ref: 'Repo', a model name that is not registered. No route populates those references, so it has no effect today.

2. Main flows

Sign up — `POST /api/auth/register`

Rate limit by client IP (auth.register).
Require name, username, email and password. Validate the email shape, the username pattern and a password of at least 8 characters.
Reject a taken username or email with 409.
Hash the password with bcrypt (cost 12) and create the user.
If mail is configured, issue a verify_email code and send it. A failed send does not fail the sign-up; the response reports verificationSent.

Sign in — NextAuth credentials (`src/lib/auth.ts`)

Rate limit login:<username>, and login:<client IP> when TRUST_PROXY=true.
Look up the user and compare the password with bcrypt.
Issue a JWT session cookie carrying userId and username.

Password reset — `/forgot-password`

POST /api/auth/otp with purpose: 'password_reset' and a username or email. Rate limited per IP and per account.
The response is the same whether or not the account exists. If it exists and has an email, any earlier reset code is deleted, a new 6-digit code is stored as a hash, and the email is sent in the background so timing does not reveal the account.
POST /api/auth/password with username, code and new password. Rate limited per IP, then per account for code attempts.
verifyOtp checks expiry, the attempt count and the hash (timing-safe), then deletes the code.
Save the new bcrypt hash, mark the email verified, increment tokenVersion, clear the sign-in limit for the account, and send a "password changed" email.

CLI sign-in — `nucleus login`

The CLI calls GET /api/health and checks service: 'nucleus', so a wrong URL is reported as such.
It listens on a random port on 127.0.0.1 and opens /api/auth/cli-token?port=N in the browser.
The server checks the port is digits in 1024 to 65535. If the browser is not signed in it redirects to /login and back.
The server signs { userId, username, tv } for 30 days and redirects to http://127.0.0.1:N/callback with the token.
The CLI saves the token to ~/.nucleus/config.json with mode 600.

Create a repository — `POST /api/repos`

Identify the caller (session or bearer token).
Slugify the name and reject a duplicate for the same owner.
initBareRepo runs git init in <owner>/<slug>.git, writes a README and commits it.
Start a background snapshot, create the MongoDB record, and add it to the user's repos.

nucleus init only writes .nucleus.json; the first nucleus push creates the repository through the upload route if it does not exist.

Push from the CLI — `POST /api/repos/[owner]/[repo]/upload`

The CLI walks the folder, honors .gitignore plus a built-in ignore list, skips files over 25 MB, and sends text as UTF-8 and anything else as base64. It sends baseCommit from .nucleus.json and completeTree: true. nucleus push also sends partialUpload: true when it skipped a file; nucleus sync does not.
The route verifies the bearer token and its tokenVersion (or falls back to the browser session), and requires the caller to be the owner.
It checks the push quota (file count and sizes).
It finds the repository record by slug or creates it (and the git repository) on first push.
planPush compares three trees: the repository at baseCommit, the repository now, and the upload. Files changed on both sides, differently, are conflicts. Without a baseCommit, or when it is unknown, the upload is taken as is.
Conflict: 409 with each file's three versions. nucleus sync then offers keep mine, take theirs, or save both as .theirs files.
Otherwise commitFiles runs under the repository lock: it validates every path first, writes the files, stages them, removes tracked files missing from the upload only when completeTree is set and nothing was skipped, and commits.
A background snapshot goes to the bucket, the record's language and size are updated, and the new head is returned for the CLI to store.

Edit with the AI workspace — `/chat`

Pick a repository. The page loads every file and the head commit with GET /api/repos/[owner]/[repo]/files.
A prompt goes to POST /api/ai/chat with the file list and the open file. The route rate limits, picks the provider, and asks for files in ===FILE: path=== blocks and deletions as ===DELETE: path===.
The parsed files appear in the editor marked unsaved. Deletions only remove files from the editor's list.
Save sends the changed files to POST /api/repos/[owner]/[repo]/files (write access, push rate limit). This route never deletes files.
The route commits under the lock, fires the repository's on_push actions in the background, and snapshots the repository.

Pull requests

POST .../branches creates a branch from another ref without checking it out.
POST .../pulls (any signed-in user who can read the repository) records the title, source and target branch.
GET .../compare?base=&head= reads both trees and returns an LCS line diff per file plus the commits on head that are not on base.
PATCH .../pulls with action: 'merge' (write access) runs an isomorphic-git merge that fast-forwards or creates a merge commit and aborts on conflict, then checks the target branch out so the working tree matches. The PR is marked merged and a snapshot is taken.

Restore after a wiped disk

repoAccess and repoAccessById call ensureRepoLocal before anything reads a repository. If the directory is missing and a bucket is configured, the snapshot is downloaded and extracted. Concurrent requests for the same repository wait on the same restore. The upload route does not go through repoAccess, so it does not trigger a restore by itself.

Previews

Static: a repository with index.html and no package.json is served from GET /api/repos/[owner]/[repo]/serve/... with a sandbox Content Security Policy, so the page cannot act on the Nucleus origin.
In the browser: a repository with package.json is fetched with /download and mounted into a WebContainer in the visitor's browser.
Container: Python, Ruby, Go, Java, Rust, PHP and Flutter projects can start a Docker container (POST /api/preview/container, write access). The files are written to a temporary folder, an install and start script runs, and the container is published on a host port starting at 9100. It stops after 30 minutes. Without a Docker socket the route answers 503.

AI agents and actions

A writer hires an agent from the catalog onto a repository (POST /api/agents/team).
POST /api/agents/run builds a prompt from the repository's name, description, tasks and the last 20 messages, calls the model the agent is configured with, and parses ===TASKS===, ===ISSUES===, ===COMMS===, ===STATUS=== and ===FILE: blocks.
Tasks, messages and activity entries are stored. Task status updates are applied. File blocks are returned to the page but not committed.
Actions run the same kind of prompt through triggerAction on on_push (web saves through the files route), on_issue (new issues) and manual (the Run button). Each run is appended to the action's runs.

3. Roles and permissions

All repository routes resolve access through repoAccess() in src/lib/authz.ts. A private repository the caller cannot see answers 404, not 403, so its existence is not disclosed.

Level	Who	Used for
read	Anyone for a public repository. For a private one: the owner and every collaborator	Browse files, commits, branches, compare, download, ZIP, static preview files, issues, PRs, discussions, wiki, projects, actions list, agent data, fork
contribute	Signed in and able to read	Open, comment on, close and reopen issues. Open PRs. Create, comment on, upvote and pin discussions. Star
write	The owner, or a collaborator with role write or admin	Save files through the `files` route, create branches, close, merge, reopen and comment on PRs, wiki, projects, actions, architecture analysis, container previews, hire and run agents, agent tasks and messages
admin	The owner, or a collaborator with role admin	Repository settings and collaborators, list secret names

Extra rules on top of those levels:

Adding, deleting or decrypting a secret requires the actual owner.
Deleting an issue requires the repository owner or the issue's author.
Deleting a repository requires the signed-in owner.
The upload route (CLI push, the repository page's AI chat, saving a presentation) accepts only the owner, so collaborators write through the files route (the web workspace and the MCP write_files tool).
Organizations: anyone can read an organization. Its owner, or a member with role owner or admin, can edit its details and members.
Profiles: only the user can edit their own profile. Other people never see the email address, emailVerified or tokenVersion.

4. Limits

Rate limits (src/lib/ratelimit.ts, in memory, fixed windows):

Key	Limit	Applied in
`ai.chat`	30 per hour per user	`api/ai/chat`
`ai.chat.daily`	200 per day per user	`api/ai/chat`
`ai.agent`	20 per hour per user	`api/agents/run`
`ai.architecture`	10 per hour per user	`architecture`
`preview`	10 per hour per user	`api/preview/container` POST
`push`	120 per hour per user	`files` POST
`auth.attempt`	10 per 15 minutes	Sign-in, per username and per IP
`auth.register`	5 per hour per IP	Sign-up
`otp.request`	5 per hour	Code requests, per IP and per account
`otp.verify`	10 per 15 minutes per account	Password reset
`password.reset`	5 per hour per IP	Password reset
`repo.create`, `write`	20 and 300 per hour	Defined; no route applies them

Size and other caps:

What	Cap	Where
One file in a push	25 MB	`src/lib/quota.ts` (upload route) and the CLI
One push	100 MB	Upload route
Repository after a push	500 MB	Upload route
Files in one push	20,000	Upload route
One-time codes	6 digits, 10 minutes, 5 wrong guesses	`src/lib/otp.ts`
CLI and MCP tokens	30 days	`src/lib/jwt.ts`
AI output	16,384 tokens for chat and architecture analysis, 8,192 for agents and actions	`src/lib/ai.ts` callers
Architecture input	About 50 KB, 3 KB per file	`architecture` route
Diffs	Files over 512 KB or binary are skipped; 3,000 lines per file	`src/lib/diff.ts`
Container previews	512 MB memory, 1 CPU, 30 minutes	`src/lib/preview/docker-manager.ts`
Passwords	At least 8 characters	Sign-up and reset

The files route (web saves and MCP writes) does not apply the size quota. Action runs are not counted against ai.agent.

5. Failure handling

Missing configuration. In production, src/instrumentation.ts stops the server at boot if MONGODB_URI, AUTH_SECRET or SECRETS_ENCRYPTION_KEY is missing, and warns about missing NUCLEUS_REPOS_DIR, HARAKUMO_API_KEY, STORAGE_* and TRUST_PROXY. The modules that use those secrets also fail closed on use.
Health. GET /api/health returns 503 unless MongoDB is connected and the repository directory is writable. The Docker image's HEALTHCHECK and docker-compose.yml use it.
Database connection. One cached connection per process. A failed connect clears the cache so the next request retries.
Push conflicts return 409 with the files and their versions. Nothing is written.
Bad paths in a commit reject the whole batch before any file is written.
Merge conflicts on a PR return 409 with the conflicting paths; the merge is aborted.
Backups log and swallow errors, so a storage problem never fails a push. Restores never throw; when one fails, the route that needed the files fails as if the repository were missing.
Mail never throws. Sign-up succeeds even when the verification email fails. A reset code is sent in the background and a failure is only logged.
AI. If the chosen provider has no key, the other is used; with neither, the chat route returns the configuration error and the architecture route returns a basic outline built from the file list. A Grok answer cut off while reasoning raises a clear error. A failed agent run resets the agent to idle; a failed action run is stored with status failed and the error.
Decryption of a secret with the wrong key or tampered data returns an empty string, never partial plaintext.
Locks. A failing write releases the repository lock for the next one.
Previews. No Docker socket answers 503 "not available". A container that fails to start has its temporary folder removed.
Process restart clears every in-memory structure: rate-limit counters, locks, in-flight restores, and the map of running preview containers with their 30-minute stop timers.
Numbering. Issue, PR and discussion numbers are "last + 1". Two simultaneous creates can pick the same number; the unique index rejects the second and the route answers 500.

6. Security

Passwords are bcrypt hashes (cost 12). Sign-in is rate limited per username and, behind a trusted proxy, per IP.
Sessions are NextAuth JWT cookies signed with AUTH_SECRET.
CLI and MCP tokens are JWTs signed with the same secret, valid 30 days, and stamped with tokenVersion. A password reset increments it, so every earlier bearer token stops working. Browser session cookies do not carry the version and are not revoked by a reset.
No development fallbacks in production. The signing key and the encryption key refuse to fall back to the defaults in the source when NODE_ENV=production.
Repository secrets use AES-256-GCM with a key derived from SECRETS_ENCRYPTION_KEY, kept separate from AUTH_SECRET so rotating one does not destroy the other. Older AES-256-CBC values still decrypt. Listing returns names only. The AI receives secret names, never values. The owner can fetch decrypted values with the decrypt_for_deploy action, which remains from a deployment feature that is not part of this release.
Private repositories. One access check for every repository route, 404 for repositories the caller cannot see, and secrets and collaborators excluded from the repository list and detail payloads.
Paths. Read paths go through resolveReadPath and write paths through assertSafeRepoPath: no absolute paths, nothing outside the repository and nothing under .git. tests/path-traversal.test.mts covers this.
User files served from the Nucleus origin carry Content-Security-Policy: sandbox ... and nosniff.
Headers on every page: X-Content-Type-Options, Referrer-Policy, Permissions-Policy, X-Frame-Options: SAMEORIGIN, and HSTS in production. The WebContainer preview page also gets COOP and COEP.
One-time codes are random (crypto.randomInt), stored as an HMAC peppered with AUTH_SECRET, bound to one purpose, replaced on every new request, and compared in constant time. Reset responses do not reveal whether an account exists.
CLI login redirect accepts only a numeric port and always redirects to 127.0.0.1, so the token cannot be sent to another host.
Client IPs are read from X-Forwarded-For only when TRUST_PROXY=true, and then from the last hop.
Container previews need write access and run with memory and CPU caps. Mounting the Docker socket gives the Nucleus container root-equivalent access to the host, as DEPLOYING.md warns.
The Docker image runs as an unprivileged nucleus user.

7. How it scales

One instance. Locks, rate limits, in-flight restores and the preview container map live in process memory. A second instance would not see them, and two pushes to the same repository could interleave. Scale up, not out. Comments in src/lib/locks.ts, src/lib/ratelimit.ts and src/lib/storage/backup.ts name the next steps: a shared lock and limit store (Redis or MongoDB), and object storage as the source of truth with local disk as a cache.
Whole-repository reads. getRepoFiles reads every file into memory for /files, /download, /zip, forks, container previews and architecture analysis, so cost grows with repository size. The 500 MB push cap bounds it.
Backups upload a full tar.gz of the repository on every write and keep only the latest snapshot per repository.
MongoDB. Indexes cover the common lookups (owner and slug, repository and number, repository and status). Comments, cards and action runs are embedded arrays that grow inside one document.
Container previews use one host port per preview, counting up from 9100 for the life of the process.

8. Known gaps

Plain git clone and git push over HTTP are not supported. Use the web app, the CLI or the MCP server.
New repositories are initialised with isomorphic-git's default branch name, master, while the database record, the PR form and several routes default to main.
Every write path commits to the branch checked out in the working tree. Creating a branch does not check it out, and no route commits to another branch. The file browser reads the working tree and ignores ref.
Agent runs and actions build their prompt from a fileTree field that the Repository schema does not have, so the model sees no file list or file contents. Agent file output is returned to the page but not committed.
on_pr and on_schedule actions can be created, but nothing fires them.
No route creates an organization-owned repository. The organization insights, packages, projects, security and teams pages are placeholders.
AI deletions in /chat change the editor's file list only.
The automated tests are four scripts in tests/ (push integrity, OTP, path traversal, durability) that exercise library code. The API routes and pages have no tests.

github-clone

System design

1. Data model

users — src/models/User.ts

organizations — src/models/Organization.ts

repositories — src/models/Repository.ts

issues — src/models/Issue.ts

pullrequests — src/models/PullRequest.ts

discussions — src/models/Discussion.ts

projects — src/models/Project.ts

wikipages — src/models/WikiPage.ts

actions — src/models/Action.ts

agentteams — src/models/AgentTeam.ts

agenttasks — src/models/AgentTask.ts

agentactivities — src/models/AgentActivity.ts

agentcomms — src/models/AgentComms.ts

otpcodes — src/models/OtpCode.ts

activities — src/models/Activity.ts

agentdefinitions — src/models/AgentDefinition.ts