Code indexing

When you add a repository to a project, DevSpec reads it and builds a search index of your code. The index is how Dev finds the right files for your question, by name, by path or by what the code does, instead of guessing. DevSpec keeps it up to date as you push.

Indexing starts by itself when you add a repository in Project settings → Repositories. That page is also where you follow its progress and fix problems.

What indexing gives you

With an indexed repository, Dev can answer questions about your code from the code itself. When you ask "where do we handle refunds?" or "what calls this function?", Dev searches the index, then reads the files it found and answers with the real lines.

For each file it indexes, DevSpec records:

• Its structure: the functions, classes, interfaces, types, methods and variables it defines, and what it imports. This lets Dev find a symbol by name and trace which files depend on which.
• Its meaning: the file's content, split into passages and made searchable by what it does, so a question in plain words can find code that never uses those words.

One search combines exact names, file paths and meaning, and covers every repository in the project at once.

Your agents read the code in their own local checkout, so they don't need the index. The index is what lets Dev, working in DevSpec, see the same code.

Which files DevSpec indexes

DevSpec indexes source code and the text files around it. How much it understands depends on the language.

Some languages aren't indexed at all, including Ruby, PHP, C#, C and C++, Kotlin, Swift, Scala, Terraform and HCL, and XML. Plain .txt files aren't indexed either. Dev can still read any file in a tracked repository directly when you point it at the path; it can't find those files by searching.

Images, fonts, PDFs, Office documents and other binary files are skipped. To make a PDF or Word document searchable, upload it on the project's Resources page instead. See Documents and artifacts.

Which branch is indexed

DevSpec indexes each repository's Tracking branch, and Dev searches that branch by default. The tracking branch is pre-filled with your repository's default branch when you add it.

To make another branch searchable, open the repository card's branches section and click Track beside it. DevSpec indexes it and keeps it current too, and Dev can search it when you ask about that branch by name. Dev can also read a file from a branch that isn't indexed, and compare branches, by reading them from GitHub directly.

See Repositories and GitHub.

Skip files and folders with ignore patterns

Each repository card has an Indexing section with the list of paths DevSpec skips in that repository. Project Maintainers can edit it.

• Put one pattern per line. A bare name, such as dist, matches any folder or file with that name. Globs work too, such as *.generated.*.
• The default list skips build output, framework caches, vendored dependencies, virtual environments, coverage reports, editor folders, tmp and logs folders, and generated or bundled files. Most projects never need to change it.
• Restore defaults puts the default list back.
• Changes apply the next time the repository is indexed. To apply them to the whole repository straight away, use Full reindex on the card.

Some things are always skipped and can't be turned back on: node_modules, .git, lockfiles, binary files, minified files, and actual environment files such as .env and .env.local.

Environment templates such as .env.example, .env.sample and .env.production.example can be indexed, so Dev can find the configuration your project expects. Your repository's ignore patterns still apply to them.

Skipping large generated folders keeps the index focused on code your team wrote, and keeps indexing costs down.

Follow indexing progress

While a repository is indexing, its card in Project settings → Repositories shows a progress bar with a percentage and an estimated time remaining. Under the bar, two counts show how far each part has got: Parsing (reading each file's structure) and Embedding (making each file searchable by meaning). An update after a push shows Updating N files... instead of reindexing everything.

During a repository's first index, the project's Overview shows a Reading your code card with progress, and the session composer shows "Dev is still reading your code". You can start a session straight away; Dev answers questions about the code once the first index finishes.

When a repository is being updated later, for example after a push, the session header shows Indexing... with a tooltip naming the repository and noting "Responses may have incomplete context". Dev's answers about recently changed files may be out of date until the update finishes.

How long indexing takes depends on the size of the repository. The estimate on the card updates as indexing goes.

How DevSpec keeps the index current

After the first index, DevSpec only re-reads what changed. When you push to a tracked branch, DevSpec compares each file with the version it already has and indexes only the new and changed files, and removes deleted ones.

DevSpec also checks each repository for new commits periodically in the background, so a push it didn't hear about still gets picked up. The repository card shows how many commits it's behind, and a Maintainer can click Check for updates to check now.

Full reindex throws the index away and reads every file again. Use it after changing ignore patterns, or if search results look wrong.

When indexing fails

If some files couldn't be indexed, the repository card says Indexed with N failed files. The rest of the repository is searchable. Click Retry failed files to try them again.

If indexing fails outright, the card says what happened and whose problem it is:

• This one needs you: DevSpec needs something only you can give it, usually renewed access to GitHub. The message says what.
• The provider is having problems: GitHub is having trouble. You don't need to do anything.
• This is on our side: DevSpec's problem. You don't need to do anything.

An index that stalls is marked failed and retried automatically. To try again yourself, click Retry clone & index. To stop an index that's running, click Cancel indexing. These controls are for project Maintainers.

What indexing costs

The first full index of your organization's first repository is free.

After that, indexing is background work charged to your organization's balance, per piece of work, in dollars. It doesn't count toward anyone's personal allowance, though it does count toward the project's limit if it has one. On Settings → Billing → Transactions it appears as Running your project, which covers indexing new and changed files, full reindexes, a daily charge for storing your organization's search index, and keeping the project's own records, such as tasks, memories and your database schema, searchable.

While the project is paused, for example because your organization's balance ran out, indexing waits. When the project starts again, it brings the index up to date with your latest code in one pass. See Billing, usage and limits.