Introduction
Understanding an unfamiliar codebase is one of the first challenges developers face when joining a new project. Before making meaningful changes, you need to understand its architecture, dependencies, entry points, and execution flow.
This challenge becomes more interesting when working with AI coding agents. An agent needs relevant repository context to make useful decisions. Providing an entire repository as context is often inefficient, while manually selecting important files takes time.
I started building RepoLens, a static codebase intelligence tool designed to analyze repositories and generate structured information for developers and AI coding assistants.
The goal was straightforward: help developers understand unfamiliar projects through deterministic static analysis, dependency mapping, and evidence-based architectural insights.
This article explores the engineering decisions behind RepoLens, including dependency graph traversal, filesystem security, multi-language analysis, and performance testing.
1. Designing a Static Codebase Analysis Pipeline
RepoLens uses a multi-stage architecture. Instead of executing project code, it inspects source files and extracts information through static analysis.
The main components are:
- CLI: Handles commands and user input.
- Discovery Engine: Scans directories, applies ignore rules, and identifies programming languages.
- AST Analyzers: Extracts imports, symbols, and structural information.
- Dependency Graph: Resolves imports and identifies relationships between modules.
- Intelligence Layer: Detects entry points, architectural layers, hotspots, and reading order.
- Presentation Layer: Generates terminal reports, Mermaid diagrams, JSON output, and AI context files.
The execution pipeline follows this sequence:
- Discover repository files.
- Classify files by language and type.
- Analyze supported source files.
- Resolve module dependencies.
- Build a directed dependency graph.
- Generate architectural insights.
- Export structured reports.
For Python, RepoLens uses the standard library's ast module. JavaScript and TypeScript analysis uses Tree-sitter, with a regex fallback when the primary parser is unavailable or fails.
This approach avoids importing or executing the scanned project's source code.
2. Engineering Challenge: Avoiding Recursive Dependency Traversal
One important engineering problem appeared in dependency cycle detection.
Initially, RepoLens used recursive Depth-First Search (DFS) to identify circular dependencies.
Recursive traversal is easy to understand, but Python limits recursion depth. Deep dependency chains can exceed the interpreter's recursion limit and raise:
RecursionError: maximum recursion depth exceeded
For a codebase intelligence tool, this is an undesirable failure. A repository with a deep dependency chain should not cause the analysis process to crash.
Replacing recursion with an explicit stack
I refactored the cycle detection algorithm to use iterative DFS.
Instead of relying on Python's call stack, the implementation maintains traversal state explicitly.
stack = [(start_node, 0)]
rec_stack = [start_node]
rec_set = {start_node}
while stack:
current, nbr_idx = stack[-1]
neighbors = adj.get(current, [])
if nbr_idx < len(neighbors):
stack[-1] = (current, nbr_idx + 1)
neighbor = neighbors[nbr_idx]
if neighbor not in visited:
visited.add(neighbor)
rec_stack.append(neighbor)
rec_set.add(neighbor)
stack.append((neighbor, 0))
elif neighbor in rec_set:
cycle_start_idx = rec_stack.index(neighbor)
cycle = rec_stack[cycle_start_idx:] + [neighbor]
cycles.append(cycle)
else:
stack.pop()
rec_stack.pop()
rec_set.remove(current)
The explicit stack stores the current node and the next neighbor index to inspect.
Three structures manage traversal:
visited tracks nodes already explored.
rec_stack maintains the active traversal path.
rec_set provides efficient membership checks for active nodes.
When the algorithm encounters a previously visited node that remains in the active path, it identifies a back-edge and extracts the corresponding cycle path.
Testing the solution
I added a regression test named test_deep_chain_cycle_detection_no_recursion_error.
The test verifies cycle detection on a synthetic graph containing 3,000 nodes.
It completed without a Python recursion error.
One important implementation detail is that cycle extraction uses rec_stack.index() and path slicing. These operations introduce additional costs when cycles are detected. Therefore, the implementation should not be described as strictly O(V + E) in every case.
3. Engineering Challenge: Protecting Filesystem Boundaries
Repository scanners often process code from third-party projects. This creates a filesystem security concern.
Consider a repository containing a symbolic link pointing outside its root directory.
If a scanner follows that link without checking its destination, it might read files that were never intended to be part of the scanned repository.
RepoLens addresses this through canonical path resolution and repository-root containment checks.
Checking symbolic links
The scanner resolves symbolic links before processing their targets.
resolved_target = file_path.resolve(strict=True)
try:
resolved_target.relative_to(self.root_path)
if not resolved_target.is_file():
warnings.append("Skipping non-file symlink")
continue
except (ValueError, OSError, RuntimeError) as error:
warnings.append(
f"Skipping symlink escaping repository root: {error}"
)
continue
The important operation is relative_to().
If the resolved target does not belong inside the repository root, Python raises ValueError. The scanner catches this and skips the file.
Directory symlinks are also pruned to prevent traversal into external directories.
Verifying the security behaviour
I tested this with a temporary file outside the repository containing the text:
SUPER_SECRET_EXTERNAL_DATA
A symbolic link inside the test repository pointed to this external file.
The scanner rejected the link and did not read its contents.
There is an important limitation: these checks protect static cloned repositories, but a theoretical time-of-check/time-of-use race remains if another process changes filesystem entries during scanning.
This is a useful reminder that path validation and complete filesystem security are different things.
4. Engineering Challenge: Handling Multiple Programming Languages
Supporting multiple languages introduces another challenge.
Python, JavaScript, and TypeScript have different syntax structures. A single parser strategy is not suitable for every language.
RepoLens uses different analysis approaches depending on the language.
| Language | Analysis approach |
| Python | Python AST |
| JavaScript | Tree-sitter with regex fallback |
| TypeScript | Tree-sitter with regex fallback |
| Go | Regex fallback |
| Rust | Regex fallback |
| Java | Regex fallback |
| C/C++ | Regex fallback |
Separating parser errors from fallback warnings
I also improved how RepoLens reports analysis diagnostics.
Previously, using regex fallback mode could produce a parse_error message even when the source file was valid.
This mixed two different situations:
- A genuine syntax error.
- A fallback analysis mode.
The updated model separates them:
parse_error records actual syntax problems.
analysis_mode identifies the analysis method.
warnings provides additional information.
This makes reports more useful for developers and downstream AI tools.
It also avoids presenting a valid source file as syntactically incorrect simply because the preferred parser was unavailable.
5. Protecting the Analyzer from Oversized Files
Large generated files can consume significant memory and processing time.
RepoLens defines a maximum file size of 2 MiB for line counting and AST analysis.
Before parsing, the analyzer checks the file size:
stat = file_path.stat()
if stat.st_size > 2 * 1024 * 1024:
return ModuleAnalysis(
rel_path=rel_path,
language="Python",
parse_error=(
"File exceeds maximum AST parse limit."
),
)
This check happens before reading the file contents for AST parsing.
RepoLens also uses an extension-based binary file classification system and checks the first 1,024 bytes for null bytes.
These are practical safeguards, but binary detection remains heuristic rather than definitive.
6. Testing and Benchmarking RepoLens
I tested RepoLens using Python 3.12.7 on Windows 11.
The automated test suite produced the following results:
| Metric | Result |
| Tests collected | 69 |
| Passed | 69 |
| Failed | 0 |
| Skipped | 0 |
| Execution time | 2.01 seconds |
| Ruff violations | 0 |
| Mypy errors | 0 |
I also tested repository scanning against synthetic projects of different sizes.
Benchmark results
| Workload | Median runtime | Peak memory |
| 106 scanned files | 219 ms | 0.41 MiB |
| 1,051 scanned files | 2.44 s | 4.89 MiB |
| 5,251 scanned files | 11.99 s | 23.30 MiB |
The 100-file workload includes a cold-cache run of approximately 2.019 seconds. The reported median represents the warm-cache runs.
The machine used an Intel Core i7-13650HX, 16 GB DDR5 RAM, and a 1 TB PCIe Gen4 NVMe SSD.
These measurements describe my test environment, not a universal performance guarantee.
The synthetic benchmark generator was maintained outside the public repository during this audit, so independent reproduction of the full benchmark requires that generator.
7. Packaging and Distribution
RepoLens is distributed on PyPI under the package name repolens-toolkit.
The verified version is 0.1.2.
I also configured GitHub Actions Trusted Publishing using OpenID Connect (OIDC), avoiding the need to store a long-lived PyPI publishing token in the workflow.
The package was installed in a clean virtual environment and tested using its CLI commands.
The following commands were verified:
pip install repolens-toolkit
repolens --version
repolens --help
repolens scan .
repolens report . --format json
The installed package successfully executed the version, help, scan, and JSON report commands.
8. Current Limitations
RepoLens is still an evolving developer tool. Its limitations are important to understand.
Dynamic imports: Static analysis cannot reliably resolve runtime-generated imports such as importlib.import_module(variable).
Language coverage: Python receives AST-based analysis. JavaScript and TypeScript use Tree-sitter with regex fallback. Several other languages currently use regex-based import extraction rather than complete semantic AST analysis.
File-size limits: Files larger than 2 MiB are skipped for line counting and AST parsing.
Secret detection: RepoLens is not a secret scanner. It excludes certain sensitive file types and paths, but it does not inspect source contents for leaked API tokens using entropy analysis.
These limitations define where future development can improve the tool.
9. Frequently Asked Questions
Does RepoLens execute repository code?
No. Its analysis is based on static inspection, parsing, and filesystem operations. It does not intentionally import or execute the scanned project's source code.
Does RepoLens support every programming language with full AST analysis?
No. Python uses AST analysis, JavaScript and TypeScript use Tree-sitter with fallback behaviour, and several other languages use regex-based import extraction.
Can RepoLens detect every dependency?
No. Dynamic imports and runtime-generated module names are difficult to resolve through static analysis alone.
Is RepoLens a security scanner?
No. It includes filesystem boundary checks and file-size safeguards, but it is not a comprehensive vulnerability or secret-detection tool.
Conclusion
Building RepoLens taught me that useful developer tools require careful engineering decisions beyond their main features.
Static analysis provides structured repository information without executing scanned project code.
Iterative DFS handles deep dependency graphs without relying on Python's recursion limit.
Filesystem boundary checks help prevent scanners from following symbolic links outside their intended repository.
Multi-language analysis requires different parsing strategies and transparent diagnostics.
Automated testing and benchmarking help validate behaviour while exposing real limitations.
RepoLens is an ongoing project, and there is more work ahead. My goal is to make codebase exploration more structured, reproducible, and useful for both developers and AI coding assistants.
References
- RepoLens GitHub Repository
- Python AST Documentation
- Tree-sitter Documentation
About the Author
Deswanth is a software developer and open-source contributor interested in AI engineering, developer tools, Python, and automation.
GitHub: https://github.com/deswanth12