DevLog 20250806: "Change Version" - File Changes History-Only Version Control for Binary Assets

Question

Overview

cv is a command-line utility I originally used for personal Unreal Engine projects, where proper version control for binary assets was lacking - but I still wanted to track what had changed, at least at the file-system level. It served its purpose well, but after switching from Unreal Engine to Godot, I haven't needed it as much.

For Divooka, as our repository grows and more GUI-related work gets involved, we've increasingly run into the issue of syncing assets across different workstations. Perhaps it's time to bring this old tool back to life.

Demand

The core requirements are simple:

1. View file change history - ideally alongside corresponding Git commit history.
2. Compare differences - similar to Git commits, but focused on files, not content.
3. Synchronize files between machines.

Regarding the third point, we’ve made some adjustments to fit our workflow:

1. Since we generally focus on one PR at a time and aren’t concerned with previous builds, collaborators mainly need the latest version of assets - not their version history.
2. Everything must remain simple, maintainable, and cost zero.

Reference Workflow

Based on my previous experience at a game company working on AAA titles, our typical flow involved:

1. Git for source code
2. Perforce for asset versioning
3. Azure DevOps (ADO) for task tracking

Each PR included a GitHub pull request, a corresponding Perforce commit, and an ADO task - usually cross-referenced manually. Every Perforce commit needed to mention which GitHub PR it related to.

The Implementation

Change Version (cv) is a CLI tool that provides quick check against a repo's changes without saving any changed contents. It does so by recording only the update time. The outputs is just like git status, without diff.

This is useful for cases when we DO NOT want to do version control yet would still want the capability to see which files has changed, as in the case of multimedia projects (e.g. game projects).

Usage

The utility supports the following key commands:

• init
• status
• log
• commit -m <Message>

Output Folders & Files

.cv

cv saves history inside the .cv folder. File update time is stored as utc.

.cvignore

cv uses a .cvignore file, which shares the same .gitignore format as git, this is used to decide which files cv should consider when issuing status command; Notice we are using a new file name instead of using .gitignore directly - this is to allow the same place having a git repo.

Ignore Rules

I've decided to follow Git's philosophy: everything is considered "important" by default, unless explicitly ignored.

Here’s a basic implementation:

private static string[]? ReadIgnoreRules()
{
string[]? ignoreRules = null;
if (File.Exists(IgnoreFilename))
ignoreRules = File.ReadAllLines(IgnoreFilename)
.Where(l => !string.IsNullOrWhiteSpace(l) && !l.Trim().StartsWith('#'))
.ToArray(); // Skip empty lines and comments
return ignoreRules;
}
private static bool ShouldIgnore(string[] rules, string path)
=> rules.Any(path.StartsWith);

To support more robust wildcard patterns, here’s an upgraded version:

public class IgnoreRule
{
    #region Properties
    private readonly Regex _regex;
    public bool IsNegation { get; }
    #endregion

    #region Construction
    public IgnoreRule(string pattern)
    {
        // Support for !foo to "unignore"
        if (pattern.StartsWith('!'))
        {
            IsNegation = true;
            pattern = pattern[1..];
        }

        // Anchor pattern at repo root if it starts with '/'
        bool anchored = pattern.StartsWith('/');
        if (anchored) 
            pattern = pattern[1..];

        // Convert Git-style glob to regex
        _regex = new Regex(
            "^" +
            Regex.Escape(pattern)
                 .Replace(@"\*\*/", "(@@SLUG@@/)")
                 .Replace(@"\*\*", ".*")
                 .Replace(@"\*", @"[^/]*")
                 .Replace(@"\?", @"[^/]")
                 .Replace("@@SLUG@@", ".*") +
            (anchored ? "$" : "(?:$|/)"),
            RegexOptions.Compiled | RegexOptions.IgnoreCase
        );
    }
    #endregion

    #region Methods
    public bool IsMatch(string path, string repoRoot)
    {
        // If anchored, the path is relative to repoRoot already
        return _regex.IsMatch(path);
    }
    #endregion
}

At this stage, creating a unit test feels essential.

Based on what we have so far, it seems plausible to implement a layer of file transfer service to achieve synching of files between workstations with the help of an intermediate file server.

File Transfer

I talked about the need for a simple file-uploading service just yesterday. Today’s use case is a bit different, as it's strictly internal - we don’t need to support multi-tenant scenarios and we have a fairly accurate idea of our storage needs. Scalability isn’t a major concern.

Server Options

We’re considering ASP.NET Core vs raw TCP sockets for the server:

Concern	Raw TCP / FTP	HTTP (ASP.NET Core)
Reliability	Requires custom framing, retries	Built-in streaming, timeouts, retries
Security	Must implement SSH/SSL manually	HTTPS support out-of-the-box, auth middleware
Chunking/Resume	Reinvent the protocol	Use Range headers, tus.io, etc.
Deployment	Requires FTP daemon, ports, etc.	ASP.NET + Kestrel or NGINX
Extensibility	Hard to add endpoints	Minimal APIs, MVC, SignalR, gRPC, DI, etc.

Because we plan to expose the server on the internet, authentication is a must, even if it adds some programming overhead.

Endpoints

To start, simple endpoints suffice for basic usages:

• GET /files: Returns a JSON list of all tracked paths.
• GET /files/{**path}: Streams the latest copy of a single file.
• PUT /files/{**path}: Uploads or overwrites a file.
• DELETE /files/{**path} Removes a file.

Testing the Server

Example usage:

# List files
curl -H "X-Api-Key: my-super-secret-key" https://localhost:5001/files

# Upload a file
curl -X PUT -H "X-Api-Key: my-super-secret-key" --data-binary @myfile.txt https://localhost:5001/files/docs/myfile.txt

# Download a file
curl -H "X-Api-Key: my-super-secret-key" https://localhost:5001/files/docs/myfile.txt -o downloaded.txt

Summary

Today, I focused on:

1. Modernizing and refactoring code; upgrading the framework
2. Making the program AoT-compatible
3. Drafting the setup for file sync and server

A few TODO items still need to be addressed before safe syncing can be implemented.

Notably, the server doesn’t need to understand cv or RepoStorage - its only job is to store the latest version of each file. Maintaining accurate timestamps isn’t practical; instead, storing an MD5 checksum is a better way to avoid unnecessary uploads/downloads.

In an ideal world, we’d have a tool tailored perfectly to our needs. In reality, we often find ourselves choosing between overengineered tools that don’t quite fit or having no suitable tool at all.

When the need is real, it’s worth rolling up your sleeves and investing some time in building the right tooling.

References

• Wiki: https://wiki.methodox.io/en/Utilities/CLI/cv
• Source: https://github.com/MethodoxTech/cv