Innovative Software Technology-Streamlining Your Git Repositories: The Case Against Storing Binaries Directly

The Hidden Costs of Binary Bloat in Git Repositories

As a software architect, I recently initiated a crucial audit: identifying and addressing the proliferation of inadvertently versioned binary files within our company’s Bitbucket repositories. I’m not referring to obvious security concerns like passwords or API keys. My focus is on common culprits such as DLLs, .so files, executables, and even large third-party libraries like jquery.js being committed directly into our version control system.

Before diving into the practical cleanup, I needed to build a robust technical justification for this significant effort. This article documents the research I conducted to underpin that decision—insights that I hope will be valuable to others facing similar challenges.

Is This Audit Worth the Effort?

In most scenarios, absolutely – but with important nuances.

The endeavor to pinpoint and remove unnecessary binaries from Git can yield substantial improvements in performance, cost efficiency, and overall maintainability. However, there are specific, limited situations where versioning binaries might be acceptable or even necessary.

Efficiency and Performance Impacts

Technical implications:

Each distinct version of a binary creates a new, complete blob in the repository’s history.
Git is designed for efficient delta compression on text files; it performs poorly with binaries.
Core Git operations such as clone, fetch, and push become proportionally slower as repository size inflates.

Quantifiable example:

A 50MB binary versioned 20 times can add approximately 1GB of historical data.
A git clone operation that once took 30 seconds could easily extend to several minutes.
For a team of 20+ developers, this accumulation of wasted time can amount to many hours lost each month.

Documented cases: Repositories containing binaries in their history frequently exhibit significant performance degradation. A microservice repository, for instance, might easily reach 800MB due to committed JAR dependencies, with 600MB consisting solely of binaries—disproportionately impacting clone times.

Storage Costs

Direct financial and operational impacts:

Bitbucket Cloud tiers often have repository size limits (e.g., 4GB for free plans).
Self-hosted Git servers face indefinitely escalating storage costs.
CI/CD pipelines experience slower checkouts, increasing both build duration and associated infrastructure costs.

Practical example:

A repository with 500MB of binaries in its history could add 10-15 minutes of extra time per build in CI/CD.
Running 100 builds per day equates to roughly 25 hours per month of wasted machine time.

Maintainability Challenges

Concrete problems encountered:

Binaries do not produce readable diffs, making effective code reviews impossible for these files.
Merge conflicts involving binaries are often unresolvable; developers are typically forced to simply choose one version, obliterating merge history.
Third-party libraries (like jquery.js) should be managed via dedicated package managers, not direct commits.

When Versioning Binaries in Git Can Be Acceptable

Valid Scenarios:

Small Binaries: Files typically under 100KB that are rarely modified.
Essential Bootstrap: Minimal tools or scripts required for the initial setup and configuration of a project.
Regulatory Compliance: Immutable artifacts specifically required for audit trails in regulated industries (e.g., finance, healthcare).
Isolated Projects: Projects with a very small number of contributors and low frequency of changes, where the impact is marginal.

Trade-offs to Consider:

For very small repositories (total size <50MB) with a reduced team (<5 people), the performance impact might be negligible.
Evaluate migration costs versus potential benefits to determine if the effort justifies the gains.

Effective Alternatives and Their Limitations

Git LFS (Large File Storage)

Ideal for: Large media assets, datasets, or other large files that are unavoidable in a repository.

Limitations:

Requires additional server-side configuration.
Introduces complexity to the developer workflow (commands like git lfs install, git lfs track, git lfs pull).
Not all Bitbucket Cloud plans offer unlimited LFS support.
The development team needs to be trained on the new workflow.

Package Managers

Recommended for: All third-party software dependencies.

JavaScript: npm, yarn, pnpm
Java: Maven, Gradle
Python: pip, Poetry
C++: Conan, vcpkg

Artifact Repositories

Suitable for: Storing build artifacts, released versions, and internal libraries.

Examples: Artifactory, Nexus, Google Artifact Registry.
Offer native integration with CI/CD pipelines.
Provide robust version control, security, and auditing capabilities.
Advantage over Git LFS: Generally better suited for automated build and deployment pipelines.

Common approach: Migrating build-generated binaries to specialized repositories (like Nexus) and configuring build tools (Maven, Gradle) to download them automatically can eliminate hundreds of megabytes from critical Git repositories.

Container Registries

Appropriate for: Binaries packaged within Docker images.

Examples: Docker Hub, Google Container Registry, Amazon ECR.
Utilize image tags for versioning.
Ideal for containerized deployment workflows.

How to Identify Problematic Files

Command to List the Largest Blobs:

git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  awk '/^blob/ {print $3, $2, $4}' |
  sort -rn |
  head -20

Specialized Tools:

# For a comprehensive repository analysis
git-sizer --verbose

# For safe and irreversible history cleaning
git filter-repo --strip-blobs-bigger-than 10M
# Alternatively
bfg --strip-blobs-bigger-than 10M

Metrics for Decision Making

Baseline Measurements Before Starting the Audit:

# To measure the total size of the repository
du -sh .git

# To measure the time taken for a full clone
time git clone <repo-url>

# To identify the largest files in the repository's history
git-sizer --verbose | grep "Maximum blob size"

After Cleanup, Compare:

Percentage reduction in repository size.
Reduction in clone time (in seconds).
Impact on CI/CD pipeline execution times.

Expected Return on Investment (ROI):

Repositories larger than 1GB will likely see significant gains, especially for medium to large teams.
For repositories under 100MB, evaluate the cost versus benefit on a case-by-case basis.

Practical Recommendations

Immediate Actions:

Configure an Appropriate .gitignore:

# Build artifacts
*.dll
*.so
*.exe
*.o
target/
dist/
build/

# Dependencies
node_modules/
vendor/

Regularly Audit with git-sizer:
```
git-sizer --verbose
```

Implement Pre-commit Hooks:

# Example: Block files larger than 10MB from being committed
#!/bin/bash
MAX_SIZE=10485760 # 10 MB in bytes
for file in $(git diff --cached --name-only); do
  size=$(wc -c < "$file")
  if [ $size -gt $MAX_SIZE ]; then
    echo "Error: $file exceeds the 10MB limit. Please remove it or use a proper artifact management solution."
    exit 1
  fi
done

Long-Term Strategy:

Define a Clear Versioning Policy: Establish and communicate guidelines for what can and cannot be committed to Git.
Migrate Existing Binaries: Systematically move historical binaries to appropriate solutions (Git LFS, Artifact Repositories).
Document Dependency Acquisition: Clearly specify how to obtain and manage project dependencies in your README.md files.
Continuously Measure Impact: Monitor repository metrics to ensure ongoing health and identify new issues.
Align with Compliance: Ensure all versioning strategies meet any relevant regulatory requirements.

Verifiable References

Official Documentation

Git SCM – Git Attributes
https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes
Explains how Git handles binary files and the limitations concerning diffs and merges.

Atlassian – Git LFS Tutorial
https://www.atlassian.com/git/tutorials/git-lfs
An official guide on when and how to effectively use Git LFS, especially with Bitbucket.

GitHub – Working with Large Files
https://docs.github.com/en/repositories/working-with-files/managing-large-files
Provides best practices and recommendations for managing large files within GitHub repositories.

Reference Books

Pro Git (Scott Chacon & Ben Straub)
https://git-scm.com/book
Relevant chapters include 2.2 (Git Basics) and 10.2 (Git Internals – Objects) for understanding Git’s underlying mechanics.

Version Control with Git (Jon Loeliger & Matthew McCullough)
O’Reilly Media
Offers in-depth technical coverage of Git’s object storage model and performance considerations.

Academic Papers

“Why Google Stores Billions of Lines of Code in a Single Repository”
Communications of ACM, Vol. 59 No. 7, Pages 78-87 (2016)
https://dl.acm.org/doi/10.1145/2854146
Discusses large-scale monorepo strategies and asset management challenges.

Conclusion

Undertaking an audit of binary files in your Git repositories almost always pays off, but the true value lies in measuring the impact within your specific operational context.

Scenarios Where Cleanup is Essential:

Repositories utilized by large development teams (>10 people).
Repository storage exceeding 500MB with continuous growth.
Git clone times that consistently exceed 2 minutes.
Sluggish or bottlenecked CI/CD pipelines.

Scenarios Where Cost-Benefit Analysis is Crucial:

Small and isolated repositories (<100MB).
Teams with very few contributors (<5 people).
Binaries that are genuinely required for regulatory compliance.
Projects with low frequency of Git operations.

Recommended Approach:

Pilot Project: Select 1-2 of your most critical repositories for an initial cleanup.
Measure: Rigorously collect ‘before’ and ‘after’ metrics for key indicators.
Calculate ROI: Quantify the time saved multiplied by the number of developers affected.
Decide: Use concrete data to determine whether to expand the policy across more repositories.

This theoretical groundwork will be complemented by real-world data from our ongoing audit. A proof-of-concept on a pilot repository will help validate these premises and quantify the specific gains for our environment.

Share Your Experience

Have you tackled similar challenges with binary files in your Git repositories? Are you considering an audit within your organization? Please share your experiences and insights in the comments below!

Tags: git devops productivity bestpractices repositorymanagement softwarearchitecture