Executive Summary
This paper talks about the shortcomings of dependency management in Go and ways to solve it.
What is a dependency?
Dependencies are packages which are required for your projects to run. For example, if your project uses protobuf package, then protobuf is a dependency on the project. Dependency management is important to get right for any programming language, and with Go, this has been tricky since its inception.
Problem Statement
Initially no dependency management system existed in Go and ‘go get’ was the only way to download dependencies which pulls the code from the master branch of a repository. This right here is a time ticking bomb.
We ran into this issue once on production, and that was the triggering point where we started researching the possible solutions for managing dependencies.
Solution Approaches
We analyzed third party tool like dep and concepts such as vendoring (it’s an approach where all the dependencies are kept inside vendor folder in the codebase) to manage the dependencies. We also analyzed Go’s new dependency management system (Go Module) introduced in Go1.11.
Analysing the pros and cons of these solutions, we preferred ‘Go Module’ due to the following reasons:
- ‘Go Module’ provides a much cleaner way to manage dependencies as compared to any other solutions.
- Since it comes with the language, no need to install any third party tool like dep.
- It is the future of dependency management in Go. Go community is actively working on it, many new features will be added in the future releases.
- Provides a feature for warming caches in docker builds which helps in reducing deployment time
What is Go Module?
Go Module is a new dependency management system inbuilt in Go that makes dependency version information explicit and easier to manage.
A module is a collection of Go packages stored in a file tree with a go.mod file at its root. The go.mod file defines the module’s module path and its dependency requirements. Each dependency requirement is written as a module path and a specific semantic version.
Go 1.11 and 1.12 include preliminary support for modules.
Starting in Go 1.13, module mode will be the default for all development.
How to enable module support on Go repositories?
It’s very easy to enable module support on go repositories by following these steps-
1. Upgrade Golang version to 1.12.x
2. Navigate to the root of the module's source tree and create the initial module definition by executing the command 'go mod init'. This will create go.mod file.
3. Execute 'go build'. This will add all required dependencies in go.mod file and create go.sum file for checksum. Sample go.mod file-
module <my-package>
go 1.12
require (
github.com/gin-gonic/gin v1.4.0
go.uber.org/zap v1.10.0
google.golang.org/grpc v1.21.0
)
Incase your project is already using dep tool for dependency management, go build takes care of the versions mentioned in dep file and creates go.mod file accordingly
How deployment time got reduced using Go Module?
The build time of Go Docker image is always a pain as there was always a need to do ‘go get’ for building the binary. This resulted in fetching the dependencies every time we wanted to build the image. The Dockerfile looked like this-
FROM centos:latest
RUN wget https://storage.googleapis.com/golang/go1.12.5.linux-amd64.tar.gz && tar -C /usr/local -xzf go1.12.5.linux-amd64.tar.gz
COPY . /usr/local/src/mypackage/
RUN go get
RUN go build -o /go/bin/hello
However, this Dockerfile comes with a major flaw because we are copying the source code every time first, all other layers are uncached, thus go get must be executed again.
Go Module provides ‘go mod download’ command, which downloads the dependencies mentioned in go.mod file instead of using the source code. As dependency file do not change frequently, they can be simply cached by the COPY command from Dockerfile as shown below -
FROM centos:latest
RUN wget https://storage.googleapis.com/golang/go1.12.5.linux-amd64.tar.gz && tar -C /usr/local -xzf go1.12.5.linux-amd64.tar.gz
# COPY go.mod and go.sum files to the workspace
COPY go.mod .
COPY go.sum .
# Get dependancies - will be cached if mod/sum is not changed
RUN go mod download
# Copy the source code as the last step
COPY . /usr/local/src/mypackage/
RUN go build -o /go/bin/hello
We used the above technique and got tremendous improvement in our deployment time for our Go repositories. For e.g. deployment time for one of our microservice goku on production reduced by 60%
What if your project has private package dependencies?
The problem with private package dependency is that Go doesn’t have the credentials to download it.
One of the way to solve is by creating a user having read access to that private package and use the user token/ssh public key to download these dependencies. We used the above technique in one of our Go project.
Having said that, this is not the ideal or cleaner way to do this, cleaner solutions will come from using GOPROXY something like Athens which can be configured to do all this for you with you only having to set GOPROXY on those hosts that need to fetch private modules.
So we can conclude…
We have implemented Go module for all our Golang repositories and it’s been a month now, we have not faced any issues in deployment or while working locally.
We have achieved the following improvements -
- Upgrading/downgrading of third party packages is much easier now, just update the version in go.mod file
- With complete control over version of third party packages, it makes us confident about code robustness and integrity
- Deployment time reduced massively for production and other staging environments
Future Scope
1. GOPROXY setup to solve the issues for downloading private packages and caching public packages at GOPROXY so that packages can be downloaded fast.
2. Setup a domain with go.mycompany.org which will serve metadata for ‘go get tool’ like-
<head>
<meta name="go-import" content="go.mycompany.org/package git https://github.com/mycompany/package">
</head>
- With this we will be able to import the private packages like
import "go.mycompany.org/mypackage" - No need to change the code if we change our code hosting site in future. We just have to change the meta information for ‘go get tool’.
So what are you waiting for? We have found our Mr. Dependable in ‘Go Module’, you can find yours too for managing your project dependencies !!
Dependency Management in Go was originally published in Backstage on Medium, where people are continuing the conversation by highlighting and responding to this story.