Introduction⌗

Buf is an open-source ecosystem management tool for Protobuf, not just for managing Protobuf files and dependencies, but also for protobuf-compiler-plugin configuration options.

It allows developers to spend less effort on the Protobuf Compiler while also supporting Plugin Registry, which better manages private protobuf-compiler-plugins. Buf has played a crucial role in building the Protobuf ecosystem, and it’s not an exaggeration to say it’s revolutionary!

Installation⌗

Installation on macOS is as follows:

brew install bufbuild/buf/buf

For other platforms, refer to the official documentation.

Usage⌗

To use Buf, you need to configure the Buf CLI workspace with a buf.yaml file, which defines a list of Protobuf file directories to be considered as a logical unit or module. Create this file using the following command:

buf config init

After running the command, there will be a buf.yaml in the workspace directory with the following content:

# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
version: v2

lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

Setting Up the Workspace⌗

The generated buf.yaml file behaves like a workspace containing one module with its path set to the current directory. To explicitly define modules in the workspace, provide paths to directories containing .proto files. Add proto directories to the buf.yaml file using the modules key:

# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
version: v2

lint:
  use:
    - STANDARD
modules:
  - path: internal/proto
breaking:
  use:
    - FILE

My project’s .proto files are located in the internal/proto directory, so I set modules.path to internal/proto here.

Setting Project Dependencies⌗

You can add dependencies by adding a deps list to the buf.yaml file and defining the required dependencies:

# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
version: v2

lint:
  use:
    - STANDARD
deps:
  - buf.build/googleapis/googleapis
modules:
  - path: internal/proto
breaking:
  use:
    - FILE

In this case, I set up the buf.build/googleapis/googleapis dependency because I’m using Google Protobuf-related RESTful definitions. Before Buf, we had to manually download the googleapis repository and specify the dependency path during compilation.

Generating Code⌗

Buf CLI

The Buf CLI provides a user-friendly experience for generating code locally that’s compatible with protoc usage. Before starting, we need to create a buf.gen.yaml configuration file to configure local code generation. It controls how the buf generate command executes protocol plugins on a given module. You can use it to configure the path where each protobuf-compiler-plugin writes its results and specify configuration options for each plugin.

touch buf.gen.yaml

version: v2

managed:
  enabled: true
  disable:
    - file_option: go_package
      module: buf.build/googleapis/googleapis
  override:
    - file_option: go_package_prefix
      value: github.com/betterde/focusly/internal/gen

plugins:
  - remote: buf.build/protocolbuffers/go
    out: internal/gen
    opt: paths=source_relative
  - remote: buf.build/grpc/go
    out: internal/gen
    opt: paths=source_relative
  - remote: buf.build/grpc-ecosystem/gateway
    out: internal/gen
    opt: paths=source_relative
  - remote: buf.build/connectrpc/go:v1.12.0
    out: internal/gen
    opt: paths=source_relative
  - local: protoc-gen-es
    out: spa/src/gen
    opt: target=ts
  - local: protoc-gen-connect-es
    out: spa/src/gen
    opt: target=ts
  - local: protoc-gen-connect-query
    out: spa/src/gen
    opt: target=ts
  - remote: buf.build/community/google-gnostic-openapi:v0.7.0
    out: docs/api
    opt: paths=source_relative

The above configuration is used to generate the following files:

*.pb.go
*.pb.gw.go
*_grpc.pb.go
connect/.connect.go
TypeScript files needed for the frontend project
OpenAPI specification

If your project needs to use gRPC with other languages, you just need to find the corresponding protobuf-compiler-plugin and install it, then you can use buf generate to generate the corresponding code.

Code Linting⌗

You can use the following command to lint the .proto files in your project:

buf lint

Compatibility Check⌗

For Protobuf, file change compatibility is very important. To avoid compatibility issues, you can use the following command for compatibility detection:

buf breaking --against ".git#subdir=internal/proto"

You can set the comparison reference through --against, with supported settings as follows:

.git#branch=master
.git#tag=v1.0.0
.git#subdir=internal/proto

Besides the local .git, you can also use a remote git as a comparison reference, such as https://github.com/betterde/focusly.git. Additionally, you can use the --error-format=json parameter to set the output format to JSON.

Formatting⌗

You can use the following command to format .proto files:

# View the changes after formatting
buf format -d

# Write the formatted changes to files
buf format -w

Conclusion⌗

Buf’s power doesn’t stop there. It has improved the ecosystem of Protobuf and gRPC. Later, I will write more about practical experiences with Buf in projects…

I hope this is helpful, Happy hacking…

Managing Protobuf with Buf