Protobuf with GO

In this second post of Go series, we will see what Protocol Buffers are, data format and how to implement it in a Go-based application. We will also see how protocol buffers are effective over XML and JSON.

Karthik Kamalakannan

Founder and CEO

Thu, Oct 24, 2019

Hey fellow coders, this is the second post of my Go series. You can check my last blog if you want to know how to set up your Go workspace. In this article, we are going to see what Protocol Buffers are, data format and how to implement it in a Go-based application.

What is Protocol Buffer?

Protocol Buffer is a data format, like JSON and XML which stores the structured data that can be serialized and deserialized. It is a data format developed by Google. The main advantage of Protobuf is that it is much smaller than the other formats.

Protobuf is 3 to 10 times smaller and 20 to 100 times faster than XML.

Now, let’s take a football game and represent the data in XML, JSON, and Protobuf.

XML:

<game>
  <home>Real Madrid</home>
  <away>Barcelona</away>
  <venue>Santiago Bernabéu Stadium</venue>
  <date>26-10-2019</date>
</game>

JSON:

{
  "home": "Real Madrid",
  "away": "Barcelona",
  "venue": "Santiago Bernabéu Stadium",
  "date": "26-10-2019"
}

Protobuf:

[10 11 82 101 97 108 32 77 97 100 114 105 100 18 9 66 97 114 99 101 108 111 110 97 26 26 83 97 110 116 105 97 103 111 32 66 101 114 110 97 98 195 169 117 32 83 116 97 100 105 117 109 34 10 50 54 45 49 48 45 50 48 49 57]

The above shown protobuf format is the encoded byte of the string, starting from position 2 in the array. The string “Real Madrid” is spelled out: R:82, e:101 and so on.

To know about this encoding, please check out Google’s own documentation here: Proton Buffer Encoding.

At this scale, the size is mostly similar. But when we scale it to larger data, the size starts to show a huge difference.

So, now let’s bring Protobuf to our Go code.

Installing Packages

➜  go_workspace go get github.com/golang/protobuf
➜  go_workspace go get github.com/golang/protobuf/proto
➜  go_workspace go get -u github.com/golang/protobuf/protoc-gen-go
➜  go_workspace export PATH=$PATH:$GOPATH/bin

This will install the necessary packages and we are now ready.

Now, let’s go and define the protobuf for the game object.

Game.proto

syntax: "proto3";

package main;

message Game {
  string home:1;
  string away:2;
  string venue:3;
  string date:4;
}

We start by specifying the version syntax of proto. Here we set syntax to ‘proto3’. Next, we define the package in which this object is to be used. After that, we will define the format of our Game object. This consists of our message format of type Game which features the following fields, home, away, venue and date.

As we defined the proto file, we are going to compile it with the protoc.

➜  go_workspace/src protoc --go_out=. *.proto

This will generate a game.pb.go file, which has the auto-generated code. For a detailed understanding of this code, check out Google’s documentation: Go generated code.

main.go

package main
import (
  fmt "fmt"
  "log"
  proto "github.com/golang/protobuf/proto"
)
func main() {
  elClasico := &Game{
    Home:  "Real Madrid",
    Away:  "Barcelona",
    Venue: "Santiago Bernabéu Stadium",
    Date:  "26-10-2019",
  }
  data, err := proto.Marshal(elClasico)
  if err != nil {
    log.Fatal("Marshaling error: ", err)
  }
  fmt.Println(data)
  newElClasico := &Game{}
  err:proto.Unmarshal(data, newElClasico)
  if err != nil {
    log.Fatal("UnMarshaling error: ", err)
  }
  fmt.Println(newElClasico.GetHome())
  fmt.Println(newElClasico.GetAway())
  fmt.Println(newElClasico.GetVenue())
  fmt.Println(newElClasico.GetDate())
}

In the above code, we use the Game struct defined in the game.pb.go to add details to the elClasico object. We use the Marshal function of the proto library to convert the object to the Protobuf format. Then the encoded bytes can be decoded with the Unmarshal function. Now we can use the function GetHome, GetAway generated in the game.pb.go file to get the values from the decoded object.

➜  go_workspace/src go run main.go game.pb.go
[10 11 82 101 97 108 32 77 97 100 114 105 100 18 9 66 97 114 99 101 108 111 110 97 26 26 83 97 110 116 105 97 103 111 32 66 101 114 110 97 98 195 169 117 32 83 116 97 100 105 117 109 34 10 50 54 45 49 48 45 50 48 49 57]
Home:  Real Madrid
Away:  Barcelona
Venue:  Santiago Bernabéu Stadium
Date:  26-10-2019

Don’t forget to include the game.pb.go. Now we have a small example up and running. But actually in the real world, the data is not going to be this simple. Now let’s see some nested fields.

We will use the same Game object. In this, we are taking the fields home and away and making it an object Team.

game.proto:

syntax: "proto3";
package main;
message Team {
  string home:1;
  string away:2;
}
message Game {
  Team team:1;
  string venue:2;
  string date:3;
}

So again, we create the auto-generated code.

➜  go_workspace/src protoc --go_out=. *.proto

And we update the main.go file correspondingly.

main.go

package main
import (
  fmt "fmt"
  "log"
  proto "github.com/golang/protobuf/proto"
)
func main() {
  elClasico := &Game{
    Venue: "Santiago Bernabéu Stadium",
    Date:  "26-10-2019",
    Team: &Team{
      Home: "Real Madrid",
      Away: "Barcelona",
    },
  }
  data, err := proto.Marshal(elClasico)
  if err != nil {
    log.Fatal("marshaling error: ", err)
  }
  fmt.Println(data)
  newElClasico := &Game{}
  err:proto.Unmarshal(data, newElClasico)
  if err != nil {
    log.Fatal("unmarshaling error: ", err)
  }
  fmt.Println("Home: ", newElClasico.Team.GetHome())
  fmt.Println("Away: ", newElClasico.Team.GetAway())
  fmt.Println("Venue: ", newElClasico.GetVenue())
  fmt.Println("Date: ", newElClasico.GetDate())
}

Now if we run the program.

➜  go_workspace/src go run main.go game.pb.go
[10 24 10 11 82 101 97 108 32 77 97 100 114 105 100 18 9 66 97 114 99 101 108 111 110 97 18 26 83 97 110 116 105 97 103 111 32 66 101 114 110 97 98 195 169 117 32 83 116 97 100 105 117 109 26 10 50 54 45 49 48 45 50 48 49 57]
Home:  Real Madrid
Away:  Barcelona
Venue:  Santiago Bernabéu Stadium
Date:  26-10-2019

Interesting, isn’t it?

Alright guys, we now know how to use protocol buffer data format with Go. That’s it for now. Will meet you guys again with another interesting topic.