grpc

• grpc
• grpc-go

grpc seamlessly integrates protobuf

protobuf

• For those of you accustomed to JSON and XML data storage formats, I believe most have never heard of Protocol Buffer.
• Protocol Buffer is actually a lightweight & efficient structured data storage format developed by Google, and its performance is truly much stronger than JSON and XML!
• Protobuf has gone through protobuf2 and protobuf3, with pb3 being much simpler than pb2. The current mainstream version is pb3.

Downloading protoc

Download protoc andgo get github.com/golang/protobuf/protoc-gen-go

syntax = "proto3";

option go_package = "."; //这名不写现在编译不了

message HelloRequest {
  string name = 1; //1 是编号不是值
}

protoc --proto_path=. \
  --go_out=. \
  --go-grpc_out=. \
  ./helloworld.proto

This will generate a helloworld.pb.go file in the directory.

Comparison of compression ratios

package main

import (
 "RpcLearn/protobuf"
 "encoding/json"
 "fmt"
 "github.com/golang/protobuf/proto"
)

// TIP <p>To run your code, right-click the code and select <b>Run</b>.</p> <p>Alternatively, click
// the <icon src="AllIcons.Actions.Execute"/> icon in the gutter and select the <b>Run</b> menu item from here.</p>
type Hello struct {
 Name string `json:"name"`
}

func main() {
 //TIP <p>Press <shortcut actionId="ShowIntentionActions"/> when your caret is at the underlined text
 // to see how GoLand suggests fixing the warning.</p><p>Alternatively, if available, click the lightbulb to view possible fixes.</p>
 //s := "gopher"
 //fmt.Printf("Hello and welcome, %s!\n", s)
 //
 //for i := 1; i <= 5; i++ {
 // //TIP <p>To start your debugging session, right-click your code in the editor and select the Debug option.</p> <p>We have set one <icon src="AllIcons.Debugger.Db_set_breakpoint"/> breakpoint
 // // for you, but you can always add more by pressing <shortcut actionId="ToggleLineBreakpoint"/>.</p>
 // fmt.Println("i =", 100/i)
 //}
 jsonStruct := Hello{
  Name: "gopher",
 }
 jsonReq, _ := json.Marshal(jsonStruct)
 fmt.Println(string(jsonReq))
 fmt.Println(len(jsonReq))
 req := __.HelloRequest{
  Name: "gopher",
 }
 rsp, _ := proto.Marshal(&req)
 fmt.Println(rsp)
 fmt.Println(len(rsp))

}

Roughly a two-fold difference.

Explanation of Changes in Protobuf Usage in Go

📌 Current Behavior Explanation

Starting from Go's Protobuf plugin v1.20+, gRPC support has been split into a separate plugin, protoc-gen-go-grpc, used for generating service interface-related code. This approach is more modular and provides clearer responsibilities.

Therefore, execute the following command:

protoc --proto_path=. \
  --go_out=. \
  --go-grpc_out=. \
  ./your_file.proto

This will generate:

• your_file.pb.go: Contains message structures (e.g., Request, Response)
• your_file_grpc.pb.go: Contains service interface definitions (e.g., YourServiceServer)

It is recommended to keep and synchronize these two files.

A brief overview of the gRPC startup process

1. Create stub files and method definitions, then compile

// proto/helloworld.proto
syntax = "proto3";
option go_package = ".;proto";

service Greeter{
    rpc SayHello (HelloRequest) returns (HelloReply);
}
message HelloRequest {
    string name = 1;
}
message HelloReply {
    string message = 1;
}

1. Server-side implementation of method calls, note the server implementation (duck typing)

package main

import (
 "RpcLearn/grpc_test/proto"
 "context"
 "google.golang.org/grpc"
 "log"
 "net"
)

type Server struct {
 proto.UnimplementedGreeterServer // 👈 重点：嵌入这个类型
}

func (s *Server) SayHello(ctx context.Context, req *proto.HelloRequest) (*proto.HelloReply, error) {
 reply := &proto.HelloReply{
  Message: "Hello " + req.Name,
 }
 return reply, nil
}

func main() {
 g := grpc.NewServer()                     // 创建一个新的gRPC服务器
 proto.RegisterGreeterServer(g, &Server{}) // 注册服务
 listener, err := net.Listen("tcp", ":9090")
 if err != nil {
  log.Fatalf("failed to listen: %v", err)
 }
 err = g.Serve(listener)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

1. Client-side call implementation

package main

import (
 "RpcLearn/grpc_test/proto"
 "context"
 "fmt"
 "google.golang.org/grpc"
 "google.golang.org/grpc/credentials/insecure"
)

func main() {
 // Create a new gRPC server
 //conn, err := grpc.Dial("localhost:9090", grpc.WithInsecure())
 // 使用安全连接
 //creds := credentials.NewTLS(&tls.Config{InsecureSkipVerify: true})
 conn, err := grpc.NewClient("localhost:9090",
  //grpc.WithInsecure()
  grpc.WithTransportCredentials(insecure.NewCredentials()), // ✅ 新方式
 )
 if err != nil {
  panic(err)
 }
 defer conn.Close()
 c := proto.NewGreeterClient(conn)
 r, err := c.SayHello(context.Background(), &proto.HelloRequest{Name: "world"})
 if err != nil {
  panic(err)
 }
 fmt.Println(r.Message)

}

gRPC Stream Mode

1. Simple RPC
2. Server-side data stream
3. Client-side data stream
4. Bidirectional

Focused Learning on gRPC

Default Values

Default Value Explanation

In Protobuf, different field types have the following default values:

• String type (string): The default value is an empty string.
• Byte type (bytes): The default value is an empty byte sequence.
• Boolean type (bool): The default value is false.
• Numeric types: The default value is 0.
• Enum types: The default value is the first defined enum value, which must be 0.
• Message types (message): The default value is unset (i.e., null).

Important Notes

1. For scalar message fields: Once a message is parsed, it's impossible to distinguish whether a field was set to its default value (e.g., a boolean to false) or if the field was never set at all.
2. Recommendation for boolean values: Avoid defining boolean fields with a default value of false, as this might lead to unintended behavior.
3. Serialization of scalar messages: If a scalar message field is set to its default value, that value will not be serialized for transmission.

You can refer to the Generated Code Guide for more detailed information on how Protobuf default values are handled.

Enum Type Definition

Enum types can be used when you need to specify a set of "predefined value sequences" for a field in a message type. Each enum value corresponds to an integer value.

Example

Below is an example of an enum definition:

message SearchRequest {
  enum Corpus {
    UNIVERSAL = 0;
    WEB = 1;
    IMAGES = 2;
    LOCAL = 3;
    NEWS = 4;
    PRODUCTS = 5;
    VIDEO = 6;
  }
  Corpus corpus = 1;
}

• Server-side data stream (for real-time data retrieval)
• Client-side data stream (for data reporting)
• Bidirectional data stream

 syntax = "proto3";

option go_package = ".;proto";

service Greeter {
    rpc GetStream (StreamReqData) returns (stream StreamResData); // 服务端流模式
    rpc PutStream (stream StreamReqData) returns ( StreamResData); // 客户端流模式
    rpc AllStream (stream StreamReqData) returns (stream StreamResData); // 双向流模式
}

message StreamReqData {
    string data = 1;
}

message StreamResData {
    string message = 1;
}

Compile to generate Go Protobuf

protoc --proto_path=. \
  --go_out=. \
  --go-grpc_out=. \
  ./helloworld.proto

Server-side code

package main

import (
 "RpcLearn/stream_grpc_test/proto"
 "fmt"
 "google.golang.org/grpc"
 "log"
 "net"
 "sync"
 "time"
)

const PORT = ":50052"

type server struct {
 proto.UnimplementedGreeterServer // 👈 重点：嵌入这个类型
}

func (s server) GetStream(data *proto.StreamReqData, res proto.Greeter_GetStreamServer) error {
 //TODO implement me
 //panic("implement me")
 i := 0
 for {
  i++
  _ = res.Send(&proto.StreamResData{
   Message: fmt.Sprintf("%v", time.Now().Unix()),
  })
  time.Sleep(time.Second)
  if i >= 10 {
   break
  }
 }
 return nil
}

func (s server) PutStream(cliStr proto.Greeter_PutStreamServer) error {
 //TODO implement me
 //panic("implement me")//
 for {
  if a, err := cliStr.Recv(); err != nil {
   fmt.Println(err)
   break
  } else {
   fmt.Println(a)
  }

 }
 return nil
}

func (s server) AllStream(allStr proto.Greeter_AllStreamServer) error {
 //TODO implement me
 //panic("implement me")
 wg := sync.WaitGroup{}
 wg.Add(2)
 go func() {
  defer wg.Done()
  for {
   if a, err := allStr.Recv(); err != nil {
    fmt.Println(err)
    break
   } else {
    fmt.Println("客户端发送的消息：", a)
   }
  }

 }()
 go func() {
  defer wg.Done()
  for {
   allStr.Send(&proto.StreamResData{
    Message: fmt.Sprintf("我是服务端数据:%v\n", time.Now().Unix()),
   })
   time.Sleep(time.Second)
  }

 }()
 wg.Wait()
 return nil
}

func (s *server) SomeRPC(ctx context.Context, in *pb.SomeRequest) (*pb.SomeResponse, error) {
    md, ok := metadata.FromIncomingContext(ctx)
    if (!ok) {
        // handle missing metadata
    }
    // do something with metadata
    return &pb.SomeResponse{}, nil
}

func main() {
 // 创建一个新的gRPC服务器
 g := grpc.NewServer()
 proto.RegisterGreeterServer(g, &server{}) // 注册服务
 listener, err := net.Listen("tcp", PORT)
 if err != nil {
  log.Fatalf("failed to listen: %v", err)
 }
 err = g.Serve(listener)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

Client-side code

package main

import (
 "RpcLearn/stream_grpc_test/proto"
 "context"
 "fmt"
 "google.golang.org/grpc"
 "google.golang.org/grpc/credentials/insecure"
 "sync"
 "time"
)

func main() {
 // Create a new gRPC server
 conn, err := grpc.("localhost:50052",
  //grpc.WithInsecure()
  grpc.WithTransportCredentials(insecure.NewCredentials()), // ✅ 新方式
 )
 if err != nil {
  panic(err)
 }
 defer conn.Close()
 c := proto.NewGreeterClient(conn)
 //res, _ := c.GetStream(context.Background(), &proto.StreamReqData{Data: "慕课网"})
 //for {
 // r, err := res.Recv()
 // if err != nil {
 //  break
 // }
 // fmt.Println(r)
 //}
 //// 客户端流模式
 //putS, _ := c.PutStream(context.Background())
 //i := 0
 //for {
 // i++
 // putS.Send(&proto.StreamReqData{Data: fmt.Sprintf("慕课网%v", i)})
 // time.Sleep(time.Second)
 // if i >= 10 {
 //  break
 // }
 //}

 //双向流模式
 allStr, _ := c.AllStream(context.Background())
 wg := sync.WaitGroup{}
 wg.Add(2)
 go func() {
  defer wg.Done()
  for {
   if a, err := allStr.Recv(); err != nil {
    fmt.Println(err)
    break
   } else {
    fmt.Println("我是客户端:%v", a)
   }
  }
 }()
 go func() {
  defer wg.Done()
  i := 0
  for {
   i++
   allStr.Send(&proto.StreamReqData{Data: fmt.Sprintf("双向的慕课网:%v", i)})
   time.Sleep(time.Second)
   if i >= 10 {
    break
   }
  }
 }()
 wg.Wait()
}

Protobuf Type to Go Type Mapping (with Encoding Recommendations)

option go_package

Specifying the compilation output directory

How to import other protos in a proto file

import "base.proto"

import "google/protobuf/empty.proto"

Nested messages

Enum types

enum Gender{
  MALE = 0;
  FEMALE = 1;
}

Map type map<string,string> mp = 4

How to use timestamps google/protobuf/timestamp.proto

message HelloRequest {
  string name = 1; // Name, equivalent to documentation
  string url = 2;
  Gender g = 3;
  map<string, string> mp = 4;
  google.protobuf.Timestamp addTime = 5;
}

More gRPC Features

gRPC allows us to implement remote calls just like local calls. In each RPC call, there might be some useful data, which can be passed via metadata. Metadata stores data in key-value pairs, where the key is a string type and the value is `[]string`, i.e., a string array type. Metadata enables the client and server to provide each other with information about the current call, similar to RequestHeader and ResponseHeader in an HTTP request. Just as the lifecycle of an HTTP header is one HTTP request, the lifecycle of metadata is one RPC call.

Metadata

Not directly intrusive to business logic

// 新建
type MD mp[string][]string
// 创建
// 第一种方式
md := metadata.New(map[string]string{"key1": "val1", "key2": "val2"})

// 第二种方式 key 不区分大小写，会被统一转成小写。
md := metadata.Pairs(
    "key1", "val1",
    "key1", "val1-2", // "key1" will have map value []string{"val1", "val1-2"}
    "key2", "val2",
)

Sending metadata

md := metadata.Pairs("key", "val")

// 新建一个有 metadata 的 context
ctx := metadata.NewOutgoingContext(context.Background(), md)

// 单向 RPC
response, err := client.SomeRPC(ctx, someRequest)

Receiving metadata

func (s *server) SomeRPC(ctx context.Context, in *pb.SomeRequest) (*pb.SomeResponse, error) {
    md, ok := metadata.FromIncomingContext(ctx)
    if !ok {
        // handle missing metadata
    }
    // do something with metadata
    return &pb.SomeResponse{}, nil
}

gRPC Interceptors

Server-side and client-side interceptors

package main

import (
 "RpcLearn/grpc_test/proto"
 "context"
 "google.golang.org/grpc"
 "log"
 "net"
)

type Server struct {
 proto.UnimplementedGreeterServer // 👈 重点：嵌入这个类型
}

func (s *Server) SayHello(ctx context.Context, req *proto.HelloRequest) (*proto.HelloReply, error) {
 reply := &proto.HelloReply{
  Message: "Hello " + req.Name,
 }
 return reply, nil
}

func main() {
 interceptor := func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
  log.Println("Request: Received a new request", req)
  resp, err = handler(ctx, req)
  if err != nil {
   log.Println("Error:", err)
  }
  log.Println("Response: Finished processing this new request", resp)
  return resp, err
 }
 opt := grpc.UnaryInterceptor(interceptor)
 g := grpc.NewServer(opt)                  // 创建一个新的gRPC服务器
 proto.RegisterGreeterServer(g, &Server{}) // 注册服务
 listener, err := net.Listen("tcp", ":9090")
 if err != nil {
  log.Fatalf("failed to listen: %v", err)
 }
 err = g.Serve(listener)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

package main

import (
 "RpcLearn/grpc_test/proto"
 "context"
 "fmt"
 "google.golang.org/grpc"
 "google.golang.org/grpc/credentials/insecure"
 "time"
)

func main() {
 // Create a new gRPC server
 //conn, err := grpc.Dial("localhost:9090", grpc.WithInsecure())
 // 使用安全连接
 //creds := credentials.NewTLS(&tls.Config{InsecureSkipVerify: true})
 // 添加拦截器
 clientInterceptor := func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
  start := time.Now()
  err := invoker(ctx, method, req, reply, cc, opts...)
  fmt.Println("Time taken", time.Since(start), err)
  return err
 }
 opt := grpc.WithUnaryInterceptor(clientInterceptor)
 conn, err := grpc.NewClient("localhost:9090",
  //grpc.WithInsecure()
  grpc.WithTransportCredentials(insecure.NewCredentials()), // ✅ 新方式
  opt,
 )
 if err != nil {
  panic(err)
 }
 defer conn.Close()
 c := proto.NewGreeterClient(conn)
 r, err := c.SayHello(context.Background(), &proto.HelloRequest{Name: "world"})
 if err != nil {
  panic(err)
 }
 fmt.Println(r.Message)

}

gRPC Login Verification via Metadata

// server
package main

import (
 "RpcLearn/grpc_test/proto"
 "context"
 "google.golang.org/grpc"
 "google.golang.org/grpc/codes"
 "google.golang.org/grpc/metadata"
 "google.golang.org/grpc/status"
 "log"
 "net"
)

type Server struct {
 proto.UnimplementedGreeterServer // 👈 重点：嵌入这个类型
}

func (s *Server) SayHello(ctx context.Context, req *proto.HelloRequest) (*proto.HelloReply, error) {
 reply := &proto.HelloReply{
  Message: "Hello " + req.Name,
 }
 return reply, nil
}

func main() {
 interceptor := func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
  log.Println("Request: Received a new request", req)
  md, ok := metadata.FromIncomingContext(ctx)
  if !ok {
   log.Println("No metadata")
   return resp, status.Error(codes.Unauthenticated, "No metadata")
  }

  var (
   appid  string
   appkey string
  )
  if val1, ok := md["appid"]; ok {
   appid = val1[0]
  }
  if val2, ok := md["appkey"]; ok {
   appkey = val2[0]
  }
  if appid != "1010101" || appkey != "123456" {
   return resp, status.Error(codes.Unauthenticated, "No token authentication information")
  }

  resp, err = handler(ctx, req)
  if err != nil {
   log.Println("Error:", err)
  }
  log.Println("Response: Finished processing this new request", resp)
  return resp, err
 }
 opt := grpc.UnaryInterceptor(interceptor)
 g := grpc.NewServer(opt)                  // 创建一个新的gRPC服务器
 proto.RegisterGreeterServer(g, &Server{}) // 注册服务
 listener, err := net.Listen("tcp", ":9090")
 if err != nil {
  log.Fatalf("failed to listen: %v", err)
 }
 err = g.Serve(listener)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

package main

import (
 "RpcLearn/grpc_test/proto"
 "context"
 "fmt"
 "google.golang.org/grpc"
 "google.golang.org/grpc/credentials/insecure"
)

type customCredentials struct {
}

func (c *customCredentials) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
 return map[string]string{
  "appid":  "1010101",
  "appkey": "123456",
 }, nil
}
func (c *customCredentials) RequireTransportSecurity() bool {
 return false
}

func main() {

 // 添加拦截器
 //clientInterceptor := func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
 // start := time.Now()
 // //生成metadata
 // md := metadata.New(map[string]string{
 //  "appid":  "1010101",
 //  "appkey": "123456",
 // })
 // ctx = metadata.NewOutgoingContext(ctx, md)
 // err := invoker(ctx, method, req, reply, cc, opts...)
 // fmt.Println("Time taken", time.Since(start), err)
 // return err
 //}
 //opt := grpc.WithUnaryInterceptor(clientInterceptor
 // 另一种方式
 opt := grpc.WithPerRPCCredentials(&customCredentials{}) // 有一个专门的metadata的拦截器
 conn, err := grpc.NewClient("localhost:9090",
  //grpc.WithInsecure()
  grpc.WithTransportCredentials(insecure.NewCredentials()), // ✅ 新方式
  opt,
 )
 if err != nil {
  panic(err)
 }
 defer conn.Close()
 c := proto.NewGreeterClient(conn)
 r, err := c.SayHello(context.Background(), &proto.HelloRequest{Name: "world"})
 if err != nil {
  panic(err)
 }
 fmt.Println(r.Message)

}

gRPC (protoc-gen-validate)

protoc-gen-validate (PGV for short) is a Protocol Buffers plugin used to add validation logic for struct fields in generated Go code.

By adding validation rules in .proto files, it automatically generates validation code for each field, saving you from manually writing validation logic.

syntax = "proto3";

import "validate/validate.proto";

message User {
  string name = 1 [(validate.rules).string.min_len = 3];
  int32 age = 2 [(validate.rules).int32.gte = 18];
}

View the introduction to protoc-gen-validate

grpc_proto_validate

syntax = "proto3";

import "validate.proto";

option go_package = ".;proto";


service Greeter {
  rpc SayHello (Person) returns (Person);
}

message Person {
  uint64 id = 1 [(validate.rules).uint64.gt = 999];

  string email = 2 [(validate.rules).string.email = true];

  string mobile = 3 [
    (validate.rules).string = {
      pattern: "^1[3-9]\\d{9}$"
    }
  ];
}

Compile this proto by adding --validate_out="lang=go". First, install go install github.com/bufbuild/protoc-gen-validate@latest

protoc \
  --proto_path=. \
  --go_out=. \
  --go-grpc_out=. \
  --validate_out=lang=go:. \
  ./helloworld.proto

package main

import (
 "RpcLearn/grpc_validate_test/proto"
 "context"
 "google.golang.org/grpc"
 "google.golang.org/grpc/codes"
 "google.golang.org/grpc/status"
 "log"
 "net"
)

type Server struct {
 proto.UnimplementedGreeterServer // 👈 重点：嵌入这个类型
}

func (s *Server) SayHello(ctx context.Context, request *proto.Person) (*proto.Person, error) {
 // Simulate some processing
 return &proto.Person{
  Id:     request.Id,
  Mobile: request.Mobile,
  Email:  request.Email,
 }, nil
}

type Validator interface {
 Validate() error
}

func main() {
 //p := new(proto.Person)
 //err := p.Validate()
 //if err != nil {
 // panic(err)
 //}
 var interceptor grpc.UnaryServerInterceptor
 interceptor = func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
  if v, ok := req.(Validator); ok {
   if err := v.Validate(); err != nil {
    return nil, status.Error(codes.InvalidArgument, err.Error())
   }
  }
  return handler(ctx, req)
 }
 opt := grpc.UnaryInterceptor(interceptor)
 g := grpc.NewServer(opt)                  // 创建一个新的gRPC服务器
 proto.RegisterGreeterServer(g, &Server{}) // 注册服务
 listener, err := net.Listen("tcp", ":50051")
 if err != nil {
  log.Fatalf("failed to listen: %v", err)
 }
 err = g.Serve(listener)
 if err != nil {
  log.Fatalf("failed to serve: %v", err)
 }
}

package main

import (
 "RpcLearn/grpc_validate_test/proto"
 "context"
 "fmt"
 "google.golang.org/grpc"
 "google.golang.org/grpc/credentials/insecure"
 "time"
)

type customCredential struct{}

func main() {
 // 拦截器（可选）
 clientInterceptor := func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
  start := time.Now()
  err := invoker(ctx, method, req, reply, cc, opts...)
  fmt.Println("Call duration:", time.Since(start), "Error:", err)
  return err
 }

 // 使用新方式连接（推荐）
 conn, err := grpc.NewClient("localhost:50051",
  grpc.WithTransportCredentials(insecure.NewCredentials()), // 替代 grpc.WithInsecure()
  grpc.WithUnaryInterceptor(clientInterceptor),
 )
 if err != nil {
  panic(err)
 }
 defer conn.Close()

 // 构建 gRPC 客户端
 c := proto.NewGreeterClient(conn)

 // 调用 RPC
 rsp, err := c.SayHello(context.Background(), &proto.Person{
  Id:     1000,
  Email:  "bobby@test.com",
  Mobile: "13222223333",
 })
 if err != nil {
  panic(err)
 }

 fmt.Println("Returned ID:", rsp.Id)
}

Exception Handling in gRPC

1. gRPC Status Codes

gRPC Status Codes Documentation

Go Exception Handling

1. Server-side

st := status.New(codes.InvalidArgument, "invalid username")

2. Client-side

st, ok := status.FromError(err)
if !ok {
    // Error was not a status error
}
st.Message()
st.Code()

Timeout Mechanism

ctx,_ = content.WithTimeout(context.Background(),time.Second*3)