Protocol Buffers are a language-neutral, platform-neutral, extensible way of serializing structured data for use in commnucations protocols, data storage, and more, originally designed at Google.

Protobuf.js is a pure JS implementation with TypeScript support for node.js and browser. It’s easy to use, blazingly fast and works out of the box with .proto files.

Installation

Node.js

1
npm install protobufjs
1
var protobuf = require('protobufjs')

Browsers

1
<script src="../protobuf.js"></script>

Distributions

Where bundle size is a factor, there are additional stripped-down versions of the full-library(~19kb gzipped) available that excldue certain functinality

  • when working with JSON descriptor, and/or reflection only, see the light-library(~16kb gzipped) that excludes the parser. CommonJS entry point is:
1
var protobuf = require('protobufjs/light')

When working with statically generated code only, see the minimal library(~6.5kb gzipped) that also excludes reflection. CommonJS entry point is:

1
var protobuf = require('protobufjs/minimal`)

Usage

Because JS is a dynamically typed language, protobuf.js introduces the concept of a valid message in order to provide the best possible performance.

Valid Message

A valid message is an object not missing any required fields and exclusively composed of JS types understood by the wire format writer.

There are two possible types of valid messages and the encoder is able to work with both these for convenience:

  • Message Instance (explicit instance of message classes with default values on their prototype) always (have to) satisfy the requirements of a valid message by design.

  • Plain JavaScript Objects that just so happen to be composed in a way satisfying the requirements of a valid message as well.

In a nutshell, the wire format writer understoods the following types:

Field Type Expected JS type(create, encode) Conversion (fromObject)
s-/u-/int32
s-/fixed32		 number (32 bit integer) `value
s-/u-/int64
s-/fixed64		 Long-like(optimal)
number(53 bit integer)		 Long.fromValue(value) with long.js
parseInt(value, 10) otherwise
float
double		 number Number(value)
bool boolean Boolean(value)
string string String(value)
bytes Uint8Array(optimal)
Buffer(optimal under node)
Array.<number>(8 bit integers)		 base64.decode(value) if a string
Object with non-zero.length is assumed to be buffer-like
enum number (32 bit integer) Looks up the numeric id if a string
message Valid Message Message.fromObject(value)

  • Explicit undefined and null are considered as not set if the field is optional

  • Repeated fields are Array.<T>

  • Map fields are Object.<string, T> with the key being the string representation of the respective value or an 8 character long binary hash string for Long-likes.

  • Types marked as optimal provide the best performance because no conversion step(i.e. number to low and high bits or base64 string to buffer) is required.

For performance reasons, each message class provides a distinct set of methods with each method doing just one thing. This avoid unnecessary assertions / redundant operations where performance is a concern but also forces a user to perform verification explicityly where necessary.

Methods

  • Message.verify(message: Object): null | string

  • Message.encode(message: Message | Object[, writer: Writer]): Writer

  • Message.encodeDelimited(message: Message | Object[, writer: Writer]): Writer

  • Message.decode(reader: Reader | Uint8Array): Message

  • Message.toObject(message: Message[, options: ConversionOptions]): Object

  • Message.verify(message: Object): null | string

    Verifies that a plain js object satisfies the requirements of a valid message and thus can be encoded without issues. Instead of throwing, it returns the error message as a string, if any:

    1
    2
    3
    var payload = 'invalid (not an object)'
    var err = AwesomeMessage.verify(payload)
    if (err) throw Error(err)

  • Message.encode(message: Message | Object[, writer: Writer]): Writer

Encodes a message instance or valid plain javascript object. This method does not implicit verify the message and it’s up to user to make sure that the payload is a valid message.

1
var buffer = AwesomeBuffer.encode(message).finish()
  • Message.encodeDelimited(message: Message | Object[, writer: Writer]): Writer

Works like Message.encode but additionally prepends the length of the message as a varint.

  • Message.decode(reader: Reader): Message

Decode a buffer to a message instance. If required fields are missing, it throws a util.ProtocolError with an instance property set to the so far decoded message. If the wire format is invalid, it throws an Error.

1
2
3
4
5
6
7
8
9
try {
  var decodedMessage = AwesomeMessage.decode(buffer)
} cache (e) {
  if (e instanceof protobuf.util.ProtocolError) {
    // e.instance holds the so far decoded message with missing required fields
  } else {
    // wire format is invalid
  }
}
  • Message.decodeDelimited(reader: Reader | Uint8Array): Message

works like Message.decode but additionally reads the length of the message prepended as a varint

  • Message.create(properties: Object): Message

Creates a new message instance from a set of properties that satisfy the requirements of a valid message. Where applicable, it is recommended to prefer Message.create over Message.fromObject because it doesn’t perform possibly redundant conversion.

1
var message = AwesomeMessage.create({ awesomeField: 'AwesomeString' })
  • Message.fromObject(object: Object): Message

Converts any non-valid plain javascript object to a message instance using the conversion steps outlined within the table above.

  • Message.toObject(message: Message[, options: ConversionOptions]): Object

Converts a message instance to an arbitrary plain javascript object for interoperability with other libraries or storage. The resulting plain javascript object might still satisify the requirements of a vlaid message depending on the actual conversion options specified, but most of the time it does not.

1
2
3
4
5
6
7
8
9
var object = AwesomeMessage.toObject(message, {
  enums: String,
  longs: String,
  bytes: String,
  defaults: true,
  arrays: true,
  objects: true,
  oneofs: true,
})

For reference, the following diagram aims to display relationships between the different methods and the concepts of a valid message:

In other words: verify indicates that calling create or encode directly on the plain object will [result in a valid message respectively] succeed. fromObject, on the other hand, does conversion from a broader range of plain objects to create valid message.

Examples

Using .proto files

It is possible to load existing .proto files using the full library, which parses and compiles the definitions to ready to use (reflection-based) message classes:

1
2
3
4
5
6
package awesomepackage
syntax = "proto3"

message AwesomeMessage {
  string awesome_field = 1; becomes awesomeField
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
protobuf.load('awesome.proto', function(err, root) {
  if (err) {
    throw err
  }
  // Obtain a message type
  var AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage')

  // Exemplary payload
  var payload = { awesomeField: 'AwesomeField' }

  // Verify the payload if necessary(i.e. when possible incomplete or invalid)
  var errMsg = AwesomeMessage.verify(payload)
  if (errMsg) throw Error(errMsg)

  // Create a new message
  var message = AwesomeMessage.create(payload) // or use .fromObject if conversion is necessary

  // Encode a message to an Uint8Array(browser) or Buffer (node)
  var buffer = AwesomeMessage.encode(message).finish()

  // ... do something with message

  // Decode an Uint8Array(browser) or Buffer(node) to a message
  var message = AwesomeMessage.decode(buffer)

  // ...do something with message

  // If the application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.

  // Maybe conver the message back to a plain object

  var object = AwesomeMessage.toObject(message, {
    loings: String,
    enums: String,
    bytes: String,
    // see Conversions
  })
})

Additionally, promise syntax can be used by omitting the callback, if preferred:

1
protobuf.load('awesome.proto').then(root => {})

Using JSON Descriptors

The library utilize JSON descriptor that are equivalent to a .proto definition. For example, the following is identical to the .proto definition seen above:

1
2
3
4
5
6
7
8
9
10
11
12
13
// awesome.json
{
  "nested": {
    "AwesomeMessage": {
      "fields": {
        "awesomeField: {
          "type": "string",
          "id", 1
        }
      }
    }
  }
}

JSON descriptor closely resemble the internal reflection structure:

Exclusively using JSON descriptor instead of .proto files enables the use of just the light library(the parser isn’t requried in this case)

A JSON descriptor can either be loaded the usual way:

1
2
3
protobuf.load('awesome.json', function(err, root) {
  if (err) throw Error(err)
})

Or it can be loaded inline:

1
2
3
4
var jsonDescriptor = require('./awesome.json') // exemplary for node
var root = protobuf.Root.fromJSON(jsonDescriptor)

// Continue at 'Obtain a message type' above

More Info here

Usage with TypeScript

The library ships with its own type definitions and modern editors will automatically detect and use them for code completion.

The npm package depends on @types/node because of Buffer and @types/long because of Long. If you are not building for node and/or using long.js, it should be safe to exclude them manually.

Using the JS API

The API shown above works pretty much the same with TypeScript. However, because everything is typed, accessing fields on instances of dynamically generated message classes requries either using bracket-notation(i.e. message[‘awesomeField’]), or explicit casts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import { load } from 'protobufjs'

load('awesome.proto', function(err, root) {
  if (err) throw Error(err)

  const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage')
  let message = AwesomeMessage.create({ awesomeField: 'hello' })
  cnosole.log(`message = ${JSON.stringify(message)}`)

  let buffer = AwesomeMessage.encode(message).finish()
  console.log(`Buffer = ${Array.prototype.toString.call(buffer)}`)

  let decoded = AwesomeMessage.decode(buffer)
  console.log(`decoded = ${JSON.stringify(decoded)}`)
})