Use KSP to generate protobuf from kotlin classes. It's composed of 2 parts:

• generating serializers to allow runtime serialization/deserialization
• generating protobuf files to allow interop with other languages

K2PB is designed to have models you maintain in Kotlin or KMP. Those models define the serialization format with a couple of annotations and a minimum of boilerplate.

Kotlinx Serialization works similarly but doesn't provide a strong compatibility with protobuf official protocol, which leads to subtle issues when interoperating with other services using protoc.

Add the plugin (coming from the gradle plugin portal and maven central):

plugins {
  kotlin("multiplatform")
  id("com.glureau.k2pb") version "<!--$ GRADLE_PROPERTIES version -->0.9.29<!-- END $-->"
}

Annotate your classes with @ProtoMessage

@ProtoMessage
data class MyDataClass(val myInt: Int)

Create your serializer:

    val serializer = K2PB {
        // Those methods are generated by the plugin, by module, to collect all defined serializers
        // You just need to define the modules you want to use on this serializer
        registerSampleLibSerializers() 
        registerSampleAppSerializers()
    }

Use the serializer to serialize/deserialize your objects:

    val myDataClass = MyDataClass(42)
    val bytes = serializer.encodeToByteArray<MyDataClass>(myDataClass)
    val decoded = serializer.decodeFromByteArray<MyDataClass>(bytes)

A few settings can be customized in the k2pb block in your build.gradle.kts:

k2pb {
    protoPackageName = "com.custom.protobuf"
    javaPackage = "com.custom.javapackage"
    javaOuterClassnameSuffix = "Proto"
}

See K2PBExtension for more details. Warning, if you work with multiple compilation modules, you may want to keep the same settings on all your modules.

You can also define some settings on classes and properties:

@ProtoMessage(name = "LegacyName") // Use LegacyName in protobuf files
data class MyDataClass(
  @ProtoField(
      name = "my_int", // Use my_int in protobuf files
      number = 3, // Use 3 as the "proto number" in protobuf files
      converter = MyCustomConverter::class // Use MyCustomConverter to serialize/deserialize this field
  // and more to come
  )
  val myInt: Int
)

See ProtoMessage and ProtoField for more details.

Ktx Serialization is a great library, but it's oriented Kotlin first and need to support multiple formats. As a result, some choices on kotlinx-serialization-protobuf (still experimental) have been taken that are not fully protobuf compatible. Having a proper protobuf compatibility means that you could use K2PB on clients and protoc on servers for example. In projects where you need to interoperate with other languages, it's critical.

Ktx Serialization uses Kotlin default values as a fallback when nothing is provided.

Protobuf defines a list of default values for everyone (ex: int32 is 0 by default).

Let's take an example:

data class MyMessage(val myInt: Int = 42)

with the equivalent protobuf message:

syntax = "proto3";

message MyMessage {
  int32 myInt = 1;
}

If you deserialize a protobuf message that is empty:

• Ktx Serialization will set myInt to 42
• Java classes generated by protoc will set myInt to 0

This is already very problematic, but it gets worse. Protobuf explicitly states that default values should not be serialized.

Also note that if a scalar message field is set to its default, the value will not be serialized on the wire

Let's say now you create your instance with MyMessage() in KMP, now you want to serialize it, Ktx offers 2 choices: To offer some flexibility, Ktx Serialization allows you to choose if you want to encode default values or not.

• encodeDefaults = false: You create your instance with MyMessage() in KMP, when you want to serialize it
  • Ktx Serialization will serialize 42 as an empty message because it's Kotlin default (and default should not be serialized)
  • Java classes generated by protoc will interpret this message as myInt = 0
  • => Ktx & Protoc have 2 different interpretations of the same message
• encodeDefaults = true: You create your instance with MyMessage(0) in KMP, and you want to serialize it
  • Ktx Serialization will serialize a value of 0 (not the Kotlin default + config says to encode everything)
  • Java classes generated by protoc will interpret this message as myInt = 0
  • => Ktx & Protoc have the same interpretation BUT bandwidth is wasted

See Protobuf documentation: https://protobuf.dev/programming-guides/proto3/#default

https://github.com/Kotlin/kotlinx.serialization/issues/2831

Protobuf doesn't support nullability, but optionality:

• the field is set, and contains a value that was explicitly set or parsed from the wire. It will be serialized to the wire.
• the field is unset, and will return the default value. It will not be serialized to the wire.

Kotlin works based on the nullability concept, "" != null in Kotlin.

Let's say you want to add a new nullable field of type enum, the message encoded before this change won't contain this field. Because there's no Ktx Serialization support of nullability, the deserialization of an old message requires the default Kotlin value if present (see previous chapter why it's not good), and because developer knows it's a new field, the dev should put null as the default value. But a protoc generated class, not having the kotlin default value reading the same old message will decode the first value of the enum. Eventually, Ktx Serialization & protoc have again different values.

K2PB wants to differentiate explicitly the nullability, and will add a new field to disambiguate the nullability. Protoc generated class will have a specific accessor and will need to be aware of the nullability algorithm. And for K2PB users, it should be invisible, and can be tweaked if needed via annotations.

TODO

TODO

TODO

• Serialization works at run time, K2PB works at compile time.
• At run time, Serialization COULD know the serializerModule and @Contextual custom serializer, K2PB don't know them but allow you to pass arguments to replace a class by another class
• Ktx Serialization has some limitations right now:
  • custom serializer / @Contextual are not supported (the generator should ask for a ProtoBuf instance to know those)
  • value class are not supported (creating useless message, not matching the encoded format)
  • nested classes are not supported
• At compile time, K2PB has access to documentation and more code information to be able to copy those in generated protobuf files, generating better/richer files

(Eventually I hope to be able to handle upgrades from existing protobuf files, that's another reason to get this out of KotlinXSerialization/run-time limitations.)

This is an experimental tool, don't hesitate to create github issues for any question or problem you encounter.

Current known limitations:

• proto2 is not supported
• Support reserved fields (dedicated annotation? Real usage?)
• Support upgrading proto files (checking that the newly generated files are compatible with the current ones)

Tests are located in the 2 samples modules. It's asserting that protobuf files are stable.

It's also using protoc on the generated protobuf files, allowing us to test that a message encoded via KotlinX Serialization will be decoded successfully by the code generated by the protobuf files.

Proto2 is not supported at the moment (please open an issue/PR if you need it). KotlinX Serialization support the Proto3 format IF all List<> are annotated with @ProtoPacked.

See sample-lib and sample-app for examples. This library is using those samples as the primary tests source.