Finding message size after serialization using avro and brotobuff

DevOps Kafka
#1

Hi all, I want to find out my message size after serialization using avro and brotobuff. Is there a way to find out this information. FYI I am using spring boot

#2

Is this something you want to act upon or just curious how well they are doing with the serialization of the data?

#3

You could extend and use DefaultPartitioner and log it from there as it provides key, key as bytes, value, value as bytes so you could log the length of the message from this.

#4

I want to get metrics on the message size using different serialization techniques. For example avro vs protobuf vs json

#5

If it was me, I would create a custom serializer “wrapper” that called the desired serializer (don’t forget configure would need to do the same) and add the meta-data as headers to the message. You could then inspect in a consumer the headers of the sizes for the message

(using https://www.baeldung.com/java-size-of-object to approximate) — for this Pseudo code example

    public byte[] serialize(String topic, Headers headers, T data) {

        if (data == null)
            return null;

        try {
            byte[] bytes = wrapper.serialize(topic, header, data);

            headers.add("obj_size", ("" + InstrumentationAgent.getObjectSize(data)).getBytes(StandardCharsets.UTF_8));
            headers.add("size", ("" + bytes.length).getBytes(StandardCharsets.UTF_8));
            return bytes;
        } catch (Exception e) {
            throw new SerializationException("Error serializing JSON message", e);
        }
    }```