Blog

Avro schema registry hasn’t supprt storing of Avro IDL files yet, so one of the solutions to maintain consistency of Avro schemas is to use Avro IDL file as source of truth and generate avsc files based on it.

1. Requirements:

• downloaded latest version of avro-tools.jar;
• JDK 8 or Open JDK 8 installed;
• prepared example.avdl file like below:

//Example based on: https://avro.apache.org/docs/1.8.2/idl.html

@namespace("com.bettercoding.avro")
protocol ExampleProtocol {

  enum CompanySize {
    SMALL,
    MEDIUM,
    BIG
  }

  record CompanyRecord {
    string name;

    CompanySize size;

    string countryCode;
  }

  error CustomException {
    string message;
  }

  string hello(string greeting);
  CompanyRecord echo(CompanyRecord `record`);
  int add(int arg1, int arg2);
  bytes echoBytes(bytes data);
  void `error`() throws CustomException;
  void ping() oneway;
}

2. Conversion using avro-tool

#java -jar avro-tools-1.8.2.jar

Version 1.8.2
 of Apache Avro
Copyright 2010-2015 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
----------------
Available tools:
          cat  extracts samples from files
      compile  Generates Java code for the given schema.
       concat  Concatenates avro files without re-compressing.
   fragtojson  Renders a binary-encoded Avro datum as JSON.
     fromjson  Reads JSON records and writes an Avro data file.
     fromtext  Imports a text file into an avro data file.
      getmeta  Prints out the metadata of an Avro data file.
    getschema  Prints out schema of an Avro data file.
          idl  Generates a JSON schema from an Avro IDL file
 idl2schemata  Extract JSON schemata of the types from an Avro IDL file
       induce  Induce schema/protocol from Java class/interface via reflection.
   jsontofrag  Renders a JSON-encoded Avro datum as binary.
       random  Creates a file with randomly generated instances of a schema.
      recodec  Alters the codec of a data file.
       repair  Recovers data from a corrupt Avro Data file
  rpcprotocol  Output the protocol of a RPC service
   rpcreceive  Opens an RPC Server and listens for one message.
      rpcsend  Sends a single RPC message.
       tether  Run a tethered mapreduce job.
       tojson  Dumps an Avro data file as JSON, record per line or pretty.
       totext  Converts an Avro data file to a text file.
     totrevni  Converts an Avro data file to a Trevni file.
  trevni_meta  Dumps a Trevni file's metadata as JSON.
trevni_random  Create a Trevni file filled with random instances of a schema.
trevni_tojson  Dumps a Trevni file as JSON.

2.1. How to generate avpr from avdl file

java -jar avro-tools-1.8.2.jar idl example.avdl > example.avpr

The result:

{
  "type" : "enum",
  "name" : "CompanySize",
  "namespace" : "com.bettercoding.avro",
  "symbols" : [ "SMALL", "MEDIUM", "BIG" ]
}

{
  "type" : "error",
  "name" : "CustomException",
  "namespace" : "com.bettercoding.avro",
  "fields" : [ {
    "name" : "message",
    "type" : "string"
  } ]
}

{
  "protocol" : "ExampleProtocol",
  "namespace" : "com.bettercoding.avro",
  "types" : [ {
    "type" : "enum",
    "name" : "CompanySize",
    "symbols" : [ "SMALL", "MEDIUM", "BIG" ]
  }, {
    "type" : "record",
    "name" : "CompanyRecord",
    "fields" : [ {
      "name" : "name",
      "type" : "string"
    }, {
      "name" : "size",
      "type" : "CompanySize"
    }, {
      "name" : "countryCode",
      "type" : "string"
    } ]
  }, {
    "type" : "error",
    "name" : "CustomException",
    "fields" : [ {
      "name" : "message",
      "type" : "string"
    } ]
  } ],
  "messages" : {
    "hello" : {
      "request" : [ {
        "name" : "greeting",
        "type" : "string"
      } ],
      "response" : "string"
    },
    "echo" : {
      "request" : [ {
        "name" : "record",
        "type" : "CompanyRecord"
      } ],
      "response" : "CompanyRecord"
    },
    "add" : {
      "request" : [ {
        "name" : "arg1",
        "type" : "int"
      }, {
        "name" : "arg2",
        "type" : "int"
      } ],
      "response" : "int"
    },
    "echoBytes" : {
      "request" : [ {
        "name" : "data",
        "type" : "bytes"
      } ],
      "response" : "bytes"
    },
    "error" : {
      "request" : [ ],
      "response" : "null",
      "errors" : [ "CustomException" ]
    },
    "ping" : {
      "request" : [ ],
      "response" : "null",
      "one-way" : true
    }
  }
}

2.2. How to generate avsc from avdl file

java -jar ../tools/avro-tools-1.8.2.jar idl2schemata example.avdl

The result:

{
  "type" : "enum",
  "name" : "CompanySize",
  "namespace" : "com.bettercoding.avro",
  "symbols" : [ "SMALL", "MEDIUM", "BIG" ]
}

{
  "type" : "error",
  "name" : "CustomException",
  "namespace" : "com.bettercoding.avro",
  "fields" : [ {
    "name" : "message",
    "type" : "string"
  } ]
}

{
  "type" : "record",
  "name" : "CompanyRecord",
  "namespace" : "com.bettercoding.avro",
  "fields" : [ {
    "name" : "name",
    "type" : "string"
  }, {
    "name" : "size",
    "type" : {
      "type" : "enum",
      "name" : "CompanySize",
      "symbols" : [ "SMALL", "MEDIUM", "BIG" ]
    }
  }, {
    "name" : "countryCode",
    "type" : "string"
  } ]
}