Avro schema registry hasn’t supprt storing of Avro IDL files yet, so one of the solutions to maintain consistency of Avro schemas is to use Avro IDL file as source of truth and generate avsc files based on it.
Requirements:
- downloaded latest version of avro-tools.jar;
- JDK 8 or Open JDK 8 installed;
- prepared example.avdl file like below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
//Example based on: https://avro.apache.org/docs/1.8.2/idl.html @namespace("com.bettercoding.avro") protocol ExampleProtocol { enum CompanySize { SMALL, MEDIUM, BIG } record CompanyRecord { string name; CompanySize size; string countryCode; } error CustomException { string message; } string hello(string greeting); CompanyRecord echo(CompanyRecord `record`); int add(int arg1, int arg2); bytes echoBytes(bytes data); void `error`() throws CustomException; void ping() oneway; } |
Conversion using avro-tools
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
#java -jar avro-tools-1.8.2.jar Version 1.8.2 of Apache Avro Copyright 2010-2015 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). ---------------- Available tools: cat extracts samples from files compile Generates Java code for the given schema. concat Concatenates avro files without re-compressing. fragtojson Renders a binary-encoded Avro datum as JSON. fromjson Reads JSON records and writes an Avro data file. fromtext Imports a text file into an avro data file. getmeta Prints out the metadata of an Avro data file. getschema Prints out schema of an Avro data file. idl Generates a JSON schema from an Avro IDL file idl2schemata Extract JSON schemata of the types from an Avro IDL file induce Induce schema/protocol from Java class/interface via reflection. jsontofrag Renders a JSON-encoded Avro datum as binary. random Creates a file with randomly generated instances of a schema. recodec Alters the codec of a data file. repair Recovers data from a corrupt Avro Data file rpcprotocol Output the protocol of a RPC service rpcreceive Opens an RPC Server and listens for one message. rpcsend Sends a single RPC message. tether Run a tethered mapreduce job. tojson Dumps an Avro data file as JSON, record per line or pretty. totext Converts an Avro data file to a text file. totrevni Converts an Avro data file to a Trevni file. trevni_meta Dumps a Trevni file's metadata as JSON. trevni_random Create a Trevni file filled with random instances of a schema. trevni_tojson Dumps a Trevni file as JSON. |
How to generate avpr from avdl file
|
1 |
java -jar avro-tools-1.8.2.jar idl example.avdl > example.avpr |
The result:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
{ "protocol" : "ExampleProtocol", "namespace" : "com.bettercoding.avro", "types" : [ { "type" : "enum", "name" : "CompanySize", "symbols" : [ "SMALL", "MEDIUM", "BIG" ] }, { "type" : "record", "name" : "CompanyRecord", "fields" : [ { "name" : "name", "type" : "string" }, { "name" : "size", "type" : "CompanySize" }, { "name" : "countryCode", "type" : "string" } ] }, { "type" : "error", "name" : "CustomException", "fields" : [ { "name" : "message", "type" : "string" } ] } ], "messages" : { "hello" : { "request" : [ { "name" : "greeting", "type" : "string" } ], "response" : "string" }, "echo" : { "request" : [ { "name" : "record", "type" : "CompanyRecord" } ], "response" : "CompanyRecord" }, "add" : { "request" : [ { "name" : "arg1", "type" : "int" }, { "name" : "arg2", "type" : "int" } ], "response" : "int" }, "echoBytes" : { "request" : [ { "name" : "data", "type" : "bytes" } ], "response" : "bytes" }, "error" : { "request" : [ ], "response" : "null", "errors" : [ "CustomException" ] }, "ping" : { "request" : [ ], "response" : "null", "one-way" : true } } } |
How to generate avsc from avdl file
|
1 |
java -jar ../tools/avro-tools-1.8.2.jar idl2schemata example.avdl |
The result:
|
1 2 3 4 5 6 |
{ "type" : "enum", "name" : "CompanySize", "namespace" : "com.bettercoding.avro", "symbols" : [ "SMALL", "MEDIUM", "BIG" ] } |
|
1 2 3 4 5 6 7 8 9 |
{ "type" : "error", "name" : "CustomException", "namespace" : "com.bettercoding.avro", "fields" : [ { "name" : "message", "type" : "string" } ] } |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
{ "type" : "record", "name" : "CompanyRecord", "namespace" : "com.bettercoding.avro", "fields" : [ { "name" : "name", "type" : "string" }, { "name" : "size", "type" : { "type" : "enum", "name" : "CompanySize", "symbols" : [ "SMALL", "MEDIUM", "BIG" ] } }, { "name" : "countryCode", "type" : "string" } ] } |
If you think this tutorial is valuable, please leave me +1 or share it. This action will allow me to reach a wider audience.
Thank you.
Hi article is nice, java convert string to int is a frequently asked question in java interview.
Parsing is a process of converting one data type to another. For example String to integer.
great tutorial! Congrats.
By the way, is there a way to specify that some fields are mandatory/required?
I think – Yes. All are mandatory except these which are unions with null.