How to enable LZO compression on HDInsight

Microsoft Azure

This blog post explains how to enable LZO compression on a HDInsight cluster.

ARM Template

You will need to modify the ARM template configuration and under the clusterDefinition, configuration section:

  •  Add core-site section and specify the codecs and compression codec class
  • Add a mapred-site enable map output compression and the compression codec class

Install compression libraries on cluster nodes

You will also need to install the compression libraries on the cluster nodes.

On the point of compression libraries, if you are using snappy you will need to install the snappy compression libraries with:

Leave a Reply

Your email address will not be published. Required fields are marked *