Hadoop Encryption Built on Hadoop Compression Codec Framework

Table of Contents

• Table of Contents
• Overview
• Highlights
• Crypto Design
  • General Encryption Structure Built on Hadoop Compression
  • Key Alias Centric Design
  • Build-In Crypto Providers
• Encryption Codec Design
  • Block Design
  • Stackable
  • Splittable
  • Detailed Codec Design
• Deployment
  • Components Required
  • Configuration (Hadoop 2.3.0)
  • Crypto Provider
• Using the HadoopEncryptor/HadoopDecrptor Command Line Utility
• Using AESCodec in MapReduce
• Using AESCodec Programmatically
• References

Overview

Sensitive data is stored safely at rest by encrypting it.  Haddop common provides a codec framework for compression algorithms.  A crypto codec framework builds on the compression codec framework because encryption algorithms require additional configuration and methods for key management.  The crypto codec cleanly distinguishes crypto algorithms from compression algorithms and carries extended interfaces where necessary while sharing common interfaces where possible.

The design and implementation is based on the Apache open source project: “Hadoop crypto codec framework and crypto codec implementations” located at  https://issues.apache.org/jira/browse/HADOOP-9331.  (Link test 27 Jun 2014)

The original design requires some patches on Hadoop Distribution; this modification tries to avoid this inconvenience. Also, the crypto portion of the original design is simplified by focusing on Key Alias (Key Profile is the term used in HADOOP-9331) centric interfaces.

Highlights

• Open source
• Stackable (link encryption with compression)
• Splittable, suitable for MapReduce file splits
• Highly configurable
• Provide crypto, key source and crypto provider interfaces that could be implemented easily by third party.
• Provide build-in Pi Soft Provider - a production level, comprehensive implementation of key storage, key management, authentication and authorization.
• No need to patch existing Hadoop distribution for easy deployment.

Crypto Design

This design is AES (Advanced Encryption Standard) specific, however, other cryptographic algorithms can be used. The Pi Soft provider uses AES-NI native instruction on Intel CPU which greatly improves the encryption performance.

Unlike other solutions, the key used to encrypt and decrypt data is not stored in plaintext with the data. Implementations of a codec store a Key Alias with data instead. The Key Alias is used to retrieve the key on demand as needed for decryption or re-encryption. This design provides the flexibility to choose different key storage mechanisms such as the Pi Soft distributed key store or a third party Key Management Systems which may have other desired capabilities

The encryption framework borrows from and maintains the type hierarchy and API conventions of the existing Hadoop compression codec framework. Encryptors and decryptors share the interfaces of compressors and decompressors respectively.

General Encryption Structure Built on Hadoop Compression

As shown in above class diagram, BlockEncryptor/BlockDecryptor has the same method signature as Compressor/Decompressor.  Compression simply forwards the call to encryption.

public abstract class
BlockEncryptor implements Encryptor {
  /**
   * @deprecated Use encrypt instead
   */

  @Override
  @Deprecated

  public final int compress(byte[] b, int off, int len) throws IOException {
    return encrypt(b, off, len);
  }
  public abstract int encrypt(byte[] b, int off, int len) throws IOException;
  . . .  
}

Key Alias Centric Design

As described above, this design separates the key from the data and constructs an encrypted reference between the data and its managed encryption key.  The real key used to encrypt and decrypt the data is stored in plaintext with data. Implementations of a codec can store a Key Alias with the data instead. The Key Alias is used to retrieve the key on demand as needed for decryption.

As a result, the Crypto and KeySource interface is “key alias” centric. As shown in the Encryption Codec Design  section, the Crypto and KeySource interface is implemented the key alias will be carried automatically as metadata with the file.

KeySource Interface

public interface KeySource {
    void init(Configuration conf);    
    Key createNewKey() throws CryptoException;    
    Key getKey(String alias) throws CryptoException;    
    void reset();    
}

KeySource is an interface that abstracts the key generation and correlates a key alias with the cryptographic Key.  Be sure to pay attention to the returned java Key, as shown later in Simple Provider (Reference Implementation) section, key alias information is put together with the Key.  You can consider the KeySource as a key store.

Crypto Interface

public interface Crypto {
    void init(Configuration conf, KeySource ks);
    int encrypt(ByteBuffer input, ByteBuffer output) throws CryptoException;
    int decrypt(ByteBuffer input, ByteBuffer output) throws CryptoException;
    void setKeyAlias(String alias) throws CryptoException;
    String getKeyAlias() throws CryptoException;
    int getKeySize();    
    void reset();
}

The crypto interface declares encrypt and decrypt. The method signature follows the same contract as the  java Cipher doFinal method. KeySource is passed to help get an existing key from the key alias or to create a new key and the new key alias.

CryptoProvider Interface

public interface CryptoProvider {
    Crypto createCrypto();
    KeySource createKeySource();
}

CryptoProvider is a factory-like interface to create concrete implementation of Crypto and KeySource.  The “dfs.crypto.provider” property decides which provider to use at runtime. The codec initialization will load the instance into the Crypto Providers Object Factory and use it to create a corresponding Crypto as well as KeySource.

Crypto and  KeySource Design Diagram

Build-In Crypto Providers

Pi Soft Provider

The Pi Soft Crypto Provider is based on the Pi Soft Suite framework and the Pi Soft JCE Provider.

Pi Soft is an encryption key management methodology patented by Pi Soft to separate encrypted data from its symmetric encryption keys. (http://www.eruces.com)

Pi Soft encrypts every logical data item with a unique key, separates the key from the data, and constructs an encrypted reference between the data and its managed encryption key.  This reference is called a T-Tag™; T-Tags are stored with the encrypted data.

• Encryption Keys Centrally Stored
• Ownership and Access Rights Applied to Encryption Keys
• Reference Between Encrypted Data and its Encryption Key is Encrypted

The Pi Soft JCE Provider uses the Pi Soft Suite for encryption and decryption through standard Java JCE interface.

Pi Soft Crypto Provider Dependencies

• Pi Soft JCE Provider
• Pi Soft Suite key service: a full-fledged, production level key management system that provides comprehensive key management, authentication, authorization, etc.

Simple Provider (Reference Implementation)

The source has another build-in implementation of CryptoProvider: SimpleProvider.  It is a very light implementation compared to the Pi Soft provider and is presented only as a demonstration.

KeySourceSimple Implementation

public interface CryptoProvider {
    Crypto createCrypto();
    KeySource createKeySource();
}

Cipher:  AES/CBC/PKCS5Padding

Key: 16 bytes random number

IV: 16 bytes random number

KeyStore: a plain file that saves only raw keys

Key Alias: offset of the key in the file

package org.apache.hadoop.io.crypto.aes.simple;
import java.security.Key;
import java.util.Random;
import org.apache.commons.codec.binary.Base64;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.crypto.CryptoException;
import org.apache.hadoop.io.crypto.KeySource;
/**
 *    Key alias: offset of the file 
 *    raw key: 16 bytes, base64 encoded written to the simpledb file. 
 */
public class KeySourceSimple implements KeySource {
    
    private Configuration conf = null;
    private SimpleAppendableFile file = null;
    @Override
    public void init(Configuration conf) {
        // TODO Auto-generated method stub
        this.conf = conf;
        String uri = conf.get("dfs.crypto.aes.simpledb") ;
        if(uri == null    )
            throw new RuntimeException("dfs.crypto.aes.simpledb must be given");
        
        file = new SimpleAppendableFile(conf, uri);
    }
    @Override
    public Key createNewKey() throws CryptoException {
        // TODO Auto-generated method stub
        byte[] key_material = new byte[16];
        String alias = null;
        new Random().nextBytes(key_material);
        try {
            long pos = file.append(Base64.encodeBase64String(key_material));
            alias = String.valueOf(pos);
            
        }
        catch(Exception e) {
            throw new CryptoException(e);
        }
        SimpleAliasKey key = new SimpleAliasKey(key_material, "AES", alias);
        return key;
    }
    @Override
    public Key getKey(String alias) throws CryptoException {
        // TODO Auto-generated method stub
        try {
            long pos = Long.valueOf(alias);
            String encodedKey = file.read(pos);
            byte[] key = Base64.decodeBase64(encodedKey);
            return new SimpleAliasKey(key, "AES", alias);
        }catch(Exception e) {            
            throw new CryptoException(e);
        }
    }
    @Override
    public void reset() {
        // TODO Auto-generated method stub
        if(file != null && conf !=null) {
            file.closeStream();
            init(conf);
        }
        
    }
}

The above is a simple KeySource implementation. A plain file is used to keep the raw keys. Alternatively, keys could be stored to a KeyStore, a database, distributed storage, or another production-level key management system (similar to Pi Soft).

The key store file is accessed through a Hadoop FSDataInputStream and FSDataOutputStream. It can be put under any Hadoop FileSystem supported file system like HDFS, Local, WebHDFS, HAR, FTP, etc. The only exception is Amazon's S3 since S3 does not support the append call (reopen-append).

As described earlier, the Key returned should include key alias information because the Crypto interface requires the key alias information to return from getKeyAlias().  Here, we simply wrap a java SecretKeySpec and keep the key alias with the key itself.

package org.apache.hadoop.io.crypto.aes.simple;
import java.security.Key;
import javax.crypto.spec.SecretKeySpec;

public class SimpleAliasKey implements Key {

	SecretKeySpec key;
	String alias;
	
	SimpleAliasKey(byte[] key, String algorithm, String alias) {
		this.key = new SecretKeySpec(key, algorithm);
		this.alias = alias;
	}
	
	@Override
	public String getAlgorithm() {
		// TODO Auto-generated method stub
		return key.getAlgorithm();
	}

	@Override
	public byte[] getEncoded() {
		// TODO Auto-generated method stub
		return key.getEncoded();
	}

	@Override
	public String getFormat() {
		// TODO Auto-generated method stub
		return key.getFormat();
	}
	
	public String getAlias() {
		return alias;
	}
}

CryptoSimple Implementaion

We use java crypto API to do the encryption/decryption.  In particular, we put the key alias and IV together and then return it to the Crypto getKeyAlias call. This is just for demonstration purpose. In actual implementation, put the IV wherever it is most suitable for your solution.

In the code showed below, KeySoure manages keys, Crypto works as a cipher to encrypt/decrypt data. The link between Crypto and KeySource is the Key Alias.

package org.apache.hadoop.io.crypto.aes.simple;

import java.nio.ByteBuffer;
import java.security.Key;
import java.util.Random;

import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;

import org.apache.commons.codec.binary.Base64;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.crypto.Crypto;
import org.apache.hadoop.io.crypto.CryptoException;
import org.apache.hadoop.io.crypto.KeySource;
/**
 * 
 *	Key Alias format:  raw key alias + ":" + base64_encoded(iv), where iv is byte[], 16 bytes.
 *	raw key alias: offset of the simple key db file. 
 *
 *	simple key file data: base64 encoded keys 
 *	key: 16 bytes randomly generated
 */
public class CryptoSimple implements Crypto {

	private KeySource ks;
	private Key key;
	private byte[] iv;
	
	private static final String CIPHER_NAME = 	"AES/CBC/PKCS5Padding";
	private static int IV_LENGTH = 16;
	

	@Override
	public void init(Configuration conf, KeySource ks) {
		// TODO Auto-generated method stub
		this.ks = ks;
	}

	@Override
	public int encrypt(ByteBuffer input, ByteBuffer output)
			throws CryptoException {
		// TODO Auto-generated method stub
		try {
			Cipher cipher = Cipher.getInstance(CIPHER_NAME);
			cipher.init(Cipher.ENCRYPT_MODE, getKey(), new IvParameterSpec(iv));
			return cipher.doFinal(input, output);
		 
		}
		catch(Exception e) {
			throw new CryptoException("encrypt", e);
		}
	}

	@Override
	public int decrypt(ByteBuffer input, ByteBuffer output)
			throws CryptoException {
		// TODO Auto-generated method stub
		try {
			Cipher cipher = Cipher.getInstance(CIPHER_NAME);
			cipher.init(Cipher.DECRYPT_MODE, getKey(), new IvParameterSpec(iv));
			return cipher.doFinal(input, output);
		 
		}
		catch(Exception e) {
			throw new CryptoException("encrypt", e);
		}
	}

	@Override
	public void setKeyAlias(String alias) throws CryptoException {
		// TODO Auto-generated method stub
		String[] key_iv = alias.split(":");
		if(key_iv.length != 2 )
			throw new CryptoException("Wrong format of key alias");
		
		key = ks.getKey(key_iv[0]);
		byte[] decodedBytes = Base64.decodeBase64(key_iv[1]);
		if(decodedBytes.length != IV_LENGTH)
			throw new CryptoException("Wrong format of key alias");
		iv = decodedBytes;
	}

	@Override
	public String getKeyAlias() throws CryptoException {
		// TODO Auto-generated method stub
		SimpleAliasKey aliasKey = (SimpleAliasKey)getKey();
		String rawkey_alias = aliasKey.getAlias();
		return String.format("%s:%s", rawkey_alias, Base64.encodeBase64String(iv));
	}

	@Override
	public int getKeySize() {
		try {
			// TODO Auto-generated method stub
			return getKey().getEncoded().length * 8;
		}catch(Exception e) {
			throw new RuntimeException(e);
		}
	}

	@Override
	public void reset() {
		// TODO Auto-generated method stub
		key = null;
		if(ks != null)
			ks.reset();
	}
	
	private Key getKey() throws CryptoException{
		if(key != null)
			return key;
		else {
			key = ks.createNewKey();
			updateIV();
			return key;
		}			
	}
	
	private void updateIV() {
		if(iv == null) {
			iv = new byte[IV_LENGTH];
		}
		
		new Random().nextBytes(iv);
	}
}

SimpleProvider Implementation

The following code snippet explains the logic. 

package org.apache.hadoop.io.crypto.aes.simple;

import org.apache.hadoop.io.crypto.Crypto;
import org.apache.hadoop.io.crypto.CryptoProvider;
import org.apache.hadoop.io.crypto.KeySource;

public class SimpleProvider implements CryptoProvider {

	@Override
	public Crypto createCrypto() {
		// TODO Auto-generated method stub
		return new CryptoSimple();
	}

	@Override
	public KeySource createKeySource() {
		// TODO Auto-generated method stub
		return new KeySourceSimple();
	}
}

This class is registered with the CryptoProviders Object Factory when the codec is initialized.  Once the dfs.crypto.provider property is set (Please refer to the Common Properties section ) , runtime will pick the provider specified to perform crypto function and key management.

Encryption Codec Design

Block Design

To carry/persist important metadata like key alias, IV, stream header etc., current design logically separate data with blocks.

File structure:
+----------------------------------------------+
| block      | block     | block     | ...     |
+----------------------------------------------+
 
block structure :
+----------------------------------------------------------------------------------------------------+
| 16 byte sync mark | block header | algorithm header | 4 byte original size | 4 byte encrypted size |
+----------------------------------------------------------------------------------------------------+
|   encryption data ...                                                                              |
+----------------------------------------------------------------------------------------------------+
 
encryption data structure:
+----------------------------------------------------------------------------------------------------+
| 4 byte compressed size  |  compressed data    | 4 byte compressed size  | compressed data  | ...   |
+----------------------------------------------------------------------------------------------------+
 
block header structure :
+------------------------------------------------+
| 4 byte version  |  key alias      | extension  |
+------------------------------------------------+
 
The AES algorithm header structure :
+-----------------------------------------------------------------+
| 4 byte stream header length  |  stream header    | 16 byte IV   | 
+-----------------------------------------------------------------+
 
 
The AES stream header structure :
+----------------------------------------------------------+
| AES cryptographic length  |  1 byte 'compressed' flag    | 
+----------------------------------------------------------+

Some sections are reserved for your customizations, e.g. extension. Currently the IV section is not being used in this implementation (but is used in the original design). Merge it with key alias or look it up by the key alias. Theoretically, if the Key can be looked up by key alias, so can IV.

The only abstracted and customizable field is key alias which is exactly what the design concentrates on. Once Crypto and KeySource interfaces are implemented key alias will be saved and retrieved automatically during encryption/decryption.  Please refer Key Alias Centric Design

Both encryption block size (256K by default) and compression block size (64K by default) can be configured. Do not confuse encryption/compression block size with HDFS block size which is much bigger. Encryption/compression block size is more sensitive to memory/CPU.

Smaller compressions block (chunk) size results in much faster performance but less compression ratio.  Moreover, compressor/decompressor, inputstream/outputstream internal block works as data buffer to perform the encryption/decryption, compression/decompression, which could make IO operations more efficient.

As shown in above diagram, both Encryptor/Decryptor and InputStream/OutputStream are implemented by taking block into consideration.

Stackable

Hadoop can only configure one compressor at a time. Since encryption is implemented through Hadoop compression, we are responsible for preserving compression and linking it with encryption.

Compression and encryption can be stacked. Compression is always performed prior to encryption. Multiple blocks of compressed data will be fed into one encryption block.

encryption data structure:
+----------------------------------------------------------------------------------------------------+
| 4 byte compressed size  |  compressed data    | 4 byte compressed size  | compressed data  | ...   |
+----------------------------------------------------------------------------------------------------+

The design makes underlying compression optional and configurable. Please refer to the Configuration section: Common Properties

As shown on the diagram below, BlockEncryptorWithCompressor is in charge of linking encryption with compression.  It has two attributes, encryptor and compressor. “encryptor” here will be AESEncryptor, and the “compressor” is any Hadoop distributed compressor like GZip, DEFLATE, bzip2, LZO, LZ4, or Snappy.

“compressor” can be null if the dfs.crypto.compressor property not specified.

Splittable

Spittable indicates whether the compression format supports splitting, that is, whether you can seek to any point in the stream and start reading from some point further on. Splittable compression formats are especially suitable for MapReduce, otherwise MapReduce InputFormat will not create splits for compressed files, which results in not being able to take advantage of the data locality provided by HDFS.

To support splittable Codec, the codec MUST implement the SplittableCompressionCodec interface which requires returning SplitCompressionInputStream (link tested 27 Jun 2014) while createInputStream.

The splittable stream is normally designed to start with sync marker to make it seekable.

+----------------------------------------------+
| block      | block     | block     | ...     |
+----------------------------------------------+
    block 1                    block 2
+-----------------------------------------------------------------------------+
| 16 byte sync mark | rest of the block| 16 bytes sync mark |rest of the block|
+-----------------------------------------------------------------------------+

Basic Logic Behind File Split in MapReduce

If compression is detected, file split will pass start and end of the split to the codec to create SplitCompressionInputStream, find the start and end of the sync mark, adjust the real start and end, and seek to the adjusted start. Then the stream will read from that adjusted position, start reading data block by block.

As showed above, the two arrows point to the adjusted start” and end”. Logical boundary could be cross file split boundary.

Detailed Codec Design

Deployment

Components Required

Name	Description
HadoopCryptoCompressor.jar	codecs, compressor, decompressor and block input/output streams org.apache.hadoop.io.crypto.aes.AESCodec is the main exposed codec.
HadoopCrypto.jar	Declares crypto, key source, and provider interface. Includes two implementations of above interfaces: Pi Soft provider using Pi Soft JCE and Simple provider using a simple reference implementation .
HadoopEncryptor.jar	Command line tool to encrypt a file. Supports any  FileSystem that Hadoop implemented: e.g. HDFS, Local, WebHDFS, S3, etc.
HadoopDecryptor.jar	Command line tool to decrypt a file. Supports any  FileSystem that Hadoop implemented: e.g. HDFS, Local, WebHDFS, S3, etc.
ErucesOnlineJce-.jar	Pi Soft JCE provider  is the version number of latest build.
TEAdmin-.jar	Dependency jar for Pi Soft JCE provider.  is the version number of latest build.
liberucesjce.so	JNI library that required by Pi Soft JCE provider.

Configuration (Hadoop 2.3.0)

Environment Variables

Update $HADOOP_HOME/etc/hadoop/hadoop-env.sh

1.    Put above jars into a user defined path and add or update the following line in hadoop-env.sh.

export HADOOP_CLASSPATH=$HADDOP_CLASSPATH:/*

2.    Add or update following line to use the Pi Soft crypto provider.

export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:

Update $HADOOP_HOME/etc/hadoop/core-site.xml

1.    Add org.apache.hadoop.io.crypto.aes.AESCodec to the  io.compression.codecs property

io.compression.codecs

org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec, org.apache.hadoop.io.crypto.aes.AESCodec

Crypto Provider

To use different crypto providers, a few configuration properties need to be provided. A good place to set these properties is a separate xml configuration file. You can pass it either through  -conf   Hadoop generic options or programmatically calling the Configuration addResource method.

Common Properties

dfs.crypto.compressor: underlying compression codec, [optional].
Please note that the encryption codec is built on top of the compression interface.  To make it stackable (chain encryption with compression), set any Hadoop provided compression codec as the underlying compression codec, e.g. org.apache.hadoop.io.compress.GzipCodec.  When used, encryption happens after compression.  If this property is not supplied, only encryption will be performed.

dfs.crypto.provider: two Pi Soft providers have been implemented here:

• Pi Soft Crypto Provider, a full-fledged, production level key management system which provides comprehensive key management, authentication, authorization, etc. All cryptographic functions are provided through the Pi Soft JCE.
  Property value: org.apache.hadoop.io.crypto.aes.ErucesProvider

• Simple Provider: a reference implementation to demonstrate how to implement another crypto provider.  This provider could be your own provider that implemented Crypto, KeySource and CryptoProvider interfaces.
  Property value: org.apache.hadoop.io.crypto.aes.SimpleProvider

Pi Soft Provider Properties

<configuration>
	<property>
	    <name>com.eruces.security.ServerName</name>
        <value>10.60.2.40</value>
		<description> IP address of ERUCES Key Service</description>
    </property>
	<property>
        <name>com.eruces.security.ServerPort</name>
        <value>8888</value>
		<description> Port number of ERUCES Key Service</description>
    </property>
    <property>
        <name>com.eruces.security.KeyStoreAuthenticationType</name>
        <value>Default</value>
		<description> authentication type, which could be “Default” (Certificate), “Token”, or “Kerberos” </description>
    </property>
    <property>
        <name>com.eruces.security.auth.PKCS12File</name>
        <value> hdfs://localhost/jce/hadoop.pfx</value>
		<description> PKCS12 file name, required if com.eruces.security.KeyStoreAuthenticationType is set to “Default”, 
						it could be a file on any FileSystem that Hadoop supports, e.g. HDFS, Local, S3, WebHDFS, etc.  
		</description>
    </property>
    <property>
        <name>com.eruces.security.auth.PKCS12FilePass</name>
        <value>password1A</value>
		<description> key store password, required if com.eruces.security.KeyStoreAuthenticationType is set to “Default” </description>
    </property>
    <property>
        <name>com.eruces.security.auth.Token</name>
        <value></value>
		<description> Token credential, required if com.eruces.security.KeyStoreAuthenticationType is set to “Token”</description>             
    </property>
    <property>
    	<name>dfs.crypto.provider</name>
        <value>org.apache.hadoop.io.crypto.aes.ErucesProvider</value>
		<description> Crypto Provider class name.</description>
	</property>
</configuration>

Simple Provider Properties

<configuration>
	<property>
		<name>dfs.crypto.provider</name>
		<value>org.apache.hadoop.io.crypto.aes.simple.SimpleProvider</value>
	</property>
	<property>
		<name>dfs.crypto.aes.simpledb</name>
		<value>hdfs://localhost/simple.db</value>
	</property>
	<description>a (distributed) file based simple file db that saves all the raw keys. Key alias is the offset position of the key in file db. Note:
			do not use simpledb in S3 because S3 does not support append().
	</description>
</configuration>	

Using the HadoopEncryptor/HadoopDecrptor Command Line Utility

Usage:

HadoopEncryptor:

hadoop jar HadoopEncryptor.jar -conf

HadoopDecryptor:

hadoop jar HadoopDecryptor.jar -conf

Arguments

is the provider specific xml configuration file mentioned above, please refer to the Configuration (Hadoop 2.3.0) section of this document.

is the input file to be encrypted/decrypted. It can be any Hadoop supported FileSystem file, e.g. HDFS, Local, S3, WebHDFS, etc.

is the output file to be encrypted/decrypted. It can be any Hadoop supported FileSystem file, e.g. HDFS, Local, S3, WebHDFS, etc.

Note:

• Currently, “eruces_aes” is the registered extension for AESCodec. For Map/Reduce to recognize your file as a compressed file, use “.eruces_aes” as the file extension for the encrypted-compressed file.
• Be sure to set dfs.crypto.compressor property (please refer to Common Properties) to have both encryption and compression. Specify it either through the configuration xml or -D Hadoop generic options, e.g, -D **dfs.crypto.compressor=**org.apache.hadoop.io.compress.GzipCodec

Using AESCodec in MapReduce

To use encryption in MapReduce, follow the same steps that are required for using compression in MapReduce.

Note:

• Be sure to configure properly, please refer to Configuration (Hadoop 2.3.0), especially Use Crypto Provider section
• Use the HadoopEncrytor tool to encrypt-compress files that will be used as a MapReduce input file, giving .eruces_aes as the file extension.  The MapReduce InputFormat will automatically recognize it is as AESCodec encrypted and perform the file split accordingly.
• AESCodec is designed to support splitting (splittable). Splittable compression formats are especially suitable for MapReduce (while InputFormat creates splits).
• To encrypt-compress the output of a MapReduce job, set the mapred.output.compress property to true in the job configuration and the mapred.output.compression.codec property to org.apache.hadoop.io.crypto.aes.AESCodec.

Sample command (Word Count MapRedue, input files are encrypted-compressed):

hadoop jar HadoopMRWordCount.jar
-D dfs.crypto.compressor=org.apache.hadoop.io.compress.GzipCodec
-fs file:///
-conf jce/jce-conf.xml
/hadoop_data/mr_input
/hadoop_data/mr_output

Using AESCodec Programmatically

To use AESCodec for encryption-compression, follow the same steps required by Hadoop Compression with two additions. Specify the codec as org.apache.hadoop.io.crypto.aes.AESCodec and be sure to set the configuration properly (please refer to Configuration (Hadoop 2.3.0)) section.

The following code snippet shows how to use AESCodec for encryption.

String codecClassName = "org.apache.hadoop.io.crypto.aes.AESCodec";

Configuration conf = getConf();
//or = new Configuration() if Tool interface is not implemented.

String input = args[0];
String output = args[1];
          
Class<?> codecClass = Class.forName(codecClassName);
CompressionCodec codec =
(CompressionCodec)ReflectionUtils.newInstance(codecClass, conf);
FileSystem fsIn = FileSystem.get(URI.create(input), conf);
FileSystem fsOut = FileSystem.get(URI.create(output), conf);

InputStream in = null;
OutputStream out = null;

try {
       in = fsIn.open(new Path(input));
       out = fsOut.create(new Path(output));
       CompressionOutputStreamoutCompress = codec.createOutputStream(out);
       IOUtils.copyBytes(in, outCompress, conf, false);
       outCompress.finish();
	}

finally {
       IOUtils.closeStream(in);
       IOUtils.closeStream(out);
}

References

[1] Tom White, “Hadoop - The Definitive Guide”

[2] Hadoop crypto codec framework and crypto codec implementations, https://issues.apache.org/jira/browse/HADOOP-9331

[3] Hadoop Crypto Design, https://issues.apache.org/jira/secure/attachment/12571116/Hadoop%20Crypto%20Design.pdf

(links test 27 Jun 2014)