EDRM DupeID and MIH Guide

What is the EDRM?

The Electronic Discovery Reference Model (EDRM) is a globally recognized framework that defines best practices for handling electronic data in legal proceedings. Developed by industry leaders, it guides organizations through eight distinct phases:

The DupeID Project: Solving Deduplication

In February 2023, EDRM launched the Duplicate Identification (DupeID) Project, led by Beth Patterson. Its mission: create a standardized, cross-platform method for generating unique identifiers for email messages.

Goals of DupeID:

• Enable consistent email identification across different systems
• Support deduplication without vendor lock-in
• Facilitate cross-referencing and comparison of datasets
• Reduce costs and timelines in investigations
• Create an open standard complementing proprietary methods

The outcome was the EDRM Message Identification Hash (MIH)—a simple yet elegant solution to a decades-old problem.

The Problem: Vendor Lock-In and Data Redundancy

In multi-custodian investigations involving multiple email platforms, duplicate emails dramatically inflate data volumes. A typical case might involve:

• Multiple email platforms (Gmail, Microsoft 365, Yahoo, IMAP servers)
• Various mailbox formats (PST, OST, mbox, EML, MSG)
• Different custodian accounts across organizations
• Historical archives from legacy systems

The result? Massive redundancy. The same email thread might appear dozens of times across different custodians, formats, and systems.

The Historical Solution: Costly Reprocessing

Historically, specialized tools offered deduplication—but only within their own proprietary ecosystems. Each vendor used unique algorithms that couldn’t communicate across platforms.

The traditional approach required:

1. Collect data from all sources
2. Ingest everything into a single platform
3. Deduplicate within that platform’s ecosystem
4. Accept vendor lock-in or face costly data migration

The Real-World Impact

Consider a typical multi-custodian investigation:

The lack of a standardized, cross-platform method for email identification was costing the industry billions of dollars annually.

What is the EDRM MIH?

The EDRM MIH is an MD5 hash value generated from the Message-ID field in an email’s SMTP header:

Why the Message-ID Field?

The Message-ID is a unique identifier assigned to emails by mail servers when messages are sent. Defined in RFC 822 (and later RFC 2822 and RFC 5322), it’s designed to be globally unique:

Key characteristics:

• Globally unique across all email systems
• Assigned at message creation time
• Preserved when emails are forwarded, replied to, or migrated
• Present in the vast majority of received emails

By hashing this field with MD5, the EDRM MIH creates a 128-bit fingerprint that uniquely identifies an email across any platform or vendor.

The MIH’s Limitation: Null Values

While elegant, the EDRM MIH specification has one significant limitation:

This creates a significant gap in cross-platform deduplication capabilities—a gap that Aid4Mail’s MIH+ was designed to close.

Introducing MIH+: Guaranteed Non-Null Hash Values

To address the limitations of EDRM MIH while maintaining full compatibility with the standard, Aid4Mail developed MIH+—an enhanced variant that guarantees a non-null value for every email.

How Aid4Mail Generates MIH+ for Messages Without Message-ID

When the Message-ID field is missing, Aid4Mail constructs a hash source using available metadata in a specific order of precedence:

Example: For a draft email without a Message-ID:

From: alice@company.com
Date: 2025-03-15 14:30:00
Subject: Q1 Budget Review

MIH+ = MD5("alice@company.com" + "2025-03-15 14:30:00" + "Q1 Budget Review")

This approach ensures:

• Every email gets a unique identifier
• Cross-platform comparison is always possible
• Deduplication can occur for all message types

MIH+ vs. Standard MIH: Compatibility

Performance-Optimized Architecture

While MIH+ provides critical compatibility and cross-platform identification capabilities, Aid4Mail uses a performance-optimized approach for actual deduplication operations during processing.

Why not use MIH+ directly for deduplication?

MD5 hash generation and comparison, while reliable, is computationally expensive when processing hundreds of thousands or millions of emails. Aid4Mail implements a two-tier hashing system:

Benefits of this architecture:

• 10× faster deduplication during processing
• Minimal memory footprint for large collections
• Full EDRM MIH+ compatibility for cross-platform workflows
• Best of both worlds: performance AND interoperability

Aid4Mail provides comprehensive MIH+ support across multiple features, enabling forensic examiners and eDiscovery professionals to leverage EDRM standards throughout their workflows.

1. Search and Filtering on MIH+ Values

Available in: Aid4Mail Investigator and Enterprise editions

The MIH_Plus search operator enables precise filtering based on MIH+ hash values. This is particularly powerful when working with datasets from multiple vendors.

Use cases:

1. Deduplication across vendors: Import MIH+ values from RelativityOne, Nuix, or other platforms and exclude matching emails from Aid4Mail processing
2. Targeted collection: Use MIH+ lists to identify and collect specific emails across multiple custodians
3. Cross-reference verification: Confirm that specific messages exist in both your dataset and a vendor’s production
4. Privileged document tracking: Maintain MIH+ lists of privileged communications and automatically filter them across all custodians

2. File Naming with MIH+ Signatures

When exporting emails to .eml, .msg, or .txt formats, Aid4Mail can use MIH+ values as file names.

Advantages:

• Consistent naming: Same email always has the same filename across all exports
• Cross-platform identification: Files can be matched across different processing runs
• No length limitations: Unlike subject-based naming, MIH+ produces fixed-length 32-character names
• Special character handling: No illegal characters or truncation issues
• Deduplication at OS level: Operating system tools can identify duplicates by filename

3. Template Token: {MIH_Plus}

The {MIH_Plus} template token inserts MIH+ values into custom file and folder names, email headers, and metadata exports.

4. Metadata Extraction

Aid4Mail’s Column Configuration Editor includes the EDRM.MIH_Plus token for extracting MIH+ values to CSV, XML, JSON, and TSV files.

Subject,From,Date,MIH_Plus,Folder
Q1 Budget Review,alice@company.com,2025-03-15,7b7e8488...,Inbox
Re: Q1 Budget Review,bob@company.com,2025-03-16,3d4f2a1e...,Sent Items

This enables:

• Cross-platform deduplication in Excel or databases
• Custom analysis scripts using MIH+ as the join key
• Import into other tools for further processing

5. Exporting MIH+ Lists

Aid4Mail can generate plain-text lists of MIH+ values—ideal for sharing with other platforms or vendors.

7b7e8488d0b11ff6dd30064fa5ff79c1
3d4f2a1e9c8b7a6d5e4f3a2b1c0d9e8f
a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6

Use cases:

• Share with opposing counsel for agreed deduplication
• Import into Relativity, Nuix, or other platforms
• Create privilege logs or production indexes
• Document chain of custody

1. Cost Savings

2. Time Efficiency

• Faster Investigations: Immediate deduplication across all datasets without waiting for single-platform reprocessing
• Accelerated Review: Smaller, deduplicated datasets enable focus on unique, responsive content
• Streamlined Workflows: Consistent identifiers across all tools simplify communication with co-counsel

3. Improved Accuracy and Defensibility

4. Flexibility and Vendor Independence

• Multi-Platform Workflows: Use best-in-class tools for each task without forced platform lock-in
• Collaborative Investigations: Share datasets with confidence and coordinate with multiple law firms
• Future-Proofing: EDRM standard ensures long-term compatibility and protects investment in processed data

Edition Requirements

MIH+ functionality is available in all Aid4Mail editions:

Basic Configuration

Aid4Mail’s default configuration already generates MIH+ values optimally. No special setup is required.

Using MIH+ for Cross-Platform Deduplication

Scenario: You want to exclude emails already reviewed in RelativityOne.

Exporting MIH+ Lists

To create a MIH+ list from your Aid4Mail collection:

1. Select Plain Text as target format
2. Enable Export to a single text file
3. Under Email header configuration, choose Only EDRM MIH values
4. Apply desired filters (e.g., Class:responsive if using AI classification)
5. Process

Result: A text file with one MIH+ value per line, ready to share with other platforms.