ESE Database Forensics: Basics for Investigators

TL;DR. ESE, also called JET Blue, is the embedded database engine behind Windows.edb, SRUDB.dat, WebCacheV01.dat, Active Directory's ntds.dit and Exchange. An ESE file is a header (with a shadow copy), then fixed-size pages organised in B+-trees. The MSysObjects catalog describes every table and column. Rows mix fixed, variable and tagged (sparse) columns. Big values live in a separate long-value tree. Text is often compressed. Changes go to transaction logs first, which is why a live copy is "dirty". Deleted rows are flagged defunct and cleaned up later, and that gap is where carving finds evidence. Knowing this helps you judge any ESE parser's output, including this one.

Why a forensicator should care

You do not need to write an ESE parser. You do need to know enough to answer questions like: why does tool A show 41,000 rows and tool B 40,212? Why is this column empty in one viewer and not in another? Is a dirty database safe to report on? Those answers come from the format.

Microsoft describes ESE as an indexed sequential access method (ISAM) engine with transactions, crash recovery through a write-ahead log, and support for wide tables with many sparse and multi-valued columns. That last point is precisely what Windows Search needs: hundreds of possible properties, a handful set per item. Microsoft published the engine's source code under the MIT licence in 2021. The most useful reference for forensic purposes remains Joachim Metz's libesedb format documentation.

Other ESE databases you may meet in the same case:

The file header

The first page of the file is the header. A second copy, the shadow header, follows it. The fields that matter:

The database state is the first thing to check. esentutl /mh Windows.edb prints it (State: Dirty Shutdown), as shown in Microsoft's archived esentutl VSS post. A dirty shutdown means committed changes may still be only in the logs. See fixing a dirty Windows.edb.

If the primary header is damaged, a parser can fall back on the shadow copy. The Windows Search Index Parser does that and says so in its warnings.

Pages and B+-trees

After the two header pages, the file is an array of pages of the declared size. Each page has a header (40 bytes; 80 bytes in the extended layout used by 16 and 32 KiB pages since Windows 7) and an array of tags at the end of the page pointing to the entries it contains.

Pages form B+-trees. A table's data is one tree; each index is another; long values are a third. Branch pages point to child pages; leaf pages hold the records. Entries in a page can share a common key prefix with the page's first entry, a small compression trick parsers must undo to rebuild keys.

Two page-level details matter in forensics:

• Defunct entries. A deleted record is first flagged in its tag and left in place. It disappears when the page is cleaned up or reorganised.
• Free space. Space released by deletions and page merges is reused later, not wiped at once. Old records can survive there.

The catalog: MSysObjects

MSysObjects is a table like any other, at a well-known location (its root is page 4). It lists every table, column, index and long-value tree, with their IDs, types, codepages and root pages. A parser reads the catalog first, then walks each table's tree. Windows Search tables you will see in it include SystemIndex_PropertyStore, SystemIndex_Gthr and SystemIndex_GthrPth.

Records: fixed, variable and tagged columns

An ESE record has three parts:

SystemIndex_PropertyStore relies heavily on tagged columns: the item has a WorkID and only the properties that were actually set. Multi-valued tagged columns hold lists, such as System.Kind = program; executable. A parser that ignores multi-values quietly drops data.

Long values

A value too big for the record, such as a long content snippet or a big binary property, is stored in the table's long-value tree and the record keeps only a reference (a long-value ID, 4 or 8 bytes depending on the engine version). A parser that does not follow those references shows the property as empty or as a short binary blob. When two tools disagree on whether a snippet exists, this is the first suspect.

Compression

ESE can compress column data. The libesedb documentation describes:

• 7-bit compression for short ASCII or Unicode text: characters packed into 7 bits.
• XPRESS (LZ77) for larger values, identified by a leading byte.
• Newer schemes in recent engine versions, which not every parser implements.

A parser that does not support a given scheme shows garbage or nothing. The Windows Search Index Parser supports 7-bit and XPRESS (plain LZ77) and reports values it could not decode instead of hiding them. XPRESS9 / XPRESS10 / LZ4 are not supported yet.

Transaction logs and the checkpoint

ESE writes changes to sequential log files first (for Windows Search: MSS.log, MSS00001.log…), with MSS.chk recording how far they have been applied. The database file is updated later. The consequences:

• A copy of a running database contains only what was flushed. The rest is in the logs.
• esentutl /r MSS replays the logs into the database (soft recovery). Always on a copy.
• The logs themselves are records of recent changes. Dedicated tools can parse them, which is outside what most parsers do.

Where deleted Windows Search records hide in ESE

The Windows Search Index Parser currently counts defunct entries in the tables it walks and reports the number, without recovering them. More on what survives in evidence of deleted files in the Windows Search index.

A word on validation

ESE is large and details differ between engine versions. The Windows Search Index Parser's ESE reader follows libesedb's documentation and Microsoft's source, and is tested against a synthetic ESE writer. That proves self-consistency, not correctness on every real Windows.edb. So far it has not been validated on real databases, and the UI says so. When a finding matters, compare with esedbexport, ESEDatabaseView or SIDR, as discussed in the parser comparison.

Related articles

• Windows.edb vs Windows.db
• Fixing a dirty-shutdown Windows.edb with esentutl
• Windows Search index forensics: the complete guide