Notes Storage Facility (NSF) database file format

1 Notes Storage Facility (NSF) database file formatAnalysis of the NFS database file formatBy Joachim metz Notes Storage Facility (NSF) database file is used by (IBM) Lotus Notes and Domino to store different kind of objects like e-mail, appointments and documents, but also application forms and document is intended as a working document for the NSF specification. Which should allow existing Open Source forensic tooling to be able to process this partitioning iDocument informationAuthor(s):Joachim metz document contains information about the Notes Storage Facility (NSF) database :PublicKeywords:Lotus Notes , Lotus Domino, Notes Storage Facility , NSFL icenseCopyright (c) 2010-2012 Joachim metz is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.

2 A copy of the license is included in the section entitled "GNU Free Documentation License". MetzFebruary 2010 March 2010 April 2010 July 2010 Initial MetzNovember 2012 Email version iiTable of Contents1. File Notes /Domino Summary and Data and NSF data NSF date and NSF file On-disk structure Byte Word Long The file The database database format database database replication Replication database information Special note identifier database information database information database information database information Backup Flags Superblock Superblock The summary bucket descriptor The summary bucket page The summary bucket group The summary bucket The non-summary bucket descriptor The non-summary bucket page The non-summary bucket group The non-summary bucket Bitmap Data note identifier

3 Modified note Folder Directory Superblock Bucket Descriptor Bucket descriptor block Bucket descriptor block Record Relocation Vector bucket Unique name key Unique name key table The note item The note item CLASS_NOCOMPUTE note item CLASS_ERROR note item CLASS_UNAVAILABLE note item CLASS_NUMBER note item CLASS_TIME note item CLASS_TEXT note item CLASS_FORMULA note item CLASS_USERID note item Unique name key hash Bucket descriptor block Record Relocation Vector RRV (container) bucket RRV Basic RRV BSID RRV entry format version 22 and Allocation Universal identifier The The bucket Bucket Bucket index The The note Note Note status Secondary note status Non-summary data The note item The note item field Summary note item The TYPE_TEXT_LIST note item Non-summary note item Trailing Compression CX LZ1 The access control Replication Replication Encryption User activity A.

4 B. GNU Free Documentation ivpage v1. OverviewThe Notes Storage Facility (NSF) database file is used by (IBM) Lotus Notes and Domino to store different kind of objects like e-mail, appointments and documents, but also application forms and NSF is also referred to as an object File structureAn Notes Storage Facility (NSF) database file consist of the following distinguishable elements: notesCharacteristicsDescriptionByte orderlittle-endianDate and time valuesCharacter stringLotus Multi-Byte Character Set (LMBCS)File size in increments of 65536? LayoutNSF database layout: file header database header various superblock bucket descriptor block bitmap Record Relocation Vector bucket summary buckets non-summary bucketsAn NSF database can contain various types data stored in Notes : access control list note icon note design note resource help-using document design note resource help-about document design note resource view note(s) design note form note design note agent note(s)page 1 design note data note(s) document collection(s) index(es)2.

5 Notes /Domino Summary and non-summaryIn the Notes /Domino terminology summary refers to 'summary information' about items (objects) in the NSF file, document summary information like author, creation date and time, etc. Non-summary refers to all other type of information formatted text, pictures, Data and non-dataTODO: data refers to content items, non-data to metadata NSF data NSF date and timeThe Domino and Notes TIMEDATE structure consists of 32-bit values that encode the time, the date, the time zone, and the Daylight Savings Time settings that were in effect when the structure was first 32-bit value contains the number of hundredths of seconds since midnight, Greenwich mean time.

6 If only the date is important, not the time, this field may be set to ALLDAY (0xffffffff or -1).The date and the time zone and Daylight Savings Time settings are encoded in the second 32-bit value. The 24 low-order bits contain the Julian Day, the number of days since January 1, 4713 BC. The Julian Day was originally devised as an aid to astronomers and is not the same as the Julian calendar. Since only days are counted, weeks, months, and years are ignored in calculations. The Julian Day is defined to begin at noon; for simplicity, Domino and Notes assume that the day begins at high-order byte, bits 31-24, encodes the time zone and Daylight Savings Time information. bit 31 (0x80000000) is set if Daylight Savings Time is observed; bit 30 (0x40000000) is set if the time zone is east of Greenwich mean time; bits 27-24 contain the number of hours difference between the time zone and Greenwich mean time; bits 29-28 contain the number of 15-minute intervals in the example, 2:49:04 P.

7 M., Eastern Standard Time, December 10, 1996 would be stored as:page 20x006 CDCC019 hours, 49 minutes, 4 seconds GMT0x852563 FCDST observed, zone +5, Julian Day 2,450,428If the time zone were set for Bombay, India, where Daylight Savings Time is not observed, 2:49:04 P. M., December 10, 1996 would be stored as:0x0032B8649 hours, 19 minutes, 4 seconds GMT0x652563 FCNo DST, zone 5 1/2 hours east of GMT, Julian Day 2,450, NSF file positionThe NSF file position is a 32-bit value that contains a file offset value divided by 256 (0x100). On-disk structure signaturesThe NSF format uses on-disk structure (ODS) signatures to mark the start and size of data Byte signatureThe byte signatures (BSIG) is 2 bytes of size and consists of:offsetsizevaluedescription01 Signature11 Structure Word signatureThe word signatures (WSIG) is 4 bytes of size and consists of:offsetsizevaluedescription01 Signature110xffMarker value22 Structure Long signatureThe long signatures (LSIG) is 4 bytes of size and consists of:offsetsizevaluedescription01 Signature110x00 Marker value24 Structure size4.

8 The file headerThe file header is 6 bytes of size and consists of:page 3offsetsizevaluedescription020x1a 0x00 Signature24 The database header sizeNotes/Domino considers the file header a long-signature (LSIG) of the database The database headerThe database header contains the following values: database information database identifier (DBID) flags replication information database information buffer title categories class design class (template name) special note identifiers padding database information 2 database information 3 database information 4 database information 5 padding database instance identifier (DBIID) replication history user activity log UNID database informationThe database information (DBINFO) is 178 bytes of size and consists of:offsetsizevaluedescription04 format versionon-disk structure (ODS) versionSee section: format version48 database identifier (DBID)Value consists an NSF date and time but is considered as an identifierSee section.

9 NSF date and timeAlso used as creation date and time?122 Application version144 Non-data Record Relocation Vector page 4offsetsizevaluedescription(RRV) bucket positionSee section: NSF file position184(Next) available non-data Record Relocation Vector (RRV) identifier222 Number of available non-data Record Relocation Vectors (RRVs)244 Activity log offset288 Bucket (last) modification date and timeValue consists of an NSF date and timeSee section: NSF date and timeAlso used as modification date and time?362 database classTODO382 database flagsSee section: database flags404 Bucket Descriptor Block (BDB) size444 Bucket Descriptor Block (BDB) positionSee section: NSF file position482 Bucket Descriptor Table (BDT) size504 Bucket Descriptor Table (BDT) position542 Bucket Descriptor Table (BDT) bitmaps564 Data Record Relocation Vector (RRV) bucket positionSee section.

10 NSF file position604 First data Record Relocation Vector (RRV) identifierProbably the first defined RRV not the first used644(Next) available data Record Relocation Vector (RRV) identifier682 Number of available data Record Relocation Vectors (RRVs)702 Record Relocation Vector (RRV) bucket size722 Summary bucket size742 Bitmap size762 Allocation granularity784 Extension granularity824 File sizeThe value contains 256 byte incrementsIf version >= 0x15 ?page 5offsetsizevaluedescription864 Number of file truncations904 Delivery sequence number944 Number of Bucket Descriptor Block (BDB) replacements984 Number of allocated Record Relocation Vectors (RRVs)1024 Number of de-allocations1064 Number of non-bucket allocations1104 Number of bucket allocations1148 Folder (last) modification date and timeValue consists of an NSF date and timeSee section: NSF date and time1224 Data note identifier table position1264 Data note identifier table size1348 Data (last) modification date and timeValue consists of an NSF date and timeSee section: NSF date and time1428 Next purge date and timeValue consists of an NSF date and timeSee section.