Supermicro Optane Persistent Memory 200 Series, User Manual

Page 1: ...User s Guide Revision 1 00 Intel Optane Persistent Memory 200 Series Configuration for Supermicro X12SPx X12Dxx X12Qxx Motherboards ...

Page 2: ...in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the manufacturer s instruction manual may cause harmful interference with radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case you will be required to correct the interference at you...

Page 3: ...eries modules in a Supermicro computer system based on an X12SPx X12Dxx X12Qxx motherboard Chapter 3 provides detailed instructions on how to configure Intel Persistent Memory 200 Series settings using the BIOS utility Chapter 4 provides Intel Persistent Memory 200 Series configuration instructions using ipmctl and ndctl Read this chapter when you want to configure the PMem settings using the open...

Page 4: ... 28 5215 ML s Hertogenbosch The Netherlands Tel 31 0 73 6400390 Fax 31 0 73 6416525 Email sales supermicro nl General Information support supermicro nl Technical Support rma supermicro nl Customer Support Website www supermicro nl Asia Pacific Address Super Micro Computer Inc 3F No 150 Jian 1st Rd Zhonghe Dist New Taipei City 235 Taiwan R O C Tel 886 2 8226 3990 Fax 886 2 8226 3992 Email support s...

Page 5: ...Installation Sequence 12 2 3 PMem Population for the X12SPx Motherboards 14 2 4 PMem Population for the X12Dxx Motherboards 15 2 5 PMem Population for the X12Qxx Motherboards 20 Chapter 3 Configuring Intel PMem 200 Series Using BIOS 3 1 To Enter the BIOS Setup Utility 23 3 2 To Configure PMem as App Direct AD Mode 25 3 3 To Configure PMem as Memory Mode MM 39 Chapter 4 Configuring Intel PMem 200 S...

Page 6: ...ntel PMem 200 Series supports two modes Memory Mode for volatile use and App Direct Mode for non volatile access Refer to the details below for your Intel PMem 200 Series features and support Memory Mode Volatile Memory Optimized for volatile use Memory Mode provides a high capacity main memory solution with higher power efficiency at lower operational cost All installed DDR4 DIMMs which are hidde...

Page 7: ...ormance enhancement Label Label Storage Area LSA Each PMem module contains a label storage area LSA which includes configuration instructions that define namespaces In other words the LSA is similar to a central control room that stores all configuration metadata instructions used to define namespaces in the PMem modules installed in the system Namespace A namespace is a set of signs names used to...

Page 8: ...cy file systems that do not checksum metadata instructions or applications It can be used for small boot volume RAW Mode This mode is used for a memory disk that does not support Direct Access mode 1 4 Specifications Operating Speed PMem 200 Series operates at 3200 MHz DDR4 memory bus speed Capacity PMem 200 Series modules are offered in capacities of 128GB 256GB and 512GB Additional Specification...

Page 9: ...ry error checking and correction and it also prevents the system from issuing a performance penalty before a device fails Please note that virtual lockstep mode might be affected until a faulty DRAM module is spared Patrol scrubbing Patrol Scrubbing is a process that allows the CPU to correct correctable memory errors detected on a memory module and send the correction to the requestor the origina...

Page 10: ...7 6 SUSE Linux Enterprise Server 12 4 SUSE Linux Enterprise Server 15 VMware vSphere Hypervisor ESXi 6 7 U1 Ubuntu LTS 18 04 Processor Platform Support Intel PMem 200 Series is supported by the following processors 3nd Gen Intel Xeon Scalable 83xx 63xx 53xx 4314 series processors Motherboard Platform Support Intel PMem 200 Series supports Supermicro X12SPx X12Dxx X12Qxx motherboards based on the X...

Page 11: ...therboard by its edges only do not touch its components peripheral chips memory modules or gold contacts When handling chips or modules avoid touching their pins Put the motherboard and peripherals back into their antistatic bags when not in use For grounding purposes make sure that your computer chassis provides excellent conduc tivity between the power supply the case the mounting fasteners and ...

Page 12: ...pulate the black slots Using unbalanced memory topology such as populating two DIMMs in one channel while populating one DIMM in another channel on the same motherboard will result in reduced memory performance Persistent Memory Population Requirements For proper memory installation please pay close attention to the following instructions 1 Intel PMem 200 Series is supported by Supermicro X12SPx X...

Page 13: ...of the 3rd Generation Xeon Scalable Processor used in your system Restrictions on Memory Configuration Settings When configuring memory mirroring and memory sparing settings please note the following restrictions 1 Memory mirroring is only supported when Persistent Memory is configured in App Direct Mode 2 Memory sparing is not supported when PMem modules are installed in the system ...

Page 14: ...o 3200 4Rx4 2H N A 128GB 8Rx4 4H NA 256GB LRDIMM up to 3200 4Rx4 64GB 128GB LRDIMM 3DS up to 3200 4Rx4 2H N A N A 8Rx4 4H 128GB 256GB Legend for the table above DDR4 Type and Capacity DDR4 See Validation Matrix DDR4 DIMMs validated with PMem Capacity PMem Any Capacity Uniformly for all channels for a given configuration PMem 200 Series Population Table for X12SPx Motherboards Note PMem 200 Series ...

Page 15: ...nctionality is limited in non POR configurations and select commands will not be supported Validation Matrix DDR4 DIMMS with PMem 200 Series Modules DIMM Type Ranks Per DIMM Data Width Stack DIMM Capacity GB DRAM Density 8Gb 16Gb RDIMM up to 3200 1Rx8 N A N A 1Rx4 16GB 32GB 2Rx8 16GB 32GB 2Rx4 32GB 64GB RDIMM 3DS up to 3200 4Rx4 2H N A 128GB 8Rx4 4H NA 256GB LRDIMM up to 3200 4Rx4 64GB 128GB LRDIM...

Page 16: ...mory DRAM FM Far Memory PMem Matrix targets configs for optimized PMem to DRAM cache ratio in MM mode For each individual population different PMem rearrangements among channels are permitted so long as the configuration doesn t break X12Dxx Memory population rules Ensure the same DDR4 DIMM type and capacity are used for each DDR4 PMem population If the system detects an unvalidated configuration ...

Page 17: ... targets configs for optimized PMem to DRAM cache ratio in MM mode For each individual population different PMem rearrangements among channels are permitted so long as the configuration doesn t break X12Dxx Memory population rules Ensure the same DDR4 DIMM type and capacity are used for each DDR4 PMem population If the system detects an unvalidated configuration then the system issues a BIOS warni...

Page 18: ...MMH2 P1 DIMMG1 P1 DIMMG2 P1 DIMMC2 P1 DIMMC1 P1 DIMMD2 P1 DIMMD1 P1 DIMMA2 P1 DIMMA1 P1 DIMMB2 P1 DIMMB1 4 4 AD MM One x4 PMem DDR4 PMem DDR4 DDR4 PMem DDR4 PMem One x4 DDR4 PMem DDR4 PMem PMem DDR4 PMem DDR4 6 1 AD One x1 DDR4 DDR4 DDR4 DDR4 PMem DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 PMem DDR4 DDR4 PMem DDR4 DDR4 DDR4 DDR4 PMem DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 DDR4 PMem DDR4 DDR4 DDR4 DD...

Page 19: ...s DIMM Type Ranks Per DIMM Data Width Stack DIMM Capacity GB DRAM Density 8Gb 16Gb RDIMM up to 3200 1Rx8 N A N A 1Rx4 16GB 32GB 2Rx8 16GB 32GB 2Rx4 32GB 64GB RDIMM 3DS up to 3200 4Rx4 2H N A 128GB 8Rx4 4H NA 256GB LRDIMM up to 3200 4Rx4 64GB 128GB LRDIMM 3DS up to 3200 4Rx4 2H N A N A 8Rx4 4H 128GB 256GB ...

Page 20: ...nnel Configuration 2 2 2 Modes CPU1 P1 DIMMF1 P1 DIMMF2 P1 DIMME1 P1 DIMME2 P1 DIMMD1 P1 DIMMD2 P1 DIMMA2 P1 DIMMA1 P1 DIMMB2 P1 DIMMB1 P1 DIMMC2 P1 DIMMC1 AD DRAM1 PMem DRAM1 PMem DRAM1 PMem PMem DRAM1 PMem DRAM1 PMem DRAM1 CPU2 P2 DIMMF1 P2 DIMMF2 P2 DIMME1 P2 DIMME2 P2 DIMMD1 P2 DIMMD2 P2 DIMMA2 P2 DIMMA1 P2 DIMMB2 P2 DIMMB1 P2 DIMMC2 P2 DIMMC1 AD DRAM1 PMem DRAM1 PMem DRAM1 PMem PMem DRAM1 PMe...

Page 21: ...related damages done to your memory modules or components 3 Align the key of the DIMM module with the receptive point on the memory slot Key 4 Align the notches on both ends of the module against the receptive points on the ends of the slot 5 Push both ends of the module straight down into the slot until the module snaps into place 6 Press the release tabs to the lock positions to secure the DIMM ...

Page 22: ...le below For more information on memory population please refer to the Memory Configuration User Guide for the X12 Motherboards posted on our wetsite at https www supermicro com support manuals Only App Direct Mode and Memory Mode are supported for the X12 motherboards Note Please make sure that all x8 DDR4 DIMMs and x8 PMem DCPMM DIMMs are detected prior to PMem 200 Series configuration as shown ...

Page 23: ...ow the instructions below Press the del key continuously during system boot to enter the BIOS setup utility After the system enters the BIOS setup utility use the arrow keys to select the Advanced tab on the top of the menu bar and press Enter to select it Use the down arrow key to select the Intel R Optane TM Persistent Memory Configu ration submenu the following screen will display ...

Page 24: ...es Configuration User s Guide After selecting the Intel R Optane TM Persistent Memory Configuration submenu press Enter and the screen below will display showing that 8 PMem modules are de tected on this motherboard A 2 Onboard PMem Modules Detected ...

Page 25: ...App Direct Mode Please complete the procedures below to properly configure Persistent Memory as App Direct Mode Select Platform as the default setting for the item Create goal config for This will set all PMem modules to App Direct Mode B 1 Provisioning Submenu B 2 Create Goal Configuration Submenu After selecting Provisioning press Enter and the Create goal config screen will display Press Enter ...

Page 26: ...Direct AD Mode Set the values of Reserved and Memory Mode to 0 Change PMem Memory type to App Direct as needed B 4 App Direct Mode Setup Screen Scroll down to the screen to select Create goal config again and press Enter to re confirm your Created Goal B 5 Re Confirm the Created Goal Screen ...

Page 27: ...hapter 3 Configuring Intel PMem 200 Series Settings Using BIOS The following screen will display to show all your Created Goals B 6 All the Created Goals Displayed B 7 All the Created Goals Displayed Screen ...

Page 28: ... your created goals press F4 to invoke the Save Exit Setup submenu and press Yes to save the changes B 8 Save the Configuration and Exit Reboot the system and enter the BIOS utility again to continue with PMem configuration Note Be sure to reboot your system for the changes you ve made to take effect ...

Page 29: ...steps below Select the Intel R Optane TM Persistent Memory Configuration submenu and press Enter B 9 PMem Configuration Continuation Select Regions and press Enter Please note that this feature is not applicable to Memory Mode B 10 Select Regions Submenu Current Regions information will display as shown below B 11 Current Regions Information Displayed ...

Page 30: ...D Mode configuration select Total Capacity and press Enter B 12 Total Capacity Entry Screen The total memory resource allocation across the host server submenu will be dis played as shown below Please note that this feature is only available on App Direct Mode B 13 Total Memory Resource Allocation ...

Page 31: ...s Enter to re confirm the current goal configuration B 14 Select Create Goal Config to Reconfirm The memory allocation goal configuration detail settings will display as shown below B 15 Memory Allocation Goal Configuration Screen To delete a goal configuration that you ve just created scroll down to select the Delete Goal Config submenu and press Enter B 16 Delete Goal Config Screen ...

Page 32: ...g for submenu will display as shown below Select the goal con figuration you want to delete and then press Enter B 17 Delete Goal Config For Screen B 18 Delete Goal Config Submenu Once you ve selected the goal configuration to delete select the Delete goal config sub menu and press Enter to delete it ...

Page 33: ...ted goal without first deleting it you will encounter an error message as illustrated below To immediately re create a new goal without deleting the existing goal that you ve just created and confirmed by selecting Create goal config submenu as shown below The following screen will displays to indicate that the BIOS utility will allow the user to re create a new goal at this point B 19 Re Creating...

Page 34: ...nfirmed a goal you will need to first delete this newly created goal before re creating another new goal by selecting the Delete goal config submenu B 21 An Error Message Displayed Asking to Delete the Current Goal First B 22 To Delete the Current Goal First Enter the Delete goal config submenu and press Enter to delete the existing goal config as shown below Once the existing goal configuration i...

Page 35: ...ow the steps below Follow the instructions given in Section 3 1 to enter the BIOS utility to access the Namespace submenu as shown below B 24 NameSpace Submenu Select Create namespace and press Enter to enter the Create namespace submenu Select Name and press Enter to invoke the Name field as shown below B 25 Create NameSpace Submenu B 26 Name Field Type a name in the Name field and press Enter as...

Page 36: ... reconfirm that the Name entered in the Create Namespace is correct as shown below B 28 To Confirm the Name Entered in the Namespace Change is Correct B 29 Check to Confirm the Namespace Change is Properly Created Once you ve confirmed that the Namespace change you ve made is correct Click on the Namespace chagne again to confirm that the Namespace chagne is properly created as shown below ...

Page 37: ... steps listed below Follow the instructions given in Sec 4 1 to access the Namespaces submenu as shown below B 30 Namespace Submenu Screen If a namespace has been previously created the information of the existing namespace will display as shown below B 31 Existing Namespace Information Displayed Enter the namespace you want to delete the screen below will display B 32 Enter the Namespace to Delet...

Page 38: ...taining to the namespace If you intend to delete the namespace and its related information select Yes and press Enter If you do not want to delete the namespace and its related information select No and press Enter B 34 To Confirm Namespace Deletion Once you select Yes and press Enter to confirm your desire to delete the namespace and its information the following screen will display indicating th...

Page 39: ... Mode please complete all steps listed in Section 3 1 B 35 No Namespaces in the System screen When the screen shown above displays select the Provisioning submenu as shown below C 1 Provisioning screen After you ve selected the Provisioning submenu and pressed Enter the following screen will display C 2 Create Goal Config Screen When the screen above displays select Create Goal Config submenu and ...

Page 40: ...ode and enter 100 as the default Save the setting and reboot the system for the changes you ve made to take effect Note To configure PMem as Memory Mode Memory Mode needs to be set to 100 C 3 Memory Mode Setup Scroll down to the screen to select Create goal config again and press Enter to re confirm your created goal C 4 To Confirm Goal Created ...

Page 41: ...the goal you ve created C 5 Memory Mode Setup Press F4 to invoke the Save Exit Setup submenu and press Yes to save the changes as shown below C 6 Save MM Config Setting Reboot the system and enter the BIOS utility again to continue with PMem configuration Note Be sure to reboot your system for the changes you ve made to take effect ...

Page 42: ...tel Optane Persistent Memory Configuraion Submenu To confirm the status of Memory Mode select the Total Capacity submenu which will display total memory resource allocation C 8 Total Capcity Submenu After selecting the Total Capacity submenu press Enter to access the Total memory resource allocation across the host server submenu as shown below It will also display that Memory Mode is currently us...

Page 43: ...43 Chapter 3 Configuring Intel PMem 200 Series Settings Using BIOS Note The X12 motherboards do not support PMem Mixed MX mode as shown in the screen below C 10 Screen Showing Mixed Mode Not Supported ...

Page 44: ...r download from GitHub It supports the following features Discovery Configuration Firmware management Security functionality management Health monitoring Performance tracking Debug and troubleshooting Introduction to ndctl Ndctl is an open source utility used for managing the Linux LIBNVDIMM kernel subsystem It is designed to work with various non volatile memory devices NVDIMMs from different ven...

Page 45: ...upport PMEM and BLK mode access Namespace Namespace defines a contiguously addressed range of non volatile memory which is similar in concept to a hard disk partition a SCSI Logical Unit LUN or an NVM Express namespace It is a persistent memory storage unit that cannot be used for input output Namespaces will appear as a device in ndctl dev Creating namespaces can be achieved by using ndctl Non vo...

Page 46: ...ion The full list of commands can be seen by running ipmctl help from the command line Show Topology Use the ipmctl sudo show topology command to display the PMem modules and DDR DIMMs discovered in the system by enumerating the SMBIOS Type 17 tables A 1 Show Topology CentOS A 2 Show Topology Windows ...

Page 47: ...lso displays DIMM IDs capacities health state and firmware version Show Provisioned Capacity Use the ipmctl show memoryresources command to display the provisioned capacity under different PMem mode configurations If memory capacity is displayed as 0 GiB it indicates that the current mode is set to App Direct otherwise it is set to Memory Mode B 1 Show DIMM Information CentOS B 2 Show DIMM Informa...

Page 48: ...en reboot Memory Mode Configuration Use the ipmctl create goal MemoryMode n command to provision any percentage of PMem capacity on all sockets where n represents the percentage number of capacity to be provisioned in Memory Mode A reboot is required to process new memory allocation goals D 1 Memory Mode Configuration CentOS D 2 Memory Mode Configuration Windows Note Mixed Memory MX Mode is not su...

Page 49: ...ept to JBOD in traditional storage PMem interleaving increases the throughput of reads and writes to Persistent Memory Use the ipmctl create goal PersistentMemoryType AppDirect or the default ipmctl create goal command to set a goal that creates an interleaved region across all the PMem modules discovered in the system The two commands are equivalent in this action To create a goal that creates no...

Page 50: ... save the current configuration to a file Show Current Goal Use the ipmctl show goal command to display the goal that is currently in place Delete Goal A goal will only be executed after a system reboot Use the ipmctl delete goal command to clear the goal that is currently in place G 2 Create a Goal from a Configuration File Windows G 1 Create a Goal from a Configuration File CentOS H 1 Save Curre...

Page 51: ... Change Use the ipmctl show memoryresources command to confirm if the mode has been changed successfully after a system reboot If the mode is changed from Memory Mode to App Direct Mode a single region per socket will be created upon system reboot No regions will be created when the mode is changed from App Direct Mode to Memory Mode Use the show region command to display the regions that were cre...

Page 52: ...52 Super Intel Optane Persistent Memory 200 Series Configuration User s Guide K 1 Confirm Mode Change CentOS L 1 Display Region CentOS K 2 Confirm Mode Change Windows L 2 Display Region Windows ...

Page 53: ... the namespace of the user s choice X represents the namespace on the active namespaces list Note It is imperative to stop applications and unmount used namespaces before disabling the namespace Create Namespaces with ndctl Follow the instructions below to properly create namespaces using the ndctl commands ndctl create namespace mode fsdax sector Example ndctl create namespace mode fsdax Repeat t...

Page 54: ...represents the namespace on the active namespaces list List Active Regions Use ndctl list R to display the active regions list Delete Region Use the ndctl disable region X command to delete the region of the user s choice X represents the region on the active regions list N 1 List Active Namespaces CentOS O 1 Disable Namepace namespace0 0 CentOS O 2 Destroy Namepace namespace0 0 CentOS ...