US Pat. No. 9,684,575

FAILOVER HANDLING IN MODULAR SWITCHED FABRIC FOR DATA STORAGE SYSTEMS

Liqid Inc., Lafayette, C...

1. A data storage system, comprising:
a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured
to store data and retrieve the data stored on associated storage media responsive to storage operations;

a plurality of processing modules each comprising one or more processors, where a first processor is configured to receive
at least a portion of the storage operations and transfer the portion of the storage operations for delivery to associated
ones of the plurality of storage drives over a PCIe interface;

a control processor configured to initialize the plurality of storage drives for use on the PCIe interfaces by the processors
on the processing modules, assign each of the plurality of storage drives an associated unique identifier using at least a
sideband communication link distinct from the PCIe interfaces, wherein the sideband communication link communicatively couples
the control processor and the ones of the plurality of storage drives, and report the associated unique identifiers to the
processing modules over at least an associated interface; and

a monitoring system configured to identify when the first processor fails, and responsively instruct a second processor to
handle further storage operations directed to storage drives associated with the first processor.

US Pat. No. 10,180,924

PEER-TO-PEER COMMUNICATION FOR GRAPHICS PROCESSING UNITS

Liqid Inc., Broomfield, ...

19. A data processing apparatus comprising:one or more computer readable storage media;
a processing system operatively coupled with the one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media, that when executed by the processing system, direct the processing system to at least:
establish a peer-to-peer arrangement between graphics processing units (GPUs) over a Peripheral Component Interconnect Express (PCIe) fabric by at least providing an isolation function in the PCIe fabric configured to isolate a device PCIe address domain associated with the GPUs from at least a local PCIe address domain associated with a host processor that initiates the peer-to-peer arrangement between the GPUs;
wherein the isolation function comprises synthetic PCIe devices representing the GPUs in the local PCIe address domain;
wherein the isolation function is configured to redirect traffic transferred by the host processor for the GPUs in the local PCIe address domain for delivery to corresponding ones of the GPUs in the device PCIe address domain; and
wherein the isolation function is further configured to redirect peer-to-peer traffic transferred by a first of the GPUs indicating the second of the GPUs as a destination in the local PCIe address domain to the second of the GPUs in the device PCIe address domain.

US Pat. No. 10,037,296

POWER HANDLING IN A SCALABLE STORAGE SYSTEM

Liqid Inc., Broomfield, ...

1. A data storage assembly, comprising:a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) host interface, solid state storage media, and holdup power storage elements, with each of the plurality of storage drives configured to store and retrieve data responsive to storage operations received over the associated PCIe host interface;
a PCIe switch circuit coupled to the PCIe host interfaces of the plurality of storage drives and configured to receive the storage operations issued by a plurality of host systems over a shared PCIe interface and transfer the storage operations for delivery to the plurality of storage drives over selected ones of the PCIe host interfaces; and
power control circuitry configured to redistribute holdup power of the holdup power storage elements among ones the plurality of storage drives, the PCIe switch circuit, and the plurality of host systems after input power is lost to the data storage assembly.

US Pat. No. 9,285,827

SYSTEMS AND METHODS FOR OPTIMIZING DATA STORAGE AMONG A PLURALITY OF SOLID STATE MEMORY SUBSYSTEMS

Liqid Inc., Lafayette, C...

1. A method of operating a data storage device, the method comprising:
monitoring a utilization level of at least a processing system of the data storage device;
determining a quantity of links to aggregate for an external communication interface of the data storage device based at least
on the utilization level of at least the processing system;

providing an indication of the quantity of links to aggregate for the external communication interface to an interface system
of the data storage device;

providing an indication of the utilization level to a control circuit of the data storage device; and
throttling a clock frequency associated with at least the processing system of the data storage device based at least on the
indication of the utilization level.

US Pat. No. 10,206,297

MESHED ARCHITECTURE RACKMOUNT STORAGE ASSEMBLY

Liqid Inc., Broomfield, ...

1. A rackmount assembly, comprising:one or more storage modules each comprising at least one storage drive;
one or more controller modules each comprising at least one processing system;
one or more fabric modules each comprising at least one cluster cross-connect Peripheral Component Interconnect Express (PCIe) switch configured to couple to at least one further rackmount assembly;
first interconnect PCIe links that couple each of the one or more storage modules to each of the one or more controller modules and each of the one or more fabric modules;
second interconnect PCIe links that couple each of the one or more controller modules to each of the one or more fabric modules;
third interconnect PCIe links that couple the one or more controller modules to each other;
the one or more fabric modules each configured to receive storage operations directed to ones of the storage modules over at least one external interface of the rackmount assembly and transfer the storage operations for handling by ones of the controller modules.

US Pat. No. 10,180,889

NETWORK FAILOVER HANDLING IN MODULAR SWITCHED FABRIC BASED DATA STORAGE SYSTEMS

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a plurality of storage drives each configured to store data and retrieve the data stored on associated storage media responsive to storage operations;
a first processor configured to establish a network connection with an external system, receive first storage operations transferred by the external system over the network connection, and handle the first storage operations related to storage and retrieval of data on at least one of the plurality of storage drives assigned to a first address space portion of a Peripheral Component Interconnect Express (PCIe) address space shared among the first processor and at least a second processor;
after establishing the network connection with the external system, the first processor configured to transfer information describing the network connection comprising at least a network address and a network port for delivery to at least the second processor; and
the second processor configured to identify when the first processor has failed, responsively inherit the network connection with the external system based at least on the information describing the network connection and packet sequence information received from the external system to resume packet acknowledge/sequence counts established between the first processor and the external system, and handle second storage operations transferred by the external system over the network connection, wherein the second storage operations are related to storage and retrieval of data on at least the one of the plurality of storage drives assigned to the first address space portion of the PCIe address space.

US Pat. No. 9,501,110

ADJUSTABLE DATA STORAGE DRIVE MODULE CARRIER ASSEMBLY

Liqid Inc., Lafayette, C...

1. A data storage sled, comprising:
a circuit card assembly comprising a plurality of connectors that couple to drive modules and a host connector for coupling
the data storage sled to an external connector;

an enclosure comprising hinged covers each configured to cover a plurality of apertures in the enclosure through which individual
ones of the drive modules can be inserted into an associated connector on the circuit card assembly;

drive module mounting assembles configured to hold the individual ones of the drive modules into the associated connector
by at least including movable mounting features to accommodate varying lengths among the individual ones of the drive modules.

US Pat. No. 9,653,124

DUAL-SIDED RACKMOUNT STORAGE ASSEMBLY

Liqid Inc., Lafayette, C...

1. A data storage assembly, comprising: a midplane assembly comprising at least two circuit boards and configured to: electrically
couple via a first circuit board on a first side of the midplane assembly to a plurality of storage modules; electrically
couple via the first circuit board on the first side of the midplane assembly to a plurality of compute modules, wherein first
ones of the plurality of communication modules comprise external communication interfacing modules and second ones of the
plurality of communication modules comprise cross-coupling modules which span between the first circuit board and a second
circuit board; electrically couple via the second circuit board on a second side of the midplane assembly opposite the first
side to a plurality of communication modules; electrically interconnect ones of the plurality of storage modules, the plurality
of compute modules, and the plurality of communication modules; receive power from one or more power supply modules and distribute
the power to the plurality of storage modules, the plurality of compute modules, and the plurality of communication modules;
one or more fan modules mounted between the first circuit board and the second circuit board, the one or more fan modules
configured to: provide airflow to the data storage assembly through at least a plurality of apertures in the first circuit
board and the second circuit board; and a chassis configured to: mechanically house and structurally support each of the plurality
of storage modules, the plurality of compute modules, the plurality of communication modules, and the one or more power supply
modules when coupled to the midplane assembly to form the data storage assembly and allow installation of the data storage
assembly into a rackmount environment.

US Pat. No. 10,019,388

ENHANCED INITIALIZATION FOR DATA STORAGE ASSEMBLIES

Liqid Inc., Broomfield, ...

1. A data storage platform, comprising:a plurality of data storage assemblies each comprising one or more storage drives that service data storage operations over associated storage interfaces;
a control processor coupled to ones of the data storage assemblies over at least two types of sideband communication interfaces different than the storage interfaces of the storage drives;
during an initialization process for the one or more storage drives, the control processor configured to transfer initialization data to each of the data storage assemblies over a first type of sideband communication interface and transfer further initialization data to at least one of the data storage assemblies over a second type of sideband communication interface when the at least one of the data storage assemblies does not respond to the initialization data over the first type of sideband communication interface.

US Pat. No. 9,798,636

FRONT END TRAFFIC HANDLING IN MODULAR SWITCHED FABRIC BASED DATA STORAGE SYSTEMS

Liqid Inc., Lafayette, C...

1. A data storage system, comprising:
a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured
to store data and retrieve the data stored on associated storage media responsive to storage operations received over a switched
PCIe fabric;

a plurality of processors configured to each manage only an associated subset of the plurality of storage drives over the
switched PCIe fabric, wherein each of the processors is associated with a network interface configured to receive data packets
related to the storage operations for any of the plurality of storage drives;

a first processor configured to identify first data packets received over a network interface associated with the first processor
within a layer-2 network buffer of the first processor as comprising a storage operation associated with at least one of the
plurality of storage drives managed by a second processor, and responsively transfer the first data packets into a layer-2
network buffer of the second processor; and

the second processor configured to identify the first data packets within the layer-2 network buffer of the second processor
and handle the storage operation associated with the first data packets as if a network interface associated with the second
processor had received the first data packets by at least processing the first data packets up a network stack of the second
processor for handling of the storage operation associated with the first data packets.

US Pat. No. 10,001,819

DUAL-SIDED RACKMOUNT MODULAR DATA PROCESSING ASSEMBLY

Liqid Inc., Broomfield, ...

1. A data processing assembly, comprising: a midplane assembly comprising at least two circuit boards and configured to: electrically couple via a first circuit board on a first side of the midplane assembly to a plurality of storage modules; electrically couple via the first circuit board on the first side of the midplane assembly to a plurality of graphics processing modules; electrically couple via the first circuit board on the first side of the midplane assembly to a plurality of compute modules; electrically couple via a second circuit board on a second side of the midplane assembly opposite the first side to a plurality of communication modules, wherein first ones of the plurality of communication modules comprise external communication interfacing modules and second ones of the plurality of communication modules comprise cross-coupling modules which span between the first circuit board and the second circuit board; electrically interconnect ones of the plurality of storage modules, the plurality of compute modules, the plurality of graphics processing modules, and the plurality of communication modules; receive power from one or more power supply modules and distribute the power to the plurality of storage modules, the plurality of compute modules, the plurality of graphics processing modules, and the plurality of communication modules; one or more fan modules mounted between the first circuit board and the second circuit board, the one or more fan modules configured to: provide airflow to the data processing assembly through at least a plurality of apertures in the first circuit board and the second circuit board; and a chassis configured to: mechanically house and structurally support each of the plurality of storage modules, the plurality of compute modules, the plurality of graphics processing modules, the plurality of communication modules, and the one or more power supply modules when coupled to the midplane assembly to form the data processing assembly and allow installation of the data processing assembly into a rackmount environment.

US Pat. No. 10,198,183

TUNNELING OF STORAGE OPERATIONS BETWEEN STORAGE NODES

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a first processing module configured to receive over a first network interface one or more packets indicating the first network interface as a network destination and comprising a storage operation, inspect at an Ethernet network layer of the first network interface at least a storage address included in a header of the one or more packets to determine that the storage operation indicates a storage drive managed by a second processing module, and establish one or more further packets comprising the storage operation and having an Ethernet header modified from the one or more packets to indicate a layer-2 network address of a network interface associated with the second processing module, and transfer the one or more further packets for delivery to the network interface associated with the second processing module over a tunneled network connection; and
the second processing module configured to detect the one or more further packets comprising the storage operation received over the tunneled network connection, inspect at least the storage address included in a header of the one or more further packets to determine that the storage operation indicates the storage drive managed by the second processing module, and handle the storage operation with the storage drive.

US Pat. No. 9,645,902

MODULAR SWITCHED FABRIC FOR DATA STORAGE SYSTEMS

Liqid Inc., Lafayette, C...

1. A data storage system, comprising:
a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured
to store data and retrieve the data stored on associated storage media responsive to storage operations;

one or more processing modules comprising one or more processors communicatively coupled to the plurality of storage drives
over a PCIe fabric comprised of one or more PCIe switches;

at least one of the PCIe switches configured to communicatively couple at least one of the processors of the data storage
system with an external system over at least one external PCIe link;

the processors configured to share a PCIe address space associated with the PCIe fabric for transfer of the storage operations
to appropriate ones of the processors that manage ones of the plurality of storage drives;

a first of the processors configured to receive a first storage operation and identify a portion of the PCIe address space
associated with the first storage operation; and

based at least on the portion of the PCIe address space being associated with a second of the processors, the first of the
processors configured to transfer the first storage operation for delivery to the second of the processors over the PCIe fabric
for handling of the storage operation with ones of the plurality of storage drives assigned to the second of the processors.

US Pat. No. 10,108,422

MULTI-THREAD NETWORK STACK BUFFERING OF DATA FRAMES

Liqid Inc., Broomfield, ...

1. A networked data processing system, comprising:a processing system configured to execute an operating system that comprises a network module for handling data frames received over one or more network interfaces of the networked data processing system, the data frames directed to one or more applications;
the network module configured to establish a plurality of data buffers individually associated with application threads of the one or more applications, wherein the application threads are each associated with a messaged signaling interrupt (MSI);
the network module configured to store associated ones of the data frames for the application threads in the data buffers as the data frames are processed through a network stack of the networked data processing system; and
the network module configured to maintain data exclusivity locks for the plurality of data buffers and individually associate the data exclusivity locks with the application threads.

US Pat. No. 9,958,913

DATA STORAGE DRIVE MODULE CARRIER ASSEMBLY

Liqid Inc., Broomfield, ...

1. A data storage sled, comprising:a circuit card assembly comprising a plurality of connectors for coupling to drive modules and a host connector for coupling the data storage sled to an external connector;
an enclosure comprising apertures in the enclosure through which individual ones of the drive modules can be inserted into an associated connector on the circuit card assembly;
drive module mounting assembles configured to hold the individual ones of the drive modules into the associated connector by at least including movable mounting features to accommodate varying lengths among the individual ones of the drive modules.

US Pat. No. 10,114,784

STATISTICAL POWER HANDLING IN A SCALABLE STORAGE SYSTEM

Liqid Inc., Broomfield, ...

1. A data storage assembly, comprising:a plurality of storage drives each comprising a drive Peripheral Component Interconnect Express (PCIe) interface and solid state storage media, with each of the plurality of storage drives configured to store and retrieve data responsive to storage operations received over the associated drive PCIe interface;
a PCIe switch circuit coupled to the drive PCIe interfaces of the plurality of storage drives and configured to receive the storage operations issued by one or more host systems over a shared PCIe interface and transfer the storage operations for delivery to the plurality of storage drives over selected ones of the drive PCIe interfaces;
a control processor configured to monitor activity levels of the drive PCIe interfaces of the plurality of storage drives;
the control processor configured to alter properties of the drive PCIe interfaces for one or more of the plurality of storage drives based at least on the activity levels of the one or more of the plurality of storage drives, wherein the properties of the drive PCIe interfaces comprise one or more of a quantity of active PCIe lanes and a PCIe throughput; and
the control processor configured to instruct power control circuitry to provide power to the one or more of the plurality of storage drives based at least on activity levels of the drive PCIe interfaces of the one or more of the plurality of storage drives being above a threshold activity level, and instruct the power control circuitry to remove the power from the one or more of the plurality of storage drives based at least on the activity levels indicating the one or more of the plurality of storage drives are dormant.

US Pat. No. 10,255,215

ENHANCED PCIE STORAGE DEVICE FORM FACTORS

Liqid Inc., Broomfield, ...

1. A storage drive, comprising:a chassis that structurally supports elements of the storage drive;
a plurality of M.2 storage devices configured to store and retrieve data responsive to storage operations received over associated Peripheral Component Interconnect Express (PCIe) interfaces;
a PCIe switch circuit communicatively coupled to the PCIe interfaces of the plurality of M.2 storage devices and configured to receive the storage operations over a host input issued by a host system and transfer the storage operations for delivery to ones of the plurality of M.2 storage devices over associated ones of the PCIe interfaces;
the host input comprising at least one host PCIe interface to communicatively couple the storage drive to a host system and receive input power;
a first circuit board assembly comprising a first circuit card connector and at least three M.2 storage device connectors that couple to corresponding ones of the plurality of M.2 storage devices, wherein at least the first circuit card connector is configured to carry signaling and power related to the at least three M.2 storage device connectors;
a second circuit board assembly comprising a second circuit card connector and at least one further M.2 storage device connector that couples to at least a further one of the plurality of M.2 storage devices, wherein at least the second circuit card connector is configured to mate with the first circuit card connector; and
holdup circuitry configured to provide holdup power to at least the plurality of M.2 storage devices after the input power is lost to the storage drive by providing at least a first portion of the holdup power for delivery to the at least one further M.2 storage device on the second circuit board assembly and at least a second portion of the holdup power for delivery to the at least three M.2 storage device connectors on the first circuit board assembly.

US Pat. No. 10,191,667

SYSTEMS AND METHODS FOR OPTIMIZING DATA STORAGE AMONG A PLURALITY OF STORAGE DRIVES

Liqid Inc., Broomfield, ...

1. A data storage device, comprising:an interface system configured to communicate over an aggregated host interface to receive one or more write operations for storage of data by the data storage device, wherein the aggregated host interface comprises a selected quantity of Peripheral Component Interconnect Express (PCIe) interface lanes combined to form the aggregated host interface;
a plurality solid state drives (SSDs) each comprising an associated drive PCIe interface coupled to the interface system; and
the interface system configured to process the one or more write operations against storage allocation information to parallelize the data into individual portions for storage among more than one of the SSDs and transfer corresponding portions of the parallelized data to associated SSDs over corresponding drive PCIe interfaces.

US Pat. No. 10,191,691

FRONT-END QUALITY OF SERVICE DIFFERENTIATION IN STORAGE SYSTEM OPERATIONS

Liqid Inc., Broomfield, ...

1. A data storage platform, comprising:one or more data storage modules each comprising storage drives coupled over a Peripheral Component Interconnect Express (PCIe) fabric with at least one processing module that receives storage operations directed to the one or more data storage modules over one or more network interfaces;
the processing module configured to assign service levels in a queue to the storage operations that are received over the one or more network interfaces;
the processing module configured to pre-allocate resources in the queue for selected ones of the service levels before ones of the storage operations associated with the selected ones of the service levels are received by the processing module; and
based at least on the service levels, the processing module configured to service the storage operations from the queue with the one or more data storage modules over the PCIe fabric.

US Pat. No. 10,474,608

STACKED-DEVICE PERIPHERAL STORAGE CARD

Liqid Inc., Broomfield, ...

19. A half-height, half-length (HHHL) Peripheral Component Interconnect Express (PCIe) storage card insertable into a PCIe slot of a host system, comprising:a circuit board;
at least four storage devices comprising at least two upper storage devices having a first distance with respect to a surface of the circuit board and stacked above at least two lower storage devices having a second distance with respect to the surface of the circuit board, each of the at least four storage devices comprising an associated device PCIe interface and solid state storage media, and configured to store and retrieve data responsive to storage operations received over the associated device PCIe interface;
a PCIe switch circuit communicatively coupled to the device PCIe interfaces and configured to receive the storage operations issued by the host system over a host PCIe connector of the PCIe storage card, and transfer the storage operations for delivery to selected ones of the at least four storage devices over the associated device PCIe interfaces; and
holdup circuitry configured to provide power to the at least four storage devices after input power is lost to the PCIe storage card.

US Pat. No. 10,423,547

INITIALIZATION OF MODULAR DATA STORAGE ASSEMBLIES

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a control system communicatively coupled to at least a storage drive over sideband communication interfaces different than a storage interface of the storage drive used to service data storage operations;
during an initialization process for the storage drive, the control system configured to transfer initialization data to the storage drive over a first type of sideband communication interface and transfer further initialization data to at the storage drive over a second type of sideband communication interface responsive to the storage drive not responding to the initialization data over the first type of sideband communication interface.

US Pat. No. 10,467,166

STACKED-DEVICE PERIPHERAL STORAGE CARD

Liqid Inc., Broomfield, ...

19. A half-height, half-length (HHHL) Peripheral Component Interconnect Express (PCIe) storage card insertable into a connector of a host system, comprising:a circuit board;
at least four M.2 storage devices comprising at least two upper M.2 storage devices having a first distance with respect to a surface of the circuit board and stacked above at least two lower M.2 storage devices having a second distance with respect to the surface of the circuit board, each of the at least four M.2 storage devices comprising a PCIe interface and solid state storage media, and configured to store and retrieve data responsive to storage operations received over the associated PCIe interface;
a PCIe switch circuit communicatively coupled to the PCIe interfaces of the four M.2 storage devices and configured to receive the storage operations issued by the host system over a PCIe connector of the PCIe storage card and transfer the storage operations for delivery to ones of the four M.2 storage devices over associated PCIe interfaces; and
holdup circuitry configured to provide power to at least the four M.2 storage devices after input power is lost to the PCIe storage card.

US Pat. No. 10,402,363

MULTI-PORT INTERPOSER ARCHITECTURES IN DATA STORAGE SYSTEMS

Liqid Inc., Broomfield, ...

1. A storage interposer module, comprising:a host connector configured to couple to one or more host systems over associated host Peripheral Component Interconnect Express (PCIe) interfaces;
PCIe switch circuitry configured to receive storage operations over the host connector that are issued by the one or more host systems and transfer the storage operations for storage by one or more storage devices over drive PCIe interfaces;
the PCIe switch circuitry configured to provide an address trap for the storage operations based at least on an address range associated with the storage operations, wherein when ones of the storage operations correspond to the address trap, the PCIe switch circuitry indicates ones of the storage operations to the interposer control module over a control interface;
an interposer control module configured to monitor the storage operations in the PCIe switch circuitry via the control interface coupled between the PCIe switch circuitry and the interposer control module, and selectively direct delivery of each of the storage operations to corresponding storage areas among the one or more storage devices based at least on information monitored about the storage operations in the PCIe switch circuitry; and
the interposer control module configured to provide host-to-host communication based at least in part on the ones of the storage operations directed to the interposer control module using the address trap.

US Pat. No. 10,416,731

DATA STORAGE DRIVE MODULE CARRIER ASSEMBLY

Liqid Inc., Broomfield, ...

1. A data storage sled, comprising:a circuit card assembly comprising a plurality of connectors for coupling to drive modules and a host connector for coupling the data storage sled to an external connector;
drive module mounting assembles configured to hold the individual ones of the drive modules into the associated connector by at least including adjustable mounting features to accommodate lengths of the individual ones of the drive modules.

US Pat. No. 10,402,197

KERNEL THREAD NETWORK STACK BUFFERING

Liqid Inc., Broomfield, ...

1. A data system, comprising:a processing system configured to execute an operating system that comprises a network module for handling data frames directed to a plurality of kernel threads and received over one or more network interfaces of the data system;
the network module configured to establish a plurality of data buffers individually associated with the kernel threads;
the network module configured to store associated ones of the data frames for the kernel threads in the data buffers as the data frames are processed through a network stack of the data system; and
the network module configured to maintain data exclusivity locks for the plurality of data buffers and individually associate the data exclusivity locks with the kernel threads.

US Pat. No. 10,223,315

FRONT END TRAFFIC HANDLING IN MODULAR SWITCHED FABRIC BASED DATA STORAGE SYSTEMS

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a plurality of processors configured to each manage an associated subset of a plurality of storage devices coupled over a Peripheral Component Interconnect Express (PCIe) fabric;
at least one processor configured to:
receive data packets comprising storage operations over a network interface associated with the at least one processor;
identify when ones of the storage operations are directed to at least one of the plurality of storage devices managed by a target processor; and
transfer over the PCIe fabric associated data packets comprising the ones of the storage operations into a network buffer of the target processor for handling of the ones of the storage operations by the target processor.

US Pat. No. 10,496,504

FAILOVER HANDLING IN MODULAR SWITCHED FABRIC FOR DATA STORAGE SYSTEMS

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a plurality of network interfaces each configured to receive storage operations issued by one or more host systems for handling by the data storage system, and transfer the storage operations over a Peripheral Component Interconnect Express (PCIe) fabric;
a plurality of processors configured to receive the storage operations over the PCIe fabric, process properties of the storage operations against allocation information of a shared PCIe address space to determine target processors to handle the storage operations, and selectively transfer ones of the storage operations to the target processors for handling of the storage operations with associated storage drives coupled over the PCIe fabric, wherein the shared PCIe address space is shared among at least the plurality of processors on the PCIe fabric; and
a monitoring system configured to identify failed ones of the processors and responsively initiate backup ones of the processors to handle portions of the shared PCIe address space handled by the failed ones of the processors.

US Pat. No. 10,540,185

SOFTWARE DEPLOYMENT IN DISAGGREGATED COMPUTING PLATFORMS

Liqid Inc., Broomfield, ...

19. A computing apparatus comprising:a management processor configured to:
receive user commands to establish a compute unit among a plurality of physical computing components coupled by a Peripheral Component Interconnect Express (PCIe) fabric, the plurality of physical computing components including one or more central processing units (CPUs), one or more storage modules and one or more PCIe switches, wherein the user command identifies an operating system of a plurality of operating systems stored in a storage device connected to the management processor; and
establish the compute unit by:
instructing the PCI fabric to establish a first PCIe partition within the PCIe fabric including the management processor and a storage module of the one or more storage modules, the first PCIe partition within the PCIe fabric configured such that the management processor has visibility over the PCIe fabric to the storage module;
copying at least the identified operating system to the storage module using the first PCIe partition;
instructing the PCIe fabric to de-establish the first PCIe partition including the management processor and the storage module; and
instructing the PCIe fabric to establish a second PCIe partition within the PCIe fabric including a CPU of the one or more CPUs and the storage module of the one or more storage modules, the second PCIe partition within the PCIe fabric configured such that the CPU has visibility over the PCIe fabric to the storage module such that the CPU will load the identified operating system from the storage module during a boot operation.

US Pat. No. 10,503,618

MODULAR SWITCHED FABRIC FOR DATA STORAGE SYSTEMS

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations;
a plurality of processors communicatively coupled to network interfaces and the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches, wherein the one or more plurality of processors are configured to share a PCIe address space associated with the PCIe fabric, with each of the plurality of processors allocated to at least a corresponding portion of the PCIe address space;
each of the network interfaces configured to be associated with a different Internet Protocol (IP) address, and at least two of the network interfaces configured to be associated with an Internet Small Computer System Interface (iSCSI) portal group, wherein the network interfaces are configured to receive the storage operations over associated network links, and transfer ones of the storage operations to associated ones of the plurality of processors over the PCIe fabric; and
a first processor associated with a first network interface and the iSCSI portal group configured to identify data packets received over the first network interface as comprising a first storage operation, process the first storage operation against allocation information related to the PCIe address space to determine the first storage operation is associated with at least one of the plurality of storage drives managed by a second processor, and responsively transfer the first data packets for handling of the first storage operation by the second processor as if the data packets were received by a second network interface associated with the second processor and the iSCSI portal group.

US Pat. No. 10,362,107

SYNCHRONIZATION OF STORAGE TRANSACTIONS IN CLUSTERED STORAGE SYSTEMS

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a plurality of processing systems configured to receive storage operations for a plurality of storage drives coupled over a communication fabric, each of the plurality of storage drives configured to store data and retrieve the data stored on associated storage media responsive to the storage operations;
a control processor configured to establish a timer synchronization signal over one or more discrete links separate from the communication fabric, and distribute the timer synchronization signal to the plurality of processing systems; and
each of the plurality of processing systems configured to:
maintain an associated local counter;
reset the associated local counter to an initial value responsive to assertion of the timer synchronization signal provided by the control processor;
add timestamps indicating current values of the associated local counter into packet headers of received ones of the storage operations; and
reorder committing of the storage operations to the plurality of storage drives over the communication fabric based at least on the timestamps indicated by the storage operations.

US Pat. No. 10,660,228

PERIPHERAL STORAGE CARD WITH OFFSET SLOT ALIGNMENT

Liqid Inc., Broomfield, ...

1. A computer peripheral device, comprising:a slot offset element configured to establish an offset for at least a primary circuit card of the computer peripheral device with respect to a slot connector of a host system to generally align the primary circuit card with a centerline of a slot cover opening of a chassis of the host system when the computer peripheral device is installed into the slot connector;
the primary circuit card comprising at least a first storage device connector on a first side, and at least a second storage device connector on a second side; and
a secondary circuit card comprising an edge connector insertable into the slot connector of the host system and configured to carry host signaling for the computer peripheral device;
wherein the offset established by the slot offset element accommodates within a slot width a stackup comprising storage devices installed into at least the first storage device connector on the first side and the second storage device connector on the second side.

US Pat. No. 10,642,659

TELEMETRY HANDLING FOR DISAGGREGATED FABRIC-SWITCHED COMPUTING UNITS

Liqid Inc., Broomfield, ...

1. A compute unit in a clustered environment, comprising:a plurality of physical computing components comprising at least one central processing unit (CPU), storage module, and network interface module coupled over a Peripheral Component Interconnect Express (PCIe) fabric that is configured to communicatively couple the plurality of physical computing components and form the compute unit using logical partitioning within the PCIe fabric; and
a software component deployed by a management processor to at least an associated CPU within the compute unit responsive to formation of the compute unit, the software component comprising:
a monitoring function configured to report telemetry to the management processor related to operation of the compute unit; and
a driver function configured to emulate operation of an Ethernet interface to an operating system of the associated CPU for transfer of communications comprising at least the telemetry to the management processor over the PCIe fabric.

US Pat. No. 10,628,363

PEER-TO-PEER COMMUNICATION FOR GRAPHICS PROCESSING UNITS

Liqid Inc., Broomfield, ...

1. A method of operating a data system, the method comprising:in a management processor, initiating an isolation function in a communication fabric to form a peer-to-peer arrangement between graphics processing units (GPUs) coupled to the communication fabric;
wherein the isolation function isolates a first address domain associated with the GPUs from at least a second address domain associated with a host by at least establishing synthetic devices representing the GPUs in the second address domain.

US Pat. No. 10,592,291

DISAGGREGATED FABRIC-SWITCHED COMPUTING PLATFORM

Liqid Inc., Broomfield, ...

1. A disaggregated computing platform, comprising:a management processor configured to receive user commands to establish compute units among a plurality of physical computing components, each of the compute units comprising one or more associated physical computing components of the plurality of physical computing components selected from among central processing units (CPUs), graphics processing units (GPUs), storage modules, and network interface modules; and
the management processor configured to instruct a Peripheral Component Interconnect Express (PCIe) fabric communicatively coupling the plurality of physical computing components to establish the logical partitioning within the PCIe fabric and form at least a portion of the compute units using domain-based PCIe segregation among ports of PCIe switches that comprise the PCIe fabric.

US Pat. No. 10,614,022

PCIE FABRIC CONNECTIVITY EXPANSION CARD

Liqid Inc., Broomfield, ...

1. A connectivity card, comprising:a plurality of Peripheral Component Interconnect Express (PCIe) connectors that when the connectivity card is mounted into a chassis of a host system are configured to provide external PCIe ports to couple over one or more cables to systems external to the chassis, each of the plurality of PCIe connectors capable of carrying PCIe traffic;
a PCIe switch circuit configured to communicatively couple the plurality of connectors to a shared connectivity interface carried over a host connector of the connectivity card; and
control circuitry configured to monitor for connectivity issues that arise with regard to the one or more cables that couple to the external PCIe ports, and responsively mitigate the connectivity issues by at least reconfiguring a communication pathway in the PCIe switch circuit for at least a portion of the PCIe traffic affected by the connectivity issues.

US Pat. No. 10,585,827

PCIE FABRIC ENABLED PEER-TO-PEER COMMUNICATIONS

Liqid Inc., Broomfield, ...

1. A computing system comprising:a management processor configured to initiate a peer-to-peer arrangement between a first peripheral component interconnect express (PCIe) device and a second PCIe device over a PCIe fabric comprising one or more PCIe switches;
wherein the peer-to-peer arrangement is established to detect data transfers from the first PCIe device directed to addresses corresponding to an address range established for the second PCIe device by a peer-to-peer management entity executed on a host processor, and redirect the data transfers over the PCIe fabric to the second PCIe device such that the data transfers are received by the second PCIe device without passing through the host processor;
wherein at least one data transfer of the data transfers over the PCIe fabric to the second PCIe device is initiated by a request originated by an application executed by the host processor to transfer data from the first PCIe device to the second PCIe device.

US Pat. No. 10,585,609

TRANSFER OF STORAGE OPERATIONS BETWEEN PROCESSORS

Liqid Inc., Broomfield, ...

1. A data storage system, comprising:a first processor; and
a first network interface associated with the first processor having a tunneled network connection established with a second network interface associated with a second processor;
the first processor configured to:
receive one or more packets indicating the first network interface as a network destination and comprising a storage operation;
inspect at an Ethernet network layer of the first network interface at least a storage address included in a header of the one or more packets to determine that the storage operation corresponds to a storage drive managed by the second processor;
establish one or more further packets comprising the storage operation and having an Ethernet header modified from the one or more packets to indicate a layer-2 network address of the second network interface; and
transfer the one or more further packets for delivery to the second network interface over the tunneled network connection.