US Patent No. 10,218,808

SCRIPTING DISTRIBUTED, PARALLEL PROGRAMS


Patent No. 10,218,808
Issue Date February 26, 2019
Title Scripting Distributed, Parallel Programs
Inventorship Stephen Milton, Lyons, CO (US)
Duncan McCall, Greenwich, CT (US)
Assignee PlaceIQ, Inc., New York, NY (US)

Claim of US Patent No. 10,218,808

1. A method, comprising:obtaining a specification of a data analysis to be performed in parallel on a computing cluster comprising a plurality of computing nodes;
parsing the specification of the data analysis to identify rules applicable to the data analysis;
based on the parsed specification of the data analysis, determining which data is implicated in each portion of the data analysis to be assigned to the plurality of computing nodes of the computing cluster;
determining that a portion of the implicated data is not already present in memory of at least some of the plurality of computing nodes of the computing cluster;
in response to the determination, distributing the portion of the implicated data according to an index that positions related values of the data on the same computing nodes of the computing cluster, wherein distributing the portion of the implicated data comprises calculating index values for tile records based on geographic location such that tile records for adjacent geographic locations are grouped together on the same computing nodes;
translating the parsed specification of the data analysis into mapper rules and reducer rules, at least one of which includes one or more parameters specific to data on a given computing node among the plurality of computing nodes of the computing cluster;
determining which computing nodes of the computing cluster have data relevant to which rules of the mapper rules and the reducer rules and sending the mapper rules and the reducer rules to the corresponding computing nodes for execution in MapReduce routines;
executing the mapper rules and the reducer rules on the corresponding computing nodes; and
aggregating results of executing one or more of the mapper rules and the reducer rules.