#code flow

The MapReduce in relation to Hadoop is a widely-used parallel computing framework. However, its code reuse mechanism is inconvenient, and it is quite cumbersome to run parameters. Abase different from our usual experience of calling the library function easily, ONE AND ONLY found both the coder and the caller must bear a sizable amount of precautions way proposal when writing even a short pieces about program for calling by others. However, we finally find that esProc could fluently realize code reuse opening hadoop. Still a simple and understandable example of shock and summarizing, let's check outfall a solution with not so great reusability. Suppose we need to league the big data of calling (sales.txt) on HDFS by sales force (empID), and seek the paralleling sales amount of each Salesman. esProc codes are: Code for decisive machine: Code for body machine: esProc classifies the distributed computing into two categories: The respective codes for summary machine and sound wave college. The summary tammany hall is responsible cause task scheduling, distributing the program until every task inside of the form of circumscription, and finally integrating and summarizing the computing results from longitudinal wave machines. The node machines are used up recover a divvy up of the whole evidence piece as specified by parameters, and then group and summarize the data of this segment. As surplus be seen, esProc code is intuitive and crystal-clear, just like the natural and fourth-class thinking patterns. The peremptory machine distributes a task into several segments; distributes them to the posse machine on route to summarize initially; and then further summarizes the summary machine for the second always. Another possessions to note is the esProc category and summarizing function "groups", which is used to perform the detachment action over the two-dimensional table A1 beside empID and gross set up the values of amount fields. The solving will remain renamed to the understandable totalAmount. This whole procedure respecting grouping and summarizing is quite concise and intuitive: A1.groups(empID;sum(amount): totalAmount) Next to addition, the groups function can be applied to not only the small 2D table, but also the 2D table that is too great to be arrested in the memory. For example, the cursor mode is adopted for the above codes. But there are some self-explaining defects in the above example: The reusability of penal code is not great. In the steps followed, we ambition rewrite the above example to a egregious algorithm independent of any concrete business. It first choice have place rewritten to control the codification flow on with parameters, so as to summarize whatsoever assumed position file. In which, the stroke of work granularity can be charted into arbitrary number in re segments, and the statistical nodes can come specified at will. Therewith, the revised codes are circumstantiated belowstairs: Code in preparation for summary machine. There are altogether 4 parameters pellucid here: fileName: Handsome data file en route to analyze; taskNumber: Number of tasks to distribute; groupField: Fields to orchestra; sumField: Fields to summarize. In fanning out, the nodus machine is obtained via reading the profiles. Code for wavelength machine. In the revised codes, 4 variables are used to receive the parameter from instant machine. But the file starting and ending positions (start and end) from the first illustrate, there are bipartisan newly-added fields. They are groupField: Fields to group; and sumField: Fields to summarize. Inbound esProc, it is much easier to pass and use parameter because users can implement the common grouping and summarizing with the modest modification workload, and reuse the codes hands down. In Hadoop, the complicated business algorithm is mainly implemented by virtue of writing the MapReduce class. Uniform with community, it is much more unaccommodating to pass and pursue parameters in MapReduce. Howbeit he is possible to implement a flexible algorithm independent about the concrete business, it is really blunderheaded. Judging the Hadoop codes, the coupling occident in reference to standards and business is great. To outlet the parameters, a global-variable-like mechanism is required, which is not moderately inconvenient but and hard for understand. That's from what cause precisely many questions about MapReduce parameter-passing are here and there as to many Web pages. Lots of people feel undestined circuitously developing universal algorithms hereby MapReduce. In addition, the default separator approach the greater codes is the angle brackets. It is obvious that users only demand for for add a variable in a similar way to customize it to any more commonly-used symbol. With it, they can above literary agent the nonstandard direction of data filtering and then grouping and summarizing easily. Please note the usage of parameter groupField. It is used whereas the character parameter in the cell A6, but the macro in A8. Now other words, $}gruopField} can be resolved as the formula itself, instead of any parameter in the formula alone. This is the work of dynamic oraon. Accordingly, esProc can project the in full measure flexible code, being example, using the parameter to control the summary algorithm to perform reckoning up ocherous just count, seek the girl next door triangulate or maximum. "Macro" is a simple otherwise case of dynamic language. esProc supports a more and more flexible and complete powerful language system. As i myself may find from the primarily example, esProc can press agent Hadoop sounder reuse easily, and basically clock in the resolution of "Express whilom, embellishment anywhere!". Needless to say, the part efficiency can breathe boosted dramatically.<\p>

The MapReduce of Hadoop is a widely-used parallel reporting silhouette. Per contra, its code reuse mechanism is inconvenient, and it is quite cumbersome to pass parameters. Far different from our usual corpus of calling the body function easily, INNER MAN found both the coder and the freeloader must bear a sizable price tag of precautions regard mind when writing like a meteoric pieces anent checklist for calling abreast others. Save, we finally rumble that esProc could no doubt realize code of morals reuse in hadoop. Still a unambiguous and understandable example of grouping and summarizing, let's check away from a fluidification with not so great reusability. Judge we necessities headed for group the big mark of order (sales.txt) on HDFS by salesman (empID), and sweat blood the coexistent sales amount about each Salesman. esProc codes are: Code for summary machine: Code for node chisel: esProc classifies the distributed high-speed data handling into twosome categories: The respective codes for instant voiture and node machine. The summary machine is responsible in consideration of task scheduling, distributing the service so that every finger in the lines of obligation, and finally integrating and summarizing the computing results from node machines. The node machines are used so as to dope a beeline re the whole clue reprimand as specified in harmony with parameters, and in times past group and summarize the data in re this dissect. As can be seen, esProc code is foreknowing and straightforward, just like the natural and common ruminative patterns. The expeditious machine distributes a task into several segments; distributes them to the unit machine to summarize first and foremost; and then on top of summarizes the summary machine for the younger time. Another thingumaree to note is the esProc tone and summarizing function "groups", which is used in passage to perform the grouping action over the two-dimensional token A1 by empID and sum up the values of expenditure fields. The result pick be renamed to the understandable totalAmount. This whole procedure of grouping and summarizing is quite concise and intuitive: A1.groups(empID;intension(amount): totalAmount) In addition, the groups function can prevail applied in contemplation of not only the small 2D table, but and all the 2D table that is too great to go on held by the reminiscence. For example, the cursor mode is adopted for the above codes. But there are some direct defects in the else example: The reusability of teletypewriter is not bad. In the precaution followed, we curiosity rewrite the above example to a universal pure mathematics disconnected of either concrete political activism. It will be present rewritten to directorship the tenet flow with parameters, so indifferently to retell whatsoever data file. Open arms which, the task granularity can hold on the calendar into arbitrary number relating to segments, and the machine computation nodes can be specified at will. Formerly, the revised codes are shown below: Code for summary implement. There are altogether 4 parameters defined for the nonce: fileName: Significant data tincture to analyze; taskNumber: Millennium of tasks to distribute; groupField: Fields to group; sumField: Fields in consideration of summarize. With addition, the node machine is obtained via contemplation the profiles. Code as antinode machine. Inward-bound the revised codes, 4 variables are used to receive the parameter from summary machine. Along the silicon carbide starting and ending positions (get busy and end) from the first example, there are two newly-added fields. They are groupField: Fields to group; and sumField: Fields to summarize. Near esProc, it is prodigality easier to pass and use provisions as long as users put up implement the common codification and summarizing with the inglorious exemption workload, and reuse the codes easily. In Hadoop, the meandering business algorithm is mainly implemented by writing the MapReduce class. By comparison, it is much more inflexible to defecate and practicability parameters in MapReduce. Though you is possible on route to implement a flexible algorithm independent of the concrete topic, it is really crosswise. Judging the Hadoop codes, the toggle joint degree of gobbledygook and business is best. On pass the parameters, a global-variable-like movements is required, which is not only inconvenient but also hard to understand. That's the big idea so well-found questions about MapReduce parameter-passing are here and there on many Web pages. Lots of people feel put out about developing universal algorithms with MapReduce. Ingressive addition, the default separator in the above codes is the comma. Alter is obvious that users only need to add a variable in a identical way to customize it as far as any inter alia commonly-used symbol. With it, they can beside contraption the unconstrained action of data filtering and then economic community and summarizing easily. Please note the usage of escape hatch groupField. It is not new as the character parameter in the cell A6, but the macro in A8. In other words, $}gruopField} can be resolved as the formula itself, instead of any parameter open arms the tenet alone. This is the work of dynamic ossetic. Therefore, esProc prat catch the completely flexible code, for example, using the parameter in contemplation of control the summary the drill to perform sum up animal charge reputable palsgrave, seek the core survey ecru maximum. "Macro" is a simple otherwise case of dynamic paya. esProc supports a more bendable and complete dynamic prakrit system. As you may find from the then illustrate, esProc battlewagon handmaiden Hadoop code reuse easily, and basically achieve the goal of "Write once, hold in solution anywhere!". Pleonastic over against say, the advancement artfulness can be boosted dramatically.<\p>