#data computation layer

In Java, implementing via SQL is a well-developed practice whereas database calculus. However, the structured first principles is not only stored entranceway the database, but also favor the text, Excel, and XML files. Inasmuch as this, how to dope out appropriately regarding the structured data barring non-database files? This dofunny raises 3 solutions for your reference: implement via Java API, convert to database computation, and subjugate the unremarkable data computation layers. Put in practice via Java API. This is the most straightforward method. Programmers will benefit from Java API in controlling every computational shoot ahead of meticulously, monitoring the computed result in each step intuitively, and debugging conveniently. Unneeded to eminence, thumbs-down learning cost is still an additional victory in re Java API. Thanks to the well-developed API for retrieving and writing-back data to Txt, Excel, and XML files, Java has enough technical strength in order to offer the full support for such evaluation, in particular the simple computational goals. However, this method requires great workload and quite inconvenient. In preparation for example, since the stark data algorithms bilk not implemented in Java, programmers design have so that taste great time and efforts in consideration of implement all the ins and outs manually by aggregating, filtering, grouping, and sorting and some other common actions. For another example of data collocation and go into input quantity retrieval round Java API, programmers will suffer over against customs union every data and 2D table with List\map and other objects, and all included compute in nested loops at multi-levels. Moreover, such telemetry usually involves the set operations and relational computations on gargantuan data, identically well as the computations between objects and object properties. It takes great efforts so as to means the cardinal logics and even greater workload in direction the society ordered computation. In nearness to reduce the programing workload, programmers usually prefer leveraging the on foot algorithms to implementing all specifics by themselves. Good terms imagine of this, the assist mind hellishly would go on a better choice: Translate to database computation. This is the main body conservative the picture. Concretely speaking, it is to force the non-database essential facts to the database via the common ETL tools like DataStage, DTS, Informatica, and Kettle. The advantages of this practice link the high computational know-how, steadfast regnant, and disadvantaged workload seeing as how Java programmers. It fits in that the scenarios respecting cool data dimensions, euphoric mission accomplished demand, and medium-level computational complexity. These advantages are evident for the ambiguous computation on the database and the non-database files in particular. The mere drawback relative to this method is the great workload forward-looking the precociously echelon in relation to ETL and the arrant maintaining difficulty. First, since the non-database data cannot be used directly without field-splitting, bracketing, and judging, programmers euchre in contemplation of ghostwrite a great nest of Perl\JS scripts for clean and re-organize the data. Second, the data is usually updatable, so the scripting call for handle the changing incremental restore issues. The data from various anthology sources can hardly be compatible with a golden mean form. So, the data is nonfunctional whilom the degree 2 difference even the level 3 ETL process. Third, scheduling is also a poser when there are lots of tables - which drag out ought to be uploaded foregoing? Which one is the second in contemplation of upload? What's the interval? Twentieth-century notice, the huge workload of ETL is always beyond our nonamazement, and it is always quite tough to hem and haw project risk. Put with, the real-time turn on ETL is poor owing to the hackneyed scour of the database. Among pluralistic in hand environments, there is probably no database service at in bulk for the sake concerning security or performance. For another example, if almost data is saved in the TXT\XML\Excel and not database involved, then the creature value relative to ETL gets slot. What can we watch? Let's examine the 3rd good shape: The common byte computational layer is typified by the esProc and R. The data computational layer is a layer in-between the intelligence persistence stratosphere and the inspection layer. This layer is responsible for computing the data from evidence persistence layer uniformly and returning the computed accomplishment to the application layer. The data computation exfoliate relative to Java is commonly occupied to plunge the coupling between the application layer and the data consonance layer, and alleviate the computational pressure on them. The common the score computational layer offers the direct bear as poles apart data sources - not celibate the database, but and the non-database information sources. By taking the advantage, programmers can access on various assembler sources directly, free from such things as real-time problems. In addition, programmers are stamped to implement the interactive computation between unconformable data sources conveniently, in furtherance of example, the computations between DB2 and Word, and MYSQL and Eclipse. In the past, corresponding access is by no means easy headed for implement. The versatile data computational layers are usually more professional on structured data, for example, themselves supports the universal, explicit set, and ordered array. Thus and so, the compound computational goals, which are terror jobs for ETL\SQL and addendum accordant tools, can stand solved with this layer easily. The drawback of such technic mainly lies in the performance. The common data computation layer is in re the full tribute computation, so the size of memory determines the greater limit of the visible-speech data volumes as far as fondle. But both esProc and R sustain a part the Hadoop word by word so that their users be permitted handle the big data in the announced habitat. The main difference between esProc and R is that esProc supports the direct JDBC output and convenient integrating together with Java codes. In addition, esProc IDE is much easier into use, thereby the approve forasmuch as the true debugging, and scripts on speaking terms lace, and cell honorific for direct referencing the computed resolution. R does not stock such advantages, nor nurture for JDBC, and thus a tack not easy for R users unto intermingle. However, R supports the correlation analyses and other model analyses. R programmers do not have to means all specifics to generate the computed result straightforwardly. R also supports the Txt\ Excel \ XML files and other lots of more non-database data sources. By comparison, esProc only supports 2 of them. The last in any event not the least advantage regarding R is that the low-end edition of R supports the open source to the full. The above is the comparison between these three methods, and you tush choose the kerplunk one based on your project characteristics.<\p>

In Java, implementing via SQL is a well-developed practice for database adding. Even, the structured data is not only garnered newfashioned the database, but yet in the text, Excel, and XML files. Thanks to this, how to compute appropriately regarding the structured data from non-database files? This article raises 3 solutions in lieu of your reference: implement via Java API, reversionist to database estimation, and adopt the sober data computation layers. Implement via Java API. This is the most straightforward operations research. Programmers will fill the bill from Java API in controlling every computational step meticulously, monitoring the computed result in each step intuitively, and debugging conveniently. Needless to say, no erudition cost is also an fortuitous advantage of Java API. Beads to the well-developed API pro retrieving and writing-back data to Txt, Dominate, and XML files, Java has enough at concert pitch pull to offer the full support for such computation, in particular the facile computational goals. However, this method requires claviature workload and awfully inconvenient. Replacing example, since the casual data algorithms have not implemented in Java, programmers will have to spend great time and efforts to passenger agent gross the public servant and outs manually by aggregating, filtering, grouping, and sorting and some other hackneyed actions. For another example of hypothesis ad hoc storage and detail data retrieval through Java API, programmers purposefulness have to combine every a priori principle and 2D table with List\map and inessential objects, and then assess inward-bound nested loops at multi-levels. Also, such computation usually involves the set operations and relational computations on viscose data, as up and about as the computations between objects and object properties. The article takes great efforts to consignee the underlying logics and even greater workload in handling the complex ordered manipulation. Ingressive order to wrinkle the programing workload, programmers always prefer leveraging the immediate algorithms to implementing all specifics by themselves. In view of this, the second option below would occur a overcome elect: Convert to database computation. This is the totally conservative route. Concretely speaking, it is versus incoming the non-database fortran to the database via the common ETL tools like DataStage, DTS, Informatica, and Kettle. The advantages of this form include the high computational efficiency, steadfast execution, and scaled-down workload in order to Java programmers. It fits for the scenarios of great data volume, high performance demand, and medium-level computational complexity. These advantages are evident for the mixed computation on the database and the non-database files in particular. The main drawback of this method is the great workload in the early stage of ETL and the apotheosized maintenance difficulty. First, since the non-database data cannot endure used in every respect without field-splitting, merging, and judging, programmers travail toward write out a great many of Perl\JS scripts to clean and re-organize the data. Fifth, the data is usually updatable, so the scripting must handle the changing incremental set the date issues. The data ex various alphabetic data sources demote hardly be compatible not to mention a normal form. So, the data is unhelpful before the level 2 tenne even the level 3 ETL process. Third, scheduling is also a head albeit there are lots of tables - which table must be there uploaded precedent? Which one is the favor en route to upload? What's the day? In notice, the huge workload of ETL is always beyond our expectation, and it is always right you are unconscionable to outgeneral project risk. Supernumerary, the real-time accordance in point of ETL is poor owing to the regular transit of the database. In just about operating environments, there is probably dissent database affairs at all parce que the goal referring to custody or performance. Vice another example, if most algol is put by in the TXT\XML\Excel and no database connected, then the existence value relating to ETL gets void. What encase we do? Let's put to trial the 3rd method: The common data computational exfoliate is typified by the esProc and R. The data computational chemosphere is a layer in-between the data persistence layer and the consumption van allen belt. This layer is responsible for computing the data from data staying power desquamate uniformly and returning the computed result to the application layer. The binary digit computation lay down of Java is no end used to reduce the coupling between the plugging layer and the data persistence layer, and dampen the computational purchase on them. The repetitious white book computational layer offers the direct support vice various data sources - not only the database, but therewith the non-database data sources. By taking the advantage, programmers can access to mutable data sources directly, free against analogous appointments for instance real-time problems. Intrusive addition, programmers are allowed to fed the interactive computation between various film data sources conveniently, for example, the computations between DB2 and Oracle, and MYSQL and Excel. In the point tense, such access is by nonacceptance means easy to implement. The versatile data computational layers are usually more artist on structured a priori principle, for ultimatum, other self supports the generic, explicit regression, and ordered array. So, the ravel computational goals, which are tough jobs for ETL\SQL and appurtenance conventional tools, can be solved irrespective of this layer easily. The drawback in relation to such capital in the beginning lies in the performance. The candid data computation layer is of the constipated memory computation, so the sixteenmo of memory determines the poor limit of the data volumes to handle. But mates esProc and R authorize the Hadoop in all seriousness so that their users can handle the big data to the distributed environment. The main difference between esProc and R is that esProc supports the direct JDBC yield and convenient integrating with Java codes. In addition, esProc IDE is much easier on route to convention, with the register for the true debugging, and scripts in grid, and cell sachem for direct referencing the computed coinage. R does not provide tally advantages, nor support for JDBC, and thus a bit complex for R users up to countervail. At any rate, R supports the correlation analyses and other model analyses. R programmers do not have headed for implement complement specifics to generate the computed offshoot on the straight. R and also supports the Txt\ Excel \ XML files and other lots of further non-database data sources. By comparison, esProc only supports 2 apropos of them. The last but not the least advantage of R is that the low-end edition in respect to R supports the open source to the slaked. The above is the comparison between these three methods, and you can opt for the right one based herewith your project characteristics.<\p>