That is how I called it in DataStage below. The scripts are easy-to-read text files you can run (ad hoc or scheduled) in the GUI, or from the command line. SortCL jobs can be hand-written, generated via a job wizard, or ETL workflow palette (with an IRI Voracity data management platform subscription 2). It also uses the CoSort engine in the file system, which takes advantage of multi-threading, modern memory management and I/O optimization techniques. SortCL is fast primarily because it combines the transforms (and other functionality) in the same job script and I/O pass. The IRI CoSort product - and its SortCL data definition and manipulation 4GL program doing all the transforms - is shown above in the free IRI Workbench Eclipse GUI’s syntax-aware editor. Before I do that, it should be noted that the raw data used for this job was randomly generated with the same SortCL executable using embedded ‘RowGen’ functionality. In addition to the speed differences, which I try to account for in this article, I also explain the methodology by which I ran the same transform jobs … first alone, and then together. Below are the times it took to sort, join, aggregate, and filter two input files and create two output files: In this POC, I transformed flat files with combined row counts of 2MM, 5MM, 20MM, and 50MM in the CoSort SortCL program, IBM DataStage, and then together from within DataStage. IRI Workbench, a free GUI option, built on Eclipse™:.IBM InfoSphere DataStage® and QualityStage® 8.7.
But in this case, we tested them with this: ITKeySource, an ETL consultancy in Jacksonville, FL, recently benchmarked relative performance gains running IRI CoSort - and its SortCL program in particular - alongside IBM DataStage.ĬoSort has been used for years in this capacity in high volume data transformation environments, like this one on Solaris. Comparing Filter/Sort/Join/Aggregate Performance
0 kommentar(er)
