|
|
|
|
|
|
|
|
|
| Data Engineering is the term that we have coined to describe the process of developing structured alpha numeric codes to describe all facets of the data that is entered into an Information Technology application. |
|
|
| |
|
|
| Absence of Structured Taxonomy Prevents Effective Analysis |
|
|
| Much has been made in recent years of the capacity of computers to handle verbose file names, verbose plain language data, etc. However, what is not apparent is the extent to which the absence of structured data taxonomies prevent effective analysis of data. |
|
|
| |
|
|
| Computers Have No Intelligence IF There is No Intelligence in the Data |
|
|
| Computers are exceptionally powerful when it comes to tasks such as addition, sorting, etc but they have absolutely NO intelligence apart from the intelligence inherent in the hardware, software and data designs. |
|
|
| |
|
|
| If a substantial amount of intelligence is incorporated into the solution design and the design of the data, the resulting solution will appear to be highly intelligent. If not, it will be "stupid". |
|
|
| |
|
|
| Success is a Function of Design, NOT Fate |
|
|
| This is a function of design, not some mystical or magical capability that may or may not appear in a solution on the basis of luck, chance, fate or any other metaphysical or supernatural event. |
|
|
| |
|
|
| Data engineering involves: |
|
|
| |
|
|
| 1. FIRST PRINCIPLES DATA ENTITY ANALYSIS |
|
|
| Rigorous first principles data analysis of the entire data space to develop comprehensive entity relationship diagrams for the entire data space. |
|
|
| |
|
|
| Finest Level of Granularity is Essential |
|
|
| These entity relationships must drive down to the most fundamental level of information, for example 25mm countersunk head, phillips slot, mild steel, coarse thread, 8 gauge wood screws. |
|
|
| |
|
|
| Woodscrews or countersunk woodscrews or 25mm countersunk wood screws is NOT a unique classification UNLESS there are business rules which PERMANENTLY exclude all other variations of that screw type from occurring in the organization. Something that seldom happens. |
|
|
| |
|
|
| It is technically very easy to incorporate the full level of detail to describe an object at the time of design, and then to put a default value into every field for as long as the alternative options are not applicable. |
|
|
| |
|
|
| But, provided that provision is made, it takes no time at all to incorporate the other fields of interest IF they suddenly become relevant AND the entire database and all existing reports, data warehouses, etc, provided their design is compliant with the basic design, will immediately adjust to the additional information without difficulty. |
|
|
| |
|
|
| Adding Detail Later can be Fatal |
|
|
| If those parameters have to be added at a later date massive problems can result and simple reports can prove to be impossible. |
|
|
| |
|
|
| It is vital that design of all aspects of the solution provide for data at the finest possible level of granularity (detail) and that every possible field that can be equipped with a validated look-up table or drop down list is so equipped. |
|
|
| |
|
|
| This in turn introduces a substantial additional component of work into the average project, the design of data codes to populate all these drop down lists. THIS is data engineering. |
|
|
| |
|
|
| 2. DATA CLASSIFICATION -- STRUCTURED CODES |
|
|
| It is technically straightforward to populate a look-up table or drop down list with meaningful English language descriptions that can be easily read and selected by the reader. |
|
|
| |
|
|
| However, these same codes will sort alphabetically according to the spelling of the words and will not reflect any finer understanding of the data. |
|
|
| |
|
|
| Detailed Analysis of Data is Critical |
|
|
| Detailed technical analysis of the data whether to equip the software with greater simulated intelligence or to facilitate the analysis of the data will not be possible unless a structured code is attached to every English language description in such a way that greater knowledge and understanding of the contents of the list is encapsulated in the code. |
|
|
| |
|
|
| Thus, the contents of a drop down list or look-up table should comprise a structured alpha numeric code and a plain English description. |
|
|
| |
|
|
| It is not necessary for the code to be displayed to the user, it is possible to program the table so that only the plain English description is displayed but the list is ordered according to the structured alpha numeric code. |
|
|
| |
|
|
| The plain English description can be enhanced by the use of capitals for section headings and the use of indents to display the logic of the underlying code structure. |
|
|
| |
|
|
| This approach makes it easy for the user to select the right information. |
|
|
| |
|
|
| Computers Work Best with Structured Codes |
|
|
| However, when it comes to processing of the data by the computer the structured code is what is used, NOT the English description. Summarization, roll-up, drill down, slice and dice all take place against the structured code so that with one simple command it is possible to summarize the data to a single figure or summarize it at any level of detail that is appropriate to answer a particular question. |
|
|
| |
|
|
| If the structured code is correctly designed, this roll-up, summarization, etc will take place with a few simple commands or clicks of the mouse and no complex analysis scripting will be required. |
|
|
| |
|
|
| The benefits are enormous, data warehouse potential is easily realized, cost of operation of the data warehouse is dramatically reduced, etc. |
|
|
| |
|
|
| Design of Structured Codes is Time Consuming and Expensive -- It is the ESSENCE of Information |
|
|
| However, the design effort is considerable. It can take as long to do the detailed data engineering of an application as it takes to do the detailed architectural design. Both require senior architects with an excellent understanding of the business to develop the overall structure and then require professional teams with excellent business understanding to fully populate the tables. |
|
|
| |
|
|
| JAR&A Deliver Major Competitive Advantage |
|
|
| This is an area of particular expertise in JAR&A and is an area which has given rise to massive benefits for modest investments on a number of projects. This is reported in several of the conference presentations co-authored with clients. |
|
|
| |
|
|
| This is a critical component of creating sustainable, profitable competitive advantage using I.T. and also in design of Strategic Plans. |
|
|
|
|
|
|
|
|
|
|
|
|
|
| Top of Page |
|
| Use "Back" button on browser to return to the previous page |
|
|
