ETL (Extract, Transform & Load)
Extract, Transform & Load (ETL) is the generic term that is used when referring to the process of developing and consolidating data feeds into a system.
Adroit would typically develop in core Microsoft SQL Server using the typical SQL stack technologies for ETL, such as SSIS (SQL Server Integration Services). We also can deploy technologies such as MatchIT, Talend & other utilities which mean Adroit has an extensible set of components and framework for ETL development.
ETL developments are highly bespoke to the type of data feeds required, and our development team adapt and design appropriately. We would like to draw distinctions between writing “historic” SQL scipts and Stored Procedures and more modern SSIS developments in recent versions of SQL. These require a different development skillset, and afford clients significant advantages. We do note that putting stored procedures into SSIS is not the same thing!
We are keen that clients benefit from the data quality and error logging and recording that SSIS affords, so in particular for more complex processes, particularly those that involve heavy transforms, slowly changing dimensions, data mining lookups, etc, we see strong advantages in SSIS, through efficiency and performance improvements in regular builds compared to T-SQL alone. We are fans of SSIS’s graphical interface with its functionality to develop large, complex and reliable transforms much more easily than hand-crafted T-SQL.
We develop ETL packages that interface with additional external applications such as MatchIT to help in standardising address data, and inline cleansing of suppressions e.g. deceased or gone-aways. We are used to integrating with APIs for TPS (Telephone Preference Services) or ESP and SMS platforms.