Our team at Infinitive worked on migration and adoption solutions for the financial institution’s data streaming platform to assist the enterprise as well as several lines of business and corporate divisions.
One major area was the client’s application log files, centrally managed by their Cyber division. Our team first requested logs of teams who needed their datasets registered. From there, a parser was created, and the raw schema was registered in the unified dataset registration platform.
Once the data was registered, our team was able to route it to the correct datasets. Automation was created in tech stacks such as Jenkins, AWS Lambda, ELK, Python, Terraform, and Ansible. All the pipelines were put in place for the ingestion, transformation, and registration of the datasets. This resulted in onboarding petabytes worth of data every month via the approved enterprise pattern.
After our project’s completion, the financial institution was able to audit every dataset within the unified dataset platform. This included information such as where the data came from, who created it, how to contact them, how much data there was, and the cost to store the data. The unified dataset platform became more secure, further saving costs by avoiding additional penalties and fines. It also allowed for scalability and usability, resulting in the financial institution getting more value out of their data.
- Allowed for the automatic provisioning of ingestion sinks and storage upon successful registrations, which can then be utilized within code/data pipelines
- Tracked all sensitive data across the organization; customer data, social security numbers, geographical requirements (GDPR, etc), credit card info, etc
- Allowed for peace of mind for all data use cases, including for audit
Our team was able to strengthen the financial institution’s data ecosystem by increasing adoption of preferred enterprise patterns, thus allowing for easy access to their data while also making it more secure and scalable. Ultimately, Infinitive enabled the institution to get more value out of their data and better comply with regulations.