Controlinho: Nubank’s answer to democratizing data

Written by: Leonardo Abdalla and Michel Herszenhaut

In the fast-paced world of digital banking, data integrity and control are crucial for success. This is where Controlinho, an innovative tool developed by Nubank, steps in. As a leading force in tech innovation, Nubank’s introduction of Controlinho revolutionizes data management and control in the financial sector. 

In this comprehensive guide, we’ll delve into the inner workings of Controlinho, its role in continuous integration, automated control, monitoring, data logging, and its profound impact on Nubank’s operations.

Whether you’re a data engineer looking to streamline your data control processes or a fintech enthusiast interested in the latest data management strategies, this article provides a deep dive into how Controlinho is shaping the future of data in banking.

Nubank is renowned for its tech innovations, and Controlinho is no exception. This tool plays an essential role in our operations as it helps us manage and make sense of our data, which is at the heart of all we do at Nubank.

Software deliver tend to divide the industry into monolithic and microservices architecture. Years ago, software delivery was adopting a monolithic perspective. It was straightforward and easy to manage. Everything was in one place – a single database, a single codebase, and a single deployment. This structure has its advantages, such as easier debugging and testing due to the simplicity of the system. However, it also comes with significant drawbacks like lack of flexibility, scalability issues, and a single point of failure that could bring the entire system down.

Nubank was born onto the microservices architecture as each service could be developed, deployed, and scaled independently. However, it also introduced new challenges, one of which was ensuring data consistency across multiple microservices.

Check here what we’ve learned growing a complex system in a fintech over the past six years.

In the world of banking and financial services, reliable and timely reports are paramount. Incorrect reports can lead to poor decision-making and regulatory issues. Therefore, we at Nubank understood the importance of delivering precise and timely reports to regulators.

Furthermore, the complexity of credit risk evaluation and the role of data in this process cannot be overstated. This complexity, coupled with the need for a tool to automate data controls, led to the birth of Controlinho.

Building a tool like Controlinho required structured planning and focus. Key decisions had to be made to ensure it served its purpose effectively. We aimed to avoid repetitive tasks and simplify the creation of controls. Furthermore, we wanted to make it easy for everyone to contribute while ensuring that everything was logged for accountability.

Democratizing data through a culture of controls was central to our approach, making Controlinho a tool that empowered our teams to take control of their data.

What started as a in-house tool for a squad, rapidly was presented on a Megademo event and adopted by the entire organization accounting for over 17k commits for its usage. 

Controlinho is a tool that organizes, classifies, and executes controls. It thrives on community contributions and has an execution cycle that facilitates the creation of unique controls by different teams. The aim is not just to create a tool but to foster a culture where everyone can contribute to Controlinho.

Controlinho’s fundamental unit is called a “Control.” It consists of an SQL query accompanied by a few parameters and metadata that are defined in a JSON file. Each Control serves as a checklist for Controlinho to inspect the data passing through a service to ensure that it is delivering data as expected. If anything is amiss, Controlinho will alert the responsible analyst. Controlinho leverages Bigquery environment to query datasets and deliver over 1k controls to Nubank’s engineering and non-engineering contributors. 

We understand that in a microservice architecture, each service can represent its own world with unique programming languages and rules between engineers. However, since all important data is materialized into tables in the data warehouse, this is where we have implemented Controlinho’s automatic checks.

Scalability and continuous integration

In an environment where over 250 engineers contribute to our codebase monthly, managing code integrity becomes a considerable challenge. However, at Nubank, we’ve turned to continuous integration as a strategic ally to manage this complexity.

Continuous integration is the practice of merging all developers’ working copies to a shared mainline several times a day. This approach is instrumental in preventing the “integration hell” that typically arises when people work in isolation for an extended period. It helps us detect issues early and fix them while they are still small.

Every time code is committed, it triggers an automatic build process, which runs a suite of tests to ensure the new code doesn’t break any existing functionality. If any issue is detected, developers are notified immediately to address it. This process not only maintains the health of our codebase but also prevents overload in the review process, ensuring a streamlined workflow and faster delivery times.

Data control

Data is the lifeblood of our operations at Nubank, and ensuring its integrity is paramount. We achieve this through automated controls where frequency is defined by the user, a vital part of Controlinho. These controls monitor data integrity continuously, performing checks on various data elements to ensure they conform to defined standards.

For instance, if we consider a practical example like credit risk assessment, automated controls might verify that the credit score data is within the expected range, that there are no missing data fields, and that the data is updated in a timely manner. These controls help us prevent errors, detect anomalies, and maintain the reliability of our data-driven decision-making processes.

Control implementation

Implementation of these controls within Controlinho is a structured process. Control data is carefully organized, with fields categorized as mandatory or optional based on their relevance. This classification ensures that essential data fields are always filled, while providing flexibility where it’s needed.

Additionally, we utilize integration tests extensively to guarantee the validity of data. These tests verify that different components of Controlinho interact correctly, and that data flows through the system as expected. This meticulous approach to control implementation is what makes Controlinho a powerful and reliable tool.

Data feedback cycle

At Nubank, we believe in the power of feedback doing the best for the company, not for specific individuals or team. This mindset extends to our data operations. Once a set of controls  passes all tests, success messages are generated and sent to the relevant teams. This immediate feedback helps reinforce good practices and boosts morale.

However, we also understand that failure is an opportunity for learning. Therefore, when a control fails , error messages are generated and sent to the developers, providing them with valuable insights into what went wrong and where they need to focus their debugging efforts. It is a huge advantage to continuous monitor applications and understand changes across the web of microservices.

All these interactions, successes, and failures are recorded in logs, creating a historical record of our operations. This logging is crucial for future analysis, allowing us to identify patterns, trends, and potential areas for improvement.

Data visualization

Data visualization is another crucial aspect of our data monitoring strategy. It helps teams understand the evolution of their projects and make informed decisions. At Nubank, we use advanced visualization tools to represent our data in a way that is easy to understand, even for non-technical team members.

These tools provide a historical view of controls, allowing us to track progress over time and identify any trends or patterns. This visibility enhances our understanding of our operations, leading to more efficient and effective decision-making.

Control statistics at Nubank

Controlinho has seen widespread adoption across Nubank. We currently have a vast number of active controls, with numerous business areas relying on the tool for their daily operations. This widespread usage is a testament to the tool’s effectiveness and the value it brings to our teams.

Benefits of control at Nubank

The benefits of implementing Controlinho at Nubank are manifold.

 First, it has significantly enhanced our ability to monitor all processes, ensuring data reliability. This has led to an increase in confidence in our data, which is crucial for a data-driven organization like ours.

Secondly, Controlinho has enabled swift detection and correction of errors. With automated controls running continuously, any anomalies are quickly flagged, allowing us to rectify them before they impact our operations or customer experience.

Thirdly, Controlinho has reduced the technical barrier for Nubank employees. By automating complex data control tasks, it has made it easier for non-technical staff to engage with data, fostering a culture of data democratization.

In addition, Controlinho has been instrumental in preventing unnecessary repetition of tasks. By automating data control tasks, it has freed up our teams to focus on more strategic, high-value activities.

Lastly, Controlinho provides useful logs for auditing and decision-making purposes. These logs offer valuable insights into our operations, helping us to make informed decisions and drive continuous improvement.

In summary, Controlinho has played a pivotal role in transforming our data control processes, leading to improved operational efficiency, enhanced data integrity, and a culture of data democratization. It’s not just a tool; it’s an enabler, empowering us to take control of our data and leverage it to drive business success.

What is like being an Analytics Engineer at Nubank? Know more.