.svg)
In part 1 of our new series, we break down the key considerations for the first step of your financial reconciliation process: Data Acquisition and Transformation.
Financial reconciliation is often perceived simply as the matching of records across data sources. However, for companies handling high volumes of transactions, the reality is that this process is far more complex. So in our new 4-part series, "Techniques in Financial Reconciliation," we’re breaking down the steps of the reconciliation process in more detail and present techniques that can help you automate and improve your financial reconciliation at scale.
In this series we’ll cover:
This series serves as a primer for product & finance leaders looking to enhance their financial data management and maintain accuracy at scale. We hope in future posts to break down the technical & data engineering decisions that go into building reconciliation systems that scale.
First, let’s recap the various types of financial reconciliation that we’ll be covering in the series.
At its core, reconciliation ensures every transaction in your system of record aligns with your bank, payment partners, and internal expectations.
This verification often requires various types of reconciliation, each tailored to verify a particular element of the funds movement across your data sources. We typically categorize these types of reconciliation colloquially into transaction reconciliation, balance reconciliation, and bank reconciliation, though each share elements of the techniques we’ll discuss in this series.
We’ll refer to these general types of reconciliation processes throughout this series. And each one of these reconciliation processes starts by acquiring the relevant data for verification.
Data acquisition is the first step in the reconciliation process. It's the process of gathering transaction records from across relevant data sources to prepare for comparison and verification. This can include pulling reports from internal systems, payment processor transactions, and bank statements.
But beyond just collecting this data, the data acquisition process requires that data is transferred securely and accurately. It’s important to not introduce false positives in your reconciliation process when establishing data feeds. And for financial services companies, upstream data sources are regularly changing in response to growing product needs and changing regulatory requirements.
Let’s discuss some of the key elements and techniques for building out data acquisition systems for financial reconciliation.
First things first, it’s important to establish some baselines around the reality of financial data for many data sources. A common pitfall when building out reconciliation systems & processes is the assumption that data from upstream systems is always 'clean’. The truth is, many systems experience inconsistencies, errors, and gaps, especially in high growth environments, like financial technology startups. So when building a data acquisition process, it is important to prepare for this reality, as it will have an impact on your reconciliation and financial control processes.
When establishing this process, consider implementing validation checks around source schemas, expected data types, and expected values. Without going too much into the various technical approaches to this (a topic for another post), it’s important to establish visibility to ensure awareness of these scenarios.
Though there’s been lots of progress towards standardizing payments data formats, the reality is that for many companies, payments data can vary significantly in format and structure across schemes, processors, and providers. When establishing data pipelines for reconciliation purposes, it’s important to ensure that adaptability and extensibility are prioritized in order to keep up with changes across various formats and providers. There are many technical approaches to this, but in general, all involve the support of configurable and maintainable specifications, allowing users to apply changes quickly as new formats are introduced and existing formats change. Additionally, depending on the type of payments data being received, it is common to need to resolve a single transaction across various states.
Payment data tends to be inherently stateful, meaning individual records from a source system or file will represent different stages of a single financial transaction. Managing and resolving these transaction states effectively is important for the reconciliation process, so systems must be designed to properly handle appending and resolving records. We’ll explore approaches to this more comprehensively in Part II of this series.
While most systems adopt an append-only and immutable model which requires transaction resolution/aggregation in the reconciliation process, it is important to note that some source systems do change. Therefore, incorporating techniques like change data capture can help to ensure data accuracy and consistency in your reconciliation and reporting.
Change Data Capture (CDC) is a method for tracking and integrating changes in data sources into your financial reconciliation system. Because financial transactions can be dynamic, with updates to status and amounts as they move through their lifecycle, its important to manage and rationalize these changes to ensure seamless reconciliation. Implementing CDC or a similar approach can help you capture these changes in a timely manner, maintaining real-time or near-real-time data accuracy.
The next step, once data has been acquired for the reconciliation process, is to transform and normalize the data. This involves applying various types of operations to the transactions to allow for comparison between disparate sets. This step does not require that a unified format is established between your data sources - though this is a common practice for many reconciliation systems. Data transformation involves steps like aligning date and time stamps, standardizing currency formats, and ensuring that records are properly formatted. Producing a comparable data set is important for precise matching and reconciliation in later stages. When combined with other types of techniques, such as aggregation (which we’ll discuss in part II), this can ensure accuracy across more complex funds flows. It's important to ensure transformations are done transparently, to ensure that proper controls are maintained. As data moves from raw formats to a reconcilable data set, maintaining a record of transformations and derived data ensures transparency for internal and external parties.
During transformation, deriving new data points from existing data is necessary to ensure data is reconcilable. This derived data, such as aggregated totals, normalized account names, or updated dates are crucial, but it's equally to maintain clear data lineage: documenting the data's journey through the reconciliation process. After all, reconciliation is about verifying money is where it should be, and ensuring data lineage is clear and understandable to internal and external stakeholders is vital for ensuring transparency, enabling audit trails, and simplifying error resolution around financial transactions.
All of these techniques should fit into a financial data pipeline process that is able to be updated and maintained in accordance with your businesses resources and scale.
Effective Change Management for Financial Controls
Depending on the size and scope of your organization's reconciliation, financial reporting, and audit requirements, different degrees of control over change management may be necessary. It's important to tailor these controls to fit the specific needs of your organization and ensure that they are neither too lax nor overly stringent.
When constructing your data pipelines to handle financial reporting data, it's crucial to consider a process that can flex and grow while involving the necessary stakeholders and approvals depending on the stage of your business.
To wrap up part 1, ensuring sound data acquisition and transformation processes is a key step in effective financial reconciliation. Implementing the strategies we discussed can help ensure your financial processes remain accurate and efficient. A flexible and scalable approach is key and will ensure you can adapt to your business's unique needs and changes over time. This first part our our 4 part series lays the groundwork for a smooth reconciliation process downstream. Next, we’ll discuss:
If you are looking to improve your reconciliation processes, Proper provides financial data infrastructure, purpose-built for fintech. We help stitch together sources of financial data into a single platform and provide tools for reconciliation and financial operations. Back office infrastructure is critical to financial services, and with Proper, companies can spend less time on financial ops tools and more time focused on their end-users.
We’re modernizing the systems that many innovative fintechs are forced to build in-house and providing services that you can integrate quickly to expedite your financial operations stack.
