Many banks continue to use long-standing mainframe-based platforms for their back-end processes. However, we at Coastal have adopted cloud integration to securely and seamlessly exchange data. This transition gives us the ability to collaborate and offer customers a broader range of services. To support personalization, risk management, reporting, and compliance, we recently adopted the Databricks Data Intelligence Platform and Delta Sharing protocol to facilitate secure data sharing with partners while maintaining regulatory standards.
Leveraging Technology and Innovation in Community Banking
Coastal needed to exchange significant volumes of data in near real-time with embedded finance partners and other parties, which posed challenges due to legacy technologies and batch processing still prevalent in the financial services industry. To scale our BaaS model, we required an efficient data management and sharing solution that would also meet security, privacy, and compliance requirements. Automation became essential as customer information increased. Due to limited staff, we sought technology tools that could be easily learned and quickly implemented.
Addressing Data Complexity with Delta Sharing
We initiated our BaaS platform development to build a modern data platform to meet data sharing and compliance needs. This process included solving issues related to data ingestion, quality, speed, and scalability. After considering several solutions, we transitioned from a data lake-based system to a lakehouse architecture using Databricks, which addressed concerns associated with synchronizing batch and streaming data and managing schema changes. The Databricks platform provided tools for handling large-scale data and adapting to changing data structures.
The bank expanded use of Databricks features such as Auto Loader, Structured Streaming, Lakeflow Declarative Pipelines, Unity Catalog, and repositories for CI/CD, implementing software engineering best practices in data management. Segregated environments for development, testing, and production, along with governance over code promotion, were incorporated to maintain data privacy and security. Databricks’ compatibility with Azure, and its consolidated toolkit for data transformation and engineering, helped manage risk by reducing complexity.
Initially, the team evaluated multiple cloud-native solutions to migrate from batch processing to real-time data operations, recognizing the need to understand incidents as they occur in a highly interconnected financial environment. Integrating real-time and batch systems remains a challenge industry-wide. The selection of Databricks aimed to provide access to a community of users and ongoing innovation. After comparing ETL, BI, analytics, and data science platforms, Databricks was selected for its performance, scalability, flexibility, and cost-effectiveness compared to alternatives. Features like Auto Loader, monitoring, and cost management supported robust software engineering, allowing Coastal to increase scale despite limited resources.
Improving Efficiency and Expanding Partnerships
Since the launch of Coastal’s embedded finance division, CCBX, it has become the main channel for customer acquisition and revenue growth through BaaS program fees. Databricks has contributed to improved operational efficiency and faster project completion timelines. For example, activities previously requiring 48 hours can now often be completed in 30 minutes due to access to near real-time data. Data sharing between teams has been enhanced, enabling rapid integration and onboarding of new partners. Databricks’ ability to unify live and batch data was cited as a key factor in its adoption.
Delta Sharing facilitates secure, cross-platform data sharing with partners operating on legacy systems. Coastal uses this capability to manage inconsistent incoming data and integrate it efficiently, citing cases where partner data for tens of thousands of customers was processed within days rather than months. As a result, Coastal expanded from serving 40,000 customers directly to approximately 6 million through its partner network.
Delta Sharing on Databricks has also streamlined integration of new partner data sources, significantly reducing the time required for onboarding. Productivity improvements have been noted both internally and among partners, with about 20 organizations currently working with various aspects of Coastal’s BaaS offering. Recent initiatives, such as collaboration with fintech startup One, demonstrate the platform’s capacity for quick data sharing and integration.
Data and AI Initiatives
With a data infrastructure in place, Coastal is exploring technologies including generative AI to automate tasks across business groups, such as investigations into financial crimes. The focus is to move beyond regulatory compliance and actively minimize illicit activity in the financial system. This approach aligns with Coastal’s mission to apply finance for wider societal benefit.