Leveraging Google Cloud for Enhanced Data Warehousing: The PGS Journey with Scandic Fusion

Since 2015, Scandic Fusion has been an integral partner in evolving PGS's data analytics capabilities, starting with assisting in maintaining their existing data analytics solution, and leading to building of an Oracle-based on-premise data warehouse in 2016 (and adding multiple systems to it over the next few years).

This enduring partnership set the stage for further collaboration when the time came for a strategic shift to Google Cloud, aiming not just for a migration, but for a transformation enhancing PGS's data infrastructure in terms of performance and efficiency.

The core business of PGS involves gathering and processing a vast amount of seismic data – in fact, several terabytes of raw sensor data are generated by each of its seven vessels every day they are in production. Historically the company owned a lot of physical infrastructure and hardware, including supercomputers.

Before the cloud transition, PGS faced two core challenges related to processing of seismic data:

• Capacity constraints: up to 23 000 channels, equal to 2-3 TB of raw sensor data, are recorded by one PGS vessel per day, which means that they needed more and more extra physical space and hardware that could store all the data;
• Time-intensive processes: Processing seismic survey data can take months, and the on-demand scalability of cloud offered potential for compressing that time in certain cases.

To address these points and create a truly efficient, modern and scalable solution, the company made a decision to transition to cloud computing.

That decision then paved the way to modernize PGS’s data warehousing solution as well, addressing the key challenge of inflexible analytics: even though the on-premise DWH was serving the business with structured, reliable and relevant data, the existing reporting solution (in Oracle BI) was not enough. It lacked the intuitiveness and simplicity to be utilized by business users for flexible yet functional ad-hoc reports, the needs for which were coming from more and more business users.

As seismic data moved to the cloud, the existing Oracle-based data warehouse was also to be migrated, the initiative that commenced in 2020. Google Cloud Platform was the provider of choice for data processing needs, and Power BI was to substitute Oracle BI for business reporting purposes.

Overall PGS's decision to transition to Google Cloud was less about following trends and more about addressing the concrete needs of storing and processing extensive amounts of data. This article will provide some insight on how the data warehouse side of the migration happened, explore the biggest challenges throughout the journey, the benefits of moving to the cloud, as well as the lessons learned.

A new tool and ecosystem switch

Transitioning the existing DWH to Google Cloud wasn't just a matter of lifting and shifting the existing processes; it required a thoughtful reevaluation of tools and workflows. One of the initial challenges was that Google, at the project's outset, didn't possess an ideal “native” tool for PGS's data transformation needs. In addition to that, this was the first time that we at Scandic Fusion worked with Google Cloud Platform (GCP).

^"
Even though this was our first project with GCP, getting to grips with it came naturally to us. Our deep understanding of PGS's data – from their standard to custom systems – gave us a solid footing. Besides, the basics always stay the same no matter the tool. Knowing the fundamentals, we could navigate, analyze, and connect the dots, discerning what worked best for the project and what didn't.
‍
_{Darya Skakouskaya, Data Analytics Architect at Scandic Fusion}

_‍Another important part of the process was to focus on getting the best-fit tools for the job at hand, which in this case resulted in a multi-vendor architecture. When PGS decided to go to Google Cloud, it did not mean that everything would automatically be there – after a thorough evaluation of the business needs, the data reporting tool was chosen to be Power BI.

Migration process

Overall, the migration to the cloud was completed in 2 phases:

• The initial lift-and-shift: a simple replication of ready-made facts and dimensions from on-prem DWH to the cloud, with the purpose of populating Power BI reports from GCP, not Oracle DWH; in here we retained some of the legacy tools – like Pentaho Data Integration that was deployed on a virtual machine in the cloud – for simple tasks. In this step – speed was apriority, keeping in mind the business goal of establishing a Power BI reporting solution with the data from GCP as soon as feasible.
• Comprehensive DWH setup in Google Cloud: we have always known that when business priority from step 1 was achieved, the proper migration and DWH implementation on GCP will follow. The end goal was to make the on-premise DWH near redundant, where possible, and eliminate the dependency on it for business reporting. As the Oracle-based DWH had many source systems, we went methodically – source by source, and were rolling out the updates gradually, substituting sources for the result tables with no disruption for the business and reporting processes.

In a nutshell, the first step was ensuring that objects created on Oracle db were available within Google's environment. Data transformation at this point was done by Pentaho, but in the next phase it had to be substituted with a more cloud-compatible solution. During the decision-making process about how we’ll ensure this, Google's acquisition of a new data transformation tool, Dataform, provided a timely, cost-effective and long-term solution. There were plenty of other things to figure out (alerting and monitoring solution, various specifics of cloud functions, pub-subs, staging process from on-premise and soon), but overall Google Cloud Platform has proven to be an ever-improving but enterprise-ready platform. After seeing GCP in action we could add it to our toolbox and wholeheartedly recommend it to other clients considering a switch to the cloud.

Benefits realized

By transitioning to Google Cloud, PGS unlocked new levels of efficiency and agility. For the core business, the cloud's capacity allowed for quicker processing of data, reducing what previously took months to a matter of weeks. Additionally, the cloud environment removed the need for long-term commitments to physical infrastructure, such as leasing buildings and maintaining supercomputers. For the business data analytics, we achieved significant improvements along the same lines – speed, efficiency, excluding hardware upgrades and maintenance, optimized costs, as well as pay-for-what you use model. Cloud offers native components that are more user-friendly. It's much easier for people to access and use it versus most of on-premise solutions.

To give an idea of the speed improvements, here's how data processing looks like when comparing some of the main systems, on-prem vs cloud:

• ERP system #1: from 1.5 hours on ELT (30 min staging, 1 h transform) on-premise to <10 min (3 min staging, 4 min transform) in the cloud – 9x faster
• ERP system #2: from 2.5 hours on ELT (10 min staging, 2 h transform) on-premise to <15 min (6 min staging, 7 min transform) in the cloud – 10x faster

Key lessons

By looking back at the journey to Google Cloud, it’s easy to highlight three key elements as the main driving force – trust, technological agility, and collaboration – each playing a crucial role in the successful implementation of the project.

The Power of Trust in Long-Term Projects

Trust emerged as a critical theme throughout this project, and we cannot emphasize it enough. Client’s reliance on our expertise (which was further developed during the project and in cooperation with the great, competent team on the side of PGS), even as we ventured into the uncharted territory of Google Cloud, shows the importance of building and maintaining strong relationships over time. This helped us implement a project that involved significant shifts in technology and strategy.

^"A couple of the advantages in cloud environments are pay-as-you-go licensing models and software-as-a-service delivery.

This means that the amount of time and cost invested in testing a methodology is limited, and it is possible to change direction at an early stage if wanted functionality is not achieved.

Long term this means a shift from focusing on admin to focusing on functionality and content.
‍
_{Even Wøhni, Technical Application Expert, Technology & Digital at PGS}

Broad Technological Expertise as a Catalyst

This project reaffirmed the power of a comprehensive understanding of core technological principles and a diverse toolkit. Our prior experiences with Oracle, Microsoft, SAP, and other major providers enabled us to approach Google Cloud with confidence. It’s clear that while the tools and platforms may change, the underlying principles often remain consistent, allowing us to determine what could and could not work, even working with less familiar tech stack.

Collaboration and Openness

In projects like these, it’s also important to have a good collaboration with the vendor. In this case, our broad experience across various technological domains enabled us to engage in productive dialogues with Google’s team. Sometimes we even challenged them on the solution Google’s team deemed the best choice, and they saw that our suggestions had a solid base. Our experience here demonstrated that understanding a tool's potential in the context of client’s specific needs is just as important as the technical know-how, enabling us to help select the most effective and sensible solution for PGS.

The transition of PGS to Google Cloud was a journey marked by learning, growth, and strategic advancement. The project proved that with a solid understanding of technology fundamentals, a relationship built on trust, and collaborative exploration, even the unknown can be navigated successfully.

‍

‍