Content Centric Software-as-a-Service Solution using Google Cloud Platform

Lessons learned about how to build a multi-tenant, content centric SaaS solution utilizing Google Cloud native features, with a focus on performance, cost, operations and security.

Introduction

Having built multiple content centric, multi-tenant SaaS solutions on Google Cloud Platform, this document will capture the key application and infrastructure architectural decisions we made. Since we built on Google Cloud Platform, we leveraged the native features of GCP in order to significantly reduce the coding and provide additional access.

The design principles we aimed for were:

• Rely on GCP security features as much as possible
• Minimize the cloud costs, and provide visibility per tenant
• Support user’s access to their data through multi-channel access — APIs, CLIs, UIs, CDAP connectors, and direct access through Google tools

Our goal was to create a core capability around management of random content, with the ability to share that content across tenants. The solution was much more than this, but this was the core of the multi-tenant capabilities.

Projects / Tenants

Isolation of projects / tenants from each other is a core principle of a multi-tenant system. However, some of our tenants were very low use tenants and it didn’t make economic sense to isolate each tenant into unique GCP projects with independent application infrastructure.

We had multiple clients who wanted multiple projects and were OK with having the projects in shared infrastructure, but only shared with other projects of the same client. We also had smaller clients who were OK with having a project on a common infrastructure as long as we could assure them that their data was isolated.

Our deployment model allowed for maximum flexibility. In the image below, the projects are all isolated from each other with shared components. For additional isolation, multiple instances (which could contain multiple projects) could be contained in a single GCP project. For maximum isolation, SaaS instance could be deployed across separate GCP projects.

Authentication / Identity

More and more applications have relied on external identity providers, as did our application. We used Google identities exclusively. Any Google identity — a gmail.com email address, or a Workplace (GSuite) corporate identity is used to identify the users. We provided no control over which email addresses were accessible to each tenant — but that would have been easy enough to do.

Why Google identities exclusively? See below for our section on “authorization”. However, we did have one client who wanted Okta identity management — it required simply changing the OAuth2 service provider in 1 line of code.

When a user logs into the application, they are challenged with the Google Login panel that many of us are used to from other applications where we choose to use the Google login. This process gives our application access to the basic information such as name, image URL, email address and profile ID by using the getBasicProfile() method. The “profile ID” attribute is stable and can be used as a unique identifier for this user. (You should not use the email address as this is considered PII in many regions)

Authorization / Access Control

In addition to normal authorization for access to services, we also had to take care to only allow a user to see items in the catalog that were in the project that they were in and had access to. We build custom decorators in methods to ensure that the user had access to the item identifier that they were accessing.

However, this only restricted our view of data in the tables that indexed the items in the repository, and not to the underlying content.

We had to assure that the user couldn’t “go around” our app with Google Cloud tools to access resources and bypass access controls, while still allowing this sort of external access for appropriate actions. In order to do this, we created a group for each project. (An Active Directory group, or whatever mechanism the client uses to manage groups)

Using GCP IAM capabilities with Google identities, and Access Control Lists (ACLs) associated with different GCP resources, we “push” the responsibilities for access control “down” to GCP. As we create a project, we create the new group. As we create the resources for the project, we assign an ACL with the newly created group. Assigning users is a simple matter of adding the user ID to the group.

Once this approach was in place, we had great flexibility to include additional services securely within our architecture.

Cost Management

Variable Costs

In a solution where projects or tenants may have variable usage or storage, it is critical that we are able to trace costs back to the tenant level. In GCP, all components that are used by individual tenants would be “label”ed with the unique tenant UUID. (See labels). Cost per project or tenant can be pulled from the GCP billing tables in BigQuery to get an accurate (and slightly delayed) representation of the bill. As long as all resources specific to the project are labeled correctly, billing can be accurate.

Cost Governance / Visibility

In addition to visibility as above, users are more likely to manage costs if they understand the costs. If the users have flexibility and make decisions about how the data is used, it is critical to provide those users the visibility required to make the appropriate decisions. In the application, we used AI Notebooks (Jupyter notebooks) and that provided users great flexibility. Network egress is a frequent hidden driver of costs.

Cost Control

Google Cloud Platform is very strong in the ability to share data between GCP projects. In the case of a multi-tenant situation, ad-hoc work and costs, the work should be done in a separate GCP project. Since the users are already in the ACL for a resource, it is not required that they are working in the same project as the application. All the key GCP services allow costs to be deferred to the “requesting project”.

BigQuery, while the storage is paid for by the hosting project (and labeled), the cost of executing the queries is passed onto the requesting project. Storage can be set to “Requester Pays” for network egress charges in accessing the data. For Pub/Sub subscriptions, the subscriber pays for the egress costs of moving the data between regions.

Metadata Repository & Search

We built a simple metadata repository to manage the multi-tenant nature of the application, and included sophisticated SOLR-based search capabilities. The MDR APIs were simple /item, /folder, /user and /project URLs for managing the metadata. While we initially “hid” the SOLR implementation “behind” the API layer, we settled on directly exposing SOLR to the end users.

Exposing SOLR required us to share user identities to SOLR, but the advantages were huge and we were able to easily build complex search filters (like a shopping site) with SOLR as the provider of these services. SOLR was used for almost all retrieval requests, including the main screens where items were listed. The benefits vastly outweighed the difficulties of working with SOLR.

It is a common mistake to assume Google’s Data Catalog could replace this functionality — it cannot.

Conclusion

This paper is not intended to be the perfect solution, rather a summary of what we did.

We debated the use of SOLR versus Elasticsearch, but had SOLR skills readily available. We used a CloudSQL MySQL as the core MDR, but discovered because of just how infrequently we actually used the database (deferring to SOLR so often), we probably could have used something less efficient. The move to exposing the SOLR database was an excellent decision and allowed us to greatly improve the user experience.

The cost management worked excellently. We ended up with a very low fixed-cost infrastructure.

The solution allowed us to create a variety of customized solutions with a strong foundation upon which to base the solutions.