Technical Approach
Rather than using a relational DB approach, ACR uses a JSON document based approach. Each catalog submission is parsed and converted into an efficient JSON representation. Once that has been done, the JSON document is stored in the ACR Postgres DB via a single bulk action, versus possibly millions of separate DB actions that would be required using a relational DB. As a result of these efficiencies, even the largest catalogs require only 4-5 hours end-to-end, from submission to availability in CORS.
PostgreSql was chosen as the DB, since it supports both structured and unstructured data effectively. To host the solution, the MCaaS model was selected because it supports the vision for establishing an API based, microservices architecture and leverages the benefits of containerization and AWS managed services. Amazon Elastic Kubernetes Services (EKS) is used to host the ACR microservice (pods) nodes. AWS S3 object storage services are used as storage mechanisms in support of an event based architecture using Kafka to process the catalog through various events.
Key Technical Improvements
The technical approach implemented by ACR has resulted in three major improvements in the way that GSA processes and manages catalogs. First, it provides a catalog schema that is flexible, extensible, and maintainable. Within the schema itself, business rules and associated validations can be set-up. without having to make code changes to the underlying implementation, as would otherwise be needed in the legacy system.
Another major benefit of ACR is scalability and raw catalog processing speed, which has been increased by orders of magnitude. In the legacy system, large vendor catalogs may require a day, or even several days for availability in CORS for CO review. In ACR, even the largest catalogs are available for CO within hours of submission. Faster catalogs timelines result in faster times to market for vendor offerings.
The third major benefit of the ACR approach is a standardized JSON representation of MAS catalogs, which can be leveraged by providers and consumers of the catalog data. As an example, this has been leveraged by the new FAS Catalog Platform (FCP),which submits MAS product catalogs to ACR via the standardized JSON format, without the need for any parsing or data conversion by ACR. Moreover, any future providers or consumers of GSA MAS catalog data can use the same JSON format in providing or consuming catalog data.
ACR has implemented 22 Microservices, which are deployed into the FCS MCaaS environment to provide system logic, data monitoring, and transfer capability to FCP, ACR and Legacy Pipeline operations. Amazon Elastic Kubernetes Services (EKS) is used to host the ACR microservice (pods) nodes.
ACR utilizes the Postgres relational database to provide data persistence capability. ACR stores catalog data and related information in Postgres. All credentials and connection strings are managed by the MCaaS team and are provisioned to each microservice via Kubernetes secrets and environment variables.
AWS S3 Buckets provide static file storage and retrieval capability internally and with the FCP. ACR exchanges data with other systems, like FCP and GSA Advantage, using objects in S3. Access to S3 is granted via assumed AWS roles within the Kubernetes cluster. The ACR microservices do not store credentials as assumed roles are managed via the platform.
Kafka provides data streaming capability between ACR microservices. ACR stores catalog data and the related information in Postgres. All credentials and connection strings are managed by the MCaaS team and are provisioned to each microservice via Kubernetes secrets and environment variables.