Technical Approach

Problem Statement: The ASSIST team had to carefully account for technical assets and development tools to ensure that all necessary assets and tools were available and functional after migration into the FCS cloud ecosystem.

Solution: ASSIST is a tenant within the FCS MCaaS ecosystem and inherits MCaaS boundaries. GSA FCS architecture consists of several AWS Virtual Private Cloud (VPC) networks.It is logically isolated from other virtual networks within AWS cloud. GSA FCS MCaaS tenants have full control of their data and applications within their secure AWS VPC environment.

Within the MCaaS system boundaries, the ASSIST system assets are supported in VPC subnets. Components of the ASSIST environment are broken down into the following groups of asset types/categories.

Asset Type / Category	Vendor / Product	Provider	Description of function or service provided
JavaScript/front end UI framework	Angular	ASSIST team	Industry-proven, UI/UX frameworks for rapid development of mobile-ready, cross-browser, component-based applications (also leveraging sass for CSS ext. and NPM Webpack for code packaging and obfuscation).
Java ORM framework	Hibernate	ASSIST team	Java Persistence APIs (Data Services) and JP/HSQL via annotations on POJO Entities and Repo classes for CRUD and complex operations.
JavaScript framework	Bootstrap	ASSIST team	A JavaScript framework that provides common UI elements in use throughout the ASSIST front-end.
JavaScript framework	jQuery	ASSIST team	A JavaScript framework that provides common functionality in use throughout the ASSIST front-end.
JavaScript framework	Data Tables	ASSIST team	A JavaScript framework that provides grid layout functionality in use throughout the ASSIST front-end. Includes Editor extension of the Data tables foundation.
Authorization and Authentication	ForgeRock AM and DS	ASSIST team customized ForgeRock AM	Manages authentication and authorization for ASSIST users. Also manages the hand off for ASSIST to GSA Single Sign On solution (SecureAuth).
Java framework	Spring	Build into BaseImage by ASSIST team	Spring framework provides comprehensive infrastructure support for developing Java applications
Java framework	Spring Boot	Build into BaseImage by ASSIST team	The Spring boot provides a comprehensive programming and configuration model for modern Java-based enterprise applications - on any kind of deployment platform. It eliminates the boilerplate configurations required for setting up a Spring application.
Application Server	RedHat JBoss Application Server (EAP/AS)	Build into BaseImage by ASSIST team	Java container to deploy custom Java Applications and use of Java scheduler to execute batch jobs according to a schedule frequency.
Java framework	Java	Build into BaseImage by ASSIST team	The SDK/APIs that are essential to the underlying service applications that support the ASSIST application.
UBI	DoD Ironbank	ASSIST team	Base OS image (UBI 7.x and 8.x) for ASSIST containers
nginx	F5 Inc.	Build into BaseImage by ASSIST team	Light-weight service to serve static content including ASSIST Angular front-ends and CM Static Content.
SSH File Transfer Protocol (SFTP) Server	ASSIST	Custom application by ASSIST team	ASSIST External File Transfer (EFT) Services enable the retrieval and transmission of files between ASSIST and external systems that are required for proper system operation. EFT Services include a Secure File Transfer Protocol (SFTP) server, fielded as a Kubernetes deployment, an outbound jobs service, as well as an outbound Kubernetes cron job.
Simple Mail Transfer Protocol (SMTP) Server (GSA Google Relay)	GSA	GSA	Used to capture outgoing email, in Staging only to perform quality control of business logic in application(s) and verify email events happen as appropriate. Prod and Preprod, use GSA SMTP gateway.
Database	Postgres Aurora Serverless	MCaaS	Database server to store data in, execute stored procedures according to a schedule frequency.
Monitoring, Alerting and log management	Datadog	MCaaS	Datadog is the official Observability tool offered and managed by FCS MCaaS.
Version control / Source code repository	GitHub	MCaaS	MCaaS provides a Version Control system used as a code hosting platform for version control and collaboration by MCaaS.
CI/CD tool	Jenkins	MCaaS	MCaaS provided Continuous Integration to build code and deploy new versions of the software.
Security Scanning Tool	Anchore	MCaaS	GSA provided Container Security Services that perform CVE and Compliance scans on the container images during build, deployment, and runtime.
Security Scanning Tool	StackRox	MCaaS	StackRox is Kubernetes-native security tool
CI/CD tool	Helm	MCaaS	COTS Product used to do “Continuous (Automated) Deployments”.
Storage	Amazon	MCaaS	Secured Storage that ASSIST uses AWS S3 to support processes for both Help Document storage within FCS and Data Transfer supporting the SFTP servers. Provisioned by the FCS team.
Anti-Virus Scanning	ClamAV Virus Scanning	MCaaS	The ClamAV virus scanning solution supports performing anti-virus scans on all ASSIST Document Management attachments stored in the ASSIST S3 bucket.
Active Directory in non-production environments	OpenLDAP	ASSIST team maintains it in non-production environments	Stand-in for active directory in ASSIST’s non-production environments
Front end UI for active directory in non-production environment	phpLDAPAdmin	ASSIST team maintains it in non-production environments	Front end UI for the stand-in active directory in ASSIST’s non-production environments

Following is the list of GSA Enterprise Architecture Analytics and Reporting (GEAR) approved tools and software utilized for the development process on the Govt. Furnished Equipment (GFE) by the ASSIST team.

GEAR Approved Tools & Software
Software	Description
PuTTY 0.78	SSH tool - for generating SSH keys for access to the MCaaS jump box
PGAdmin	Database client
SQL Developer	Database client
Amazon Corretto	Amazon distribution of OpenJDK.
Apache jMeter	Performance/load testing tool primarily for API testing.
Apache Maven	Java build tool used for most ASSIST software.
DBeaver	Database client
Eclipse	Integrated development environment
Filezilla	File transfer tool for testing/verification of SFTP endpoints.
Git for Windows	Git client for Windows.
GitHub Desktop	Visual GitHub client
Helm.exe	Package manager for Kubernetes
IntelliJ IDEA Community	Integrated development environment
MS Visual Studio Code x64	Integrated development environment
Node.js	JavaScript runtime environment. Includes build tools (npm) for Angular projects
Postman	API testing utility
Spring Tool Suite	Integrated development environment based off of Eclipse but targeted to Spring
Jira	Project management and issue tracking
Confluence	Documentation collaboration tool

Note: In addition to the information in this section, please refer to the whitepaper on ASSIST’s Migration to Postgres that was recently published in the Knowledge Center.

Problem Statement: ASSIST needed to ensure a smooth transition to the FCS ecosystem that minimized licensed costs and streamlined the environment creation process.

Solution: ASSIST's underlying data store and operations have historically been supported by an Oracle RDBMS database. ASSIST has operated using an Oracle database as its primary data store both on-premise and upon migration to the FCS/MCaaS cloud ecosystem. To further modernize ASSIST and leverage cloud offerings, streamline the environment creation process, and minimize license costs, the ASSIST Team has initiated a process to migrate from AWS RDS Oracle. As part of this effort, The Team conducted research and developed white papers to assess features, constraints, and risks associated with various platforms which led to the choice of PostgreSQL as the target platform. An additional analysis was performed to evaluate the options within the MCaaS environment. These activities resulted in AWS RDS Aurora PostgreSQL being chosen as the target database platform for ASSIST.

The Oracle to Postgres migration process that the ASSIST Team is implementing is detailed below: Both SCT and DMS have been employed to perform DML and DDL migrations, respectively. The SCT process is used to generate Pre-DMS DDL scripts that populate the schema of the new Postgres database. Once those scripts are run, DMS is employed to migrate data from Oracle to Aurora PostgreSQL. Multiple checkpoints have been put in place to allow for modifications of the SCT, DMS, or SQL Scripts that are part of the migration process. With fine tuning and iterative changes in place, the ASSIST Team is finalizing the migration procedures and using them to replicate the Oracle to Postgres migration in multiple environments. Once final validation is performed, the process will be replicated across additional environments concluding with the production migration.

Lessons Learned: During the migration process, lessons learned were documented for future reference. Major lessons learned are shown below.

Problem Statement: ASSIST needs to maintain middleware to run Java applications in the cloud.

Solution: For over a decade now ASSIST has leveraged RedHat's JBoss Enterprise Application Platform as middleware for running our java applications, culminating in JBoss EAP 7.4. Previously on-prem, several of these application servers were installed directly on each VM to enable traditional system administration and operations tasks. With the aid of aliases and the flexible configuration capabilities of JBoss, the DevOps team was able to function effectively in daily maintenance of the system.

With our move to a managed cloud and away from monolith applications, we containerized our JBoss solution, working off a RHEL 8 + Java 11 base image layer. The process of layering the ASSIST images is shown below.

Implementation

Each JBoss container runs in standalone mode listening on port 8080, as part of the overall multi-container deployment for each ASSIST application. It builds on top of the Java 11 image and in turn the UBI 8 base image, which establishes the non-root user 'assist' at the home directory /jboss where all further container operations are performed. At runtime, the docker entrypoint script simply executes the standalone.sh start script included with JBoss, and upon receiving a signal to terminate the instance, graceful shutdown is initiated via the JBoss CLI.

Boilerplate configuration for each instance is established in the standalone xml file, which is mounted to the deployment by way of configmap. All other customizations and configurations are applied via environment variables, JVM runtime options, and our override properties mechanism (also mounted from a configmap) in Flux. Sensitive values are stored encrypted in the configuration, and are accessed by the init container 'cryptkeeper' to decrypt at runtime. Cryptkeeper is essentially a containerized version of the AssistStringEncryption routine already used by our applications that performs a decrypt operation once against all encrypted properties then destroys itself.

At minimum, the kubernetes resources packaged for each JBoss instances' helm chart include:

1. Deployment including the pod that consists of the cryptkeeper initContainer, in addition to both the JBoss 7 container and the istio-proxy sidecar container.
2. Service exposing port 80 within the cluster, mapped from the container port 8080.
3. Virtual Service at port 80 allowing communication from the istio cluster gateway.
4. ConfigMap containing properties files' content, and the jboss.xml. Encrypted configuration values are mounted to a separate volume to be fed as inputs to cryptkeeper.
5. HorizontalPodAutoScaler to scale the amount of replicas in the deployment based on CPU + memory utilization.

Hardening

On top of the security benefits already offered by using Ironbank's UBI8 OS image, considerable effort has been invested into ensuring JBoss EAP 7.4 is running with the latest patch offering from RedHat to address the numerous potentially vulnerable components bundled with the archive. The base JBoss 7.4 zip bundled with the latest patch zip is downloaded from s3, the archives are extracted, the patch is applied via the JBoss CLI at build time, then all unused files are removed from the image. For any vulnerabilities not covered by this method, manual patches are created and applied during the process of building the JBoss image.

Problem Statement: ASSIST requires the ability to both push and pull files to various external interfaces via SFTP or SCP.

Solution: Certain services such as SFTP are more limited for security reasons in the MCaaS environment. Because MCaaS does not support AWS Transfer Family yet, the ASSIST Team created custom solutions to work around limitations related to SCP and SFTP. Our resolution was the design and implementation of a bespoke SFTP server, a custom Java Spring application, and a custom container for SCP to manage inbound and outbound file transactions. Unique approaches were required to support both incoming and outgoing file transfers. These unconventional solutions were necessitated by the unavailability of the AWS Transfer Family service, compelling us to take an alternate, yet effective, path to ensure seamless and secure file transactions.

Incoming SFTP

To support incoming files via SFTP, ASSIST designed a container that houses an SFTP server. This container is deployed in the ASSIST clusters inside the MCaaS ecosystem along with AWS Elastic File Service for persistent storage and S3 to deliver files to applications that require SFTP functionality.

To support incoming files via SCP, a container was designed to run a Cron task at a regular interval to pull files via SCP from the external system and transfer them to an S3 bucket where it will be made available from within ASSIST.

Outgoing SFTP

For outgoing file transfers via SFTP, an S3 bucket was used in conjunction with a custom Java Spring application to push files to their intended destination.

Problem Statement: File integrity must be maintained and inspected as files are stored and moved within ASSIST.

Solution: To maintain the integrity and safety of the files stored within ASSIST, the system integrates with the MCaaS ClamAV file scanning service. This component scrutinizes files and categorizes them as clean, infected, or encrypted. In the event a file is detected to harbor a virus or malware, access to the file is promptly inhibited, and the infected file can be permanently removed if necessary. The scanning process is initiated when a file is placed into a specified S3 bucket, triggered by an AWS Lambda function. Metadata about the scan result and timestamps are stored within the object's TagSet. This metadata is utilized by ASSIST to update our database and reflect the status of the uploaded file. Since migration, the Document Scanning process has scanned over 3.4 million files. It handles approximately 1,800 files a day.

Note: In addition to the information in this section, please refer to the whitepaper on ASSIST’s Hypercare Solution that was recently published in the Knowledge Center.

Problem Statement: Following major ASSIST releases, there is greater possibility for defect findings within the production environment; which - as a byproduct - often directly correlates to an increase in user call volumes and associated incidents, straining Help Desk resources and risking breach of SLA response times.

Solution: During the ASSIST 2.0 Convergence Release, a Hypercare strategy was created to allow more resources to focus on triaging issues stemming from the release. The primary objectives guiding the converged Hypercare solution were as follows:

1. Support end user change adoption
2. Assist in the execution of data fixes for migrated data
3. Assist in removal of workflow blockers
4. Fulfill emergency, out-of-cycle (OOC) releases required within the first 2-4 weeks of the release
5. Make necessary updates to the Help and FAQ documentation

The successful completion of the first Hypercare strategy for ASSIST 2.0 led the program to continue to use the approach after the 3.0 and 4.0 releases.

The Hypercare solution consists of the following activities intended to maintain low call response times and protect the existing Program Increment backlog.

• Reserve team capacity
• Individuals contributing to any work identified as post-release support are assigned to support the Hypercare effort. Capacity for those team members are reserved during planning activities to prevent over-commitment.
• Daily standup/status updates/triage
• To promote cross-team transparency and accountability, a daily Hypercare call is established with all team members to allow for reporting and discussion of new issues or updates on known issues as they are resolved.