Technical Approach
See what tools and technologies were leveraged in the development of ASSIST to accommodate its unique business needs and ensure a smooth transition into the cloud.
Technical Stack
Due to the complex history of how ASSIST has evolved since its inception, there are various technical architectures that have been implemented across the back-end services and front-end UI for the applications deployed within ASSIST. Each of these distinct technical stacks is continuously evaluated and updated as part of our ongoing Platform and Infrastructure Maintenance process. As business functionality and applications are modernized, legacy implementations are retired from ASSIST in favor of the modern technical stacks within ASSIST.
The various technical architectures in use throughout ASSIST are detailed below. The UI and Services portions of each application are separated out where appropriate. In some cases, the UI and Services portions of each application span one deployable artifact (for applications that implement Spring MVC, for example) where others are shown as distinct components (those with an Angular UI, for example). The infrastructure that is in use in each of the technical architectures below is shown in gray (JBoss, DataDog agents/RUM, etc). In each of the applications, the major Maven dependencies are listed (listing all dependencies would not be practical). Each of the applications utilizes AWS storage services (Aurora PostgreSQL and S3) provided by MCaaS.
Note that there are variations related to the different architectures that are in place across the service and UI layers in the applications detailed below. Those variations are directly related to the Spring components that have been implemented in the ASSIST applications. For example, "API Controllers" are in use throughout the Spring Boot applications and applications that utilize modern versions of Spring Framework whereas "API Services" and "Controllers" are used in applications that were created with older versions of Spring.
Working within MCaaS
Problem Statement: The ASSIST team had to carefully account for technical assets and development tools to ensure that all necessary assets and tools were available and functional after migration into the FCS cloud ecosystem.
Solution: ASSIST is a tenant within the FCS MCaaS ecosystem and inherits MCaaS boundaries. GSA FCS architecture consists of several AWS Virtual Private Cloud (VPC) networks.It is logically isolated from other virtual networks within AWS cloud. GSA FCS MCaaS tenants have full control of their data and applications within their secure AWS VPC environment.
Within the MCaaS system boundaries, the ASSIST system assets are supported in VPC subnets. Components of the ASSIST environment are broken down into the following groups of asset types/categories.
| Asset Type / Category | Vendor / Product | Provider | Description of function or service provided |
|---|---|---|---|
| JavaScript/front end UI framework | Angular | ASSIST team | Industry-proven, UI/UX frameworks for rapid development of mobile-ready, cross-browser, component-based applications (also leveraging sass for CSS ext. and NPM Webpack for code packaging and obfuscation). |
| Java ORM framework | Hibernate | ASSIST team | Java Persistence APIs (Data Services) and JP/HSQL via annotations on POJO Entities and Repo classes for CRUD and complex operations. |
| JavaScript framework | Bootstrap | ASSIST team | A JavaScript framework that provides common UI elements in use throughout the ASSIST front-end. |
| JavaScript framework | jQuery | ASSIST team | A JavaScript framework that provides common functionality in use throughout the ASSIST front-end. |
| JavaScript framework | Data Tables | ASSIST team | A JavaScript framework that provides grid layout functionality in use throughout the ASSIST front-end. Includes Editor extension of the Data tables foundation. |
| Authorization and Authentication | ForgeRock AM and DS | ASSIST team customized ForgeRock AM | Manages authentication and authorization for ASSIST users. Also manages the hand off for ASSIST to GSA Single Sign On solution (SecureAuth). |
| Java framework | Spring | Build into BaseImage by ASSIST team | Spring framework provides comprehensive infrastructure support for developing Java applications |
| Java framework | Spring Boot | Build into BaseImage by ASSIST team | The Spring boot provides a comprehensive programming and configuration model for modern Java-based enterprise applications - on any kind of deployment platform. It eliminates the boilerplate configurations required for setting up a Spring application. |
| Application Server | RedHat JBoss Application Server (EAP/AS) | Build into BaseImage by ASSIST team | Java container to deploy custom Java Applications and use of Java scheduler to execute batch jobs according to a schedule frequency. |
| Java framework | Java | Build into BaseImage by ASSIST team | The SDK/APIs that are essential to the underlying service applications that support the ASSIST application. |
| UBI | DoD Ironbank | ASSIST team | Base OS image (UBI 7.x and 8.x) for ASSIST containers |
| nginx | F5 Inc. | Build into BaseImage by ASSIST team | Light-weight service to serve static content including ASSIST Angular front-ends and CM Static Content. |
| SSH File Transfer Protocol (SFTP) Server | ASSIST | Custom application by ASSIST team | ASSIST External File Transfer (EFT) Services enable the retrieval and transmission of files between ASSIST and external systems that are required for proper system operation. EFT Services include a Secure File Transfer Protocol (SFTP) server, fielded as a Kubernetes deployment, an outbound jobs service, as well as an outbound Kubernetes cron job. |
| Simple Mail Transfer Protocol (SMTP) Server (GSA Google Relay) |
GSA | GSA | Used to capture outgoing email, in Staging only to perform quality control of business logic in application(s) and verify email events happen as appropriate. Prod and Preprod, use GSA SMTP gateway. |
| Database | Postgres Aurora Serverless | MCaaS | Database server to store data in, execute stored procedures according to a schedule frequency. |
| Monitoring, Alerting and log management | Datadog | MCaaS | Datadog is the official Observability tool offered and managed by FCS MCaaS. |
| Version control / Source code repository | GitHub | MCaaS | MCaaS provides a Version Control system used as a code hosting platform for version control and collaboration by MCaaS. |
| CI/CD tool | Jenkins | MCaaS | MCaaS provided Continuous Integration to build code and deploy new versions of the software. |
| Security Scanning Tool | Anchore | MCaaS | GSA provided Container Security Services that perform CVE and Compliance scans on the container images during build, deployment, and runtime. |
| Security Scanning Tool | StackRox | MCaaS | StackRox is Kubernetes-native security tool |
| CI/CD tool | Helm | MCaaS | COTS Product used to do “Continuous (Automated) Deployments”. |
| Storage | Amazon | MCaaS | Secured Storage that ASSIST uses AWS S3 to support processes for both Help Document storage within FCS and Data Transfer supporting the SFTP servers. Provisioned by the FCS team. |
| Anti-Virus Scanning | ClamAV Virus Scanning | MCaaS | The ClamAV virus scanning solution supports performing anti-virus scans on all ASSIST Document Management attachments stored in the ASSIST S3 bucket. |
| Active Directory in non-production environments | OpenLDAP | ASSIST team maintains it in non-production environments | Stand-in for active directory in ASSIST’s non-production environments |
| Front end UI for active directory in non-production environment | phpLDAPAdmin | ASSIST team maintains it in non-production environments | Front end UI for the stand-in active directory in ASSIST’s non-production environments |
Following is the list of GSA Enterprise Architecture Analytics and Reporting (GEAR) approved tools and software utilized for the development process on the Govt. Furnished Equipment (GFE) by the ASSIST team.
| GEAR Approved Tools & Software | |
|---|---|
| Software | Description |
| PuTTY 0.78 | SSH tool - for generating SSH keys for access to the MCaaS jump box |
| PGAdmin | Database client |
| SQL Developer | Database client |
| Amazon Corretto | Amazon distribution of OpenJDK. |
| Apache jMeter | Performance/load testing tool primarily for API testing. |
| Apache Maven | Java build tool used for most ASSIST software. |
| DBeaver | Database client |
| Eclipse | Integrated development environment |
| Filezilla | File transfer tool for testing/verification of SFTP endpoints. |
| Git for Windows | Git client for Windows. |
| GitHub Desktop | Visual GitHub client |
| Helm.exe | Package manager for Kubernetes |
| IntelliJ IDEA Community | Integrated development environment |
| MS Visual Studio Code x64 | Integrated development environment |
| Node.js | JavaScript runtime environment. Includes build tools (npm) for Angular projects |
| Postman | API testing utility |
| Spring Tool Suite | Integrated development environment based off of Eclipse but targeted to Spring |
| Jira | Project management and issue tracking |
| Confluence | Documentation collaboration tool |
Note: In addition to the information in this section, please refer to the whitepaper on ASSIST’s Migration to Postgres that was recently published in the Knowledge Center.
Problem Statement: ASSIST needed to ensure a smooth transition to the FCS ecosystem that minimized licensed costs and streamlined the environment creation process.
Solution: ASSIST's underlying data store and operations have historically been supported by an Oracle RDBMS database. ASSIST has operated using an Oracle database as its primary data store both on-premise and upon migration to the FCS/MCaaS cloud ecosystem. To further modernize ASSIST and leverage cloud offerings, streamline the environment creation process, and minimize license costs, the ASSIST Team has initiated a process to migrate from AWS RDS Oracle. As part of this effort, The Team conducted research and developed white papers to assess features, constraints, and risks associated with various platforms which led to the choice of PostgreSQL as the target platform. An additional analysis was performed to evaluate the options within the MCaaS environment. These activities resulted in AWS RDS Aurora PostgreSQL being chosen as the target database platform for ASSIST.
The Oracle to Postgres migration process that the ASSIST Team is implementing is detailed below: Both SCT and DMS have been employed to perform DML and DDL migrations, respectively. The SCT process is used to generate Pre-DMS DDL scripts that populate the schema of the new Postgres database. Once those scripts are run, DMS is employed to migrate data from Oracle to Aurora PostgreSQL. Multiple checkpoints have been put in place to allow for modifications of the SCT, DMS, or SQL Scripts that are part of the migration process. With fine tuning and iterative changes in place, the ASSIST Team is finalizing the migration procedures and using them to replicate the Oracle to Postgres migration in multiple environments. Once final validation is performed, the process will be replicated across additional environments concluding with the production migration.
Lessons Learned: During the migration process, lessons learned were documented for future reference. Major lessons learned are shown below.
Problem Statement: ASSIST needs to maintain middleware to run Java applications in the cloud.
Solution: For over a decade now ASSIST has leveraged RedHat's JBoss Enterprise Application Platform as middleware for running our java applications, culminating in JBoss EAP 7.4. Previously on-prem, several of these application servers were installed directly on each VM to enable traditional system administration and operations tasks. With the aid of aliases and the flexible configuration capabilities of JBoss, the DevOps team was able to function effectively in daily maintenance of the system.
With our move to a managed cloud and away from monolith applications, we containerized our JBoss solution, working off a RHEL 8 + Java 11 base image layer. The process of layering the ASSIST images is shown below.
Implementation
Each JBoss container runs in standalone mode listening on port 8080, as part of the overall multi-container deployment for each ASSIST application. It builds on top of the Java 11 image and in turn the UBI 8 base image, which establishes the non-root user 'assist' at the home directory /jboss where all further container operations are performed. At runtime, the docker entrypoint script simply executes the standalone.sh start script included with JBoss, and upon receiving a signal to terminate the instance, graceful shutdown is initiated via the JBoss CLI.
Boilerplate configuration for each instance is established in the standalone xml file, which is mounted to the deployment by way of configmap. All other customizations and configurations are applied via environment variables, JVM runtime options, and our override properties mechanism (also mounted from a configmap) in Flux. Sensitive values are stored encrypted in the configuration, and are accessed by the init container 'cryptkeeper' to decrypt at runtime. Cryptkeeper is essentially a containerized version of the AssistStringEncryption routine already used by our applications that performs a decrypt operation once against all encrypted properties then destroys itself.
At minimum, the kubernetes resources packaged for each JBoss instances' helm chart include:
- Deployment including the pod that consists of the cryptkeeper initContainer, in addition to both the JBoss 7 container and the istio-proxy sidecar container.
- Service exposing port 80 within the cluster, mapped from the container port 8080.
- Virtual Service at port 80 allowing communication from the istio cluster gateway.
- ConfigMap containing properties files' content, and the jboss.xml. Encrypted configuration values are mounted to a separate volume to be fed as inputs to cryptkeeper.
- HorizontalPodAutoScaler to scale the amount of replicas in the deployment based on CPU + memory utilization.
Hardening
On top of the security benefits already offered by using Ironbank's UBI8 OS image, considerable effort has been invested into ensuring JBoss EAP 7.4 is running with the latest patch offering from RedHat to address the numerous potentially vulnerable components bundled with the archive. The base JBoss 7.4 zip bundled with the latest patch zip is downloaded from s3, the archives are extracted, the patch is applied via the JBoss CLI at build time, then all unused files are removed from the image. For any vulnerabilities not covered by this method, manual patches are created and applied during the process of building the JBoss image.
Problem Statement: ASSIST requires the ability to both push and pull files to various external interfaces via SFTP or SCP.
Solution: Certain services such as SFTP are more limited for security reasons in the MCaaS environment. Because MCaaS does not support AWS Transfer Family yet, the ASSIST Team created custom solutions to work around limitations related to SCP and SFTP. Our resolution was the design and implementation of a bespoke SFTP server, a custom Java Spring application, and a custom container for SCP to manage inbound and outbound file transactions. Unique approaches were required to support both incoming and outgoing file transfers. These unconventional solutions were necessitated by the unavailability of the AWS Transfer Family service, compelling us to take an alternate, yet effective, path to ensure seamless and secure file transactions.
Incoming SFTP
To support incoming files via SFTP, ASSIST designed a container that houses an SFTP server. This container is deployed in the ASSIST clusters inside the MCaaS ecosystem along with AWS Elastic File Service for persistent storage and S3 to deliver files to applications that require SFTP functionality.
To support incoming files via SCP, a container was designed to run a Cron task at a regular interval to pull files via SCP from the external system and transfer them to an S3 bucket where it will be made available from within ASSIST.
Outgoing SFTP
For outgoing file transfers via SFTP, an S3 bucket was used in conjunction with a custom Java Spring application to push files to their intended destination.
Problem Statement: File integrity must be maintained and inspected as files are stored and moved within ASSIST.
Solution: To maintain the integrity and safety of the files stored within ASSIST, the system integrates with the MCaaS ClamAV file scanning service. This component scrutinizes files and categorizes them as clean, infected, or encrypted. In the event a file is detected to harbor a virus or malware, access to the file is promptly inhibited, and the infected file can be permanently removed if necessary. The scanning process is initiated when a file is placed into a specified S3 bucket, triggered by an AWS Lambda function. Metadata about the scan result and timestamps are stored within the object's TagSet. This metadata is utilized by ASSIST to update our database and reflect the status of the uploaded file. Since migration, the Document Scanning process has scanned over 3.4 million files. It handles approximately 1,800 files a day.
Note: In addition to the information in this section, please refer to the whitepaper on ASSIST’s Hypercare Solution that was recently published in the Knowledge Center.
Problem Statement: Following major ASSIST releases, there is greater possibility for defect findings within the production environment; which - as a byproduct - often directly correlates to an increase in user call volumes and associated incidents, straining Help Desk resources and risking breach of SLA response times.
Solution: During the ASSIST 2.0 Convergence Release, a Hypercare strategy was created to allow more resources to focus on triaging issues stemming from the release. The primary objectives guiding the converged Hypercare solution were as follows:
- Support end user change adoption
- Assist in the execution of data fixes for migrated data
- Assist in removal of workflow blockers
- Fulfill emergency, out-of-cycle (OOC) releases required within the first 2-4 weeks of the release
- Make necessary updates to the Help and FAQ documentation
The successful completion of the first Hypercare strategy for ASSIST 2.0 led the program to continue to use the approach after the 3.0 and 4.0 releases.
The Hypercare solution consists of the following activities intended to maintain low call response times and protect the existing Program Increment backlog.
- Reserve team capacity
- Individuals contributing to any work identified as post-release support are assigned to support the Hypercare effort. Capacity for those team members are reserved during planning activities to prevent over-commitment.
- Daily standup/status updates/triage
- To promote cross-team transparency and accountability, a daily Hypercare call is established with all team members to allow for reporting and discussion of new issues or updates on known issues as they are resolved.