Technical Approach
See what tools and technologies were leveraged in the development of IAE to accommodate its unique business needs and ensure a smooth transition into the cloud.
Summary
IAE has followed an iterative approach of adding (building and integrating) the legacy functionality from 10 disparate systems into the modernized SAM environment. From a system engineering perspective, in order to consolidate 10 these systems into one, our strategy was to decompose and identify all the common business capabilities and refactor or rebuild the legacy systems from scratch one by one on a modern our FCS cloud platform - but there are cases where due to the size and complexity of the legacy system in addition to budget and timeline constraints we want to first integrate and then use the strangler pattern to incrementally transform the legacy monolith.
Search was the first business capability built in modernized SAM to enable users to retrieve data via an OpenSearch index cluster that is created for all users who search any of the public data sets the public search is a search that improves the user experience and consolidates views. This was made possible by allowing data, using AWS Kinesis, from legacy SAM and FPDS flows via the ETL layer to the OpenSearch cluster.
Databank capabilities were introduced for users to view data in reports or analyze results across the procurement data database using a single tool that provides various data fields as exports and in raw format.
The Workspace capability enabled authenticated users to create workflow and manage their work in a centralized, role-driven landing space that gives them a single place to keep track of what they should do next. The Search capability is built on OpenSearch, the data bank reports capability is built on MicroStrategy which utilizes Redshift Cluster as the data source. The workspace and data entry capabilities leverage multiple APIs like across various domains. The ISAM environment provides data entry for SAM registration information and all the sensitive data is stored there, which is a separate VPCaaS account and data is synchronized between the 2 accounts using Kineses and APIs. This approach serves as the foundation for modernized SAM and provides continuous improvements in smaller increments to ease user transition and minimize technical risk. IAE’s technical approach is based on industry established best practices and introduction of features to promote efficiency in the technical environment and user experience. Some of the steps taken as a part of this implementation approach are described in greater detail in the sections below
IAE has adopted SafeAgile practices and structured all cadence activities on a 8 week PI cycle. Each PI includes 3 sprints, 2 weeks each, and an IP sprint. The technical environment is supported by vendors (development teams and enabler teams) and government leads that serve as technical and product owners. JIRA is the system of record for capturing, managing and monitoring all activities at the program level. The IAE deployment cycles follows the following schedule
- Every Sprint development teams release well tested working software to the MINC (Integration Test) environment
- Every 02/IP Sprint the program releases well tested, working software to Alpha and production environments.
- Unit and Integration testing: built into each repository
- Web-UI Automated smoke testing: Validation and Smoke Testing after Alpha and Prod.: This includes Critical Test Scenarios For Smoke & Functional Testing SAM.gov
- Section 508 Issues – Development teams address the following:
- L1: Any AMP defect with a high severity level (7-10).
- L2: Any AMP defect with a medium severity level (4-6).
- Section 508 Testing Results: Use the In-Sprint 508 Testing Results (Confluence template) to store 508 findings from your testing resources per the 508 Resolution and Governance Process
- Performance Issues: After each major (02/IP) release the Performance Management team reviews the new changes in the Charlie env.
- Retrospectives are done (per Release) in meetings with Release Management team, TLs after each release.
IAE provides an automation testing framework that includes tools and approaches for the system at different phases of the development lifecycle. The SAM.gov application build pipeline has integrated many automated testing capabilities to validate the APIs, application UI/Front ends, OWASP dependency of open-source libraries, functional regression, and smoke testing.
- IAE has established a quality gate specification for micro-service and micro frontend application components. IAE's quality gates are built using this specification and enforced as part of the build pipeline. The detailed instructions are available here.
- For micro service or API testing, IAE recommends utilizing Postman for API validation during the development phase and utilizing the same scripts to validate the API automatically as part of the build pipeline.
- For application UI/Front ends, IAE recommends using the BDD based UI testing for functional, regression, and smoke testing. IAE has established a dedicated pipeline for the web front-end testing automation. The development team shall utilize the IAE's Web Automation Testing framework to utilize the build pipeline for automation testing and reuse the existing IAE specific use-case libraries.
- IAE's build pipeline automatically tests the open-source libraries in the test environment. The development team member can access the report directly in the Jenkins admin portal. The FAQ details are available here.
- AAccessibility testing: GSA IT recommends using AMP tool for the accessibility testing of the key public pages. The product management team is managing the list of public pages which need to be monitored and assessed every release. The AMP tool has been configured and scheduled to assess these pages automatically and publish the report.
Top Challenges
- Introducing or reintroducing people to federal award processes
- Setting expectations
- Conveying important messaging
- Preparing people for validation/registration
- Empower. Customers feel empowered to take action and make decisions.
- Customers are able to easily find answers to their questions
- Educate. Customers understand their actions and decisions.
- Reach customers at the earliest point possible
- Provide personalized information for different types of users
- Learn-as-you-go approach (help available in-context as users are working)
Build Trust. Customers trust the data and believe in the value of the website and it's functions.
Customer Touch Points
Our customers typically interact with the following touchpoints.
- SAM.gov. The SAM.gov website plus login.gov for sign-in.
- Customer Service. A customer service desk when they need assistance beyond self-help.
- Digital communications. Examples include mail notifications or blog posts (announcements).
- External sources. Organizations or people that send them to our website and sometimes provide guidance or instructions. Often, our customers first learn about SAM.gov from an external source that has a vested interest in their success.
- External systems. Many customers use their internal systems of record that send their data to SAM.gov via data services (e.g., web APIs).
Data migration is the process of moving data from one system to another. In the context of the extract/transform/load (ETL) process, any data migration (from legacy to modernized technical environment) will involve at least the transform and load steps. This means that extracted data needs to go through a series of functions in preparation, after which it can be loaded into a target location.
There are 2 types of ETL processes that are running in the IAE systems:
- Full Refresh ETL: This process migrates the full set of the required data needed to be in the Target system as per the business rules. This process is run once during the 1st Prod deployment and can be initiated again to resolve any fall of data if required.
- Incremental Refresh ETL: This migrates only the new added data into the target database from the Source database. This process runs after the Full Refresh ETL on a daily/weekly/monthly schedule.
CI/CD is the combined practice of continuous integration and continuous delivery or continuous deployment that bridges the gaps between development and operation activities and teams by enforcing automation in building, testing and deployment of applications. The SAM.gov application uses a Freedom pipeline framework for CI/CD that is built on Jenkins using pre-built framework libraries and pipeline templates. The governance for the build pipeline is managed through configuration files.
The pipeline is invoked based on the Jenkinsfile configuration for the monitored service when Jenkins detects a code change, pull request, merge, or other events. In order to support the pipeline, each service must have a Jenkinsfile called pipeline.config in its root. These Jenkinsfiles pull their pipeline configuration from the IAE Platform's GitHub organization that maintains pipelines for building Java and Node applications as Docker images. The IAE Platform's GitHub organization stores the Docket images in a Docker Trusted Registry, and deploys them as Docker Containers into the application environments. These Jenkinsfiles define pipelines as executable Groovy scripts (pipeline as code). IAE’s CI/CD automated pipeline capabilities are shown below.
In late 2021, key events converged that required the IAE program to re-visit the remaining system modernization work and the modernization end state. This effort includes a pivot to the 360 concepts. The 360 concept moves the IAE from a piecemeal approach to modernizing legacy systems and brings together a unified view of three core domain areas
- Acquisition 360: relates to the search, display and data capture of contract data through the contract lifecycle
- Search across all acquisition data
- Single view of all related acquisition data
- Single workspace to manage acquisitions
- Export reports to analyze across acquisition data
- Consolidated APIs for gathering and dissemination of acquisition management data
- Assistance 360: relates to all components of the financial assistance
- View all data related to an assistance programs in one view to include initial listing, links to opportunities, awards, sub-awards
- Reports to analyze across assistance data
- Consolidated APIs for gathering and dissemination of assistance management data
- Entity 360: relates to downstream legacy systems such as FAPIIS and CPARS
- Search across all entities
- Single view of all data related to an entity in one view: entity reporting, registration data, UEI data, etc.
- Contracting View for "complete package" to attach to their source selection documentation
- Single workspace for entity management and reporting
- Consolidated APIs
- Simplified user interaction and consolidated view across entity related information, assistance information and acquisition life-cycle, including:
- All acquisition steps/products in single view
- Streamlined single domain role and specific information permission access for managing acquisition data and viewing non-public data
- Consolidated data entry process across the life-cycle, with a flexible “path” for each acquisition, guiding the user to ensure regulations are being followed
- Simplified incorporation of regulatory and policy changes in logic and business rules with any changes/updates being accomplished in a single component service (elimination of redundancy)
- Improved policy compliance by placing system validations in line with user decision points throughout the acquisition process
- In addition to the above the transition to the 360 views also facilitate
- a better information domain search
- Feedback on single information domain display
- Refined concept for high level workflow of information domain data entry
- Single information domain roles
IAE is working towards a phased implementation of the 360 concept.