About the team
USATII develops and deploys enterprise software for organizations managing complex, mission-critical operations. Our engineering teams retain technical responsibility for the systems they design, including their performance, availability, reliability, and continued operation in production.
Our Federal Infrastructure & Reliability team combines site reliability engineering, cloud infrastructure, software development, cybersecurity, and technical operations to maintain enterprise software in government environments.
DART is USATII's deployment and workforce management platform. It integrates personnel qualifications, training, incident staffing, workforce readiness, deployment management, and operational reporting within a unified application.
We are building a reliability engineering function capable of supporting DART during routine government operations and periods of elevated mission activity, including major incidents requiring rapid workforce mobilization and coordination.
About the role
USATII is seeking Site Reliability Engineers to operate, monitor, maintain, and continuously improve the reliability of DART's prospective federal production environment.
An initial area of focus is the potential modernization of FEMA's legacy Deployment Tracking System. DART's proposed federal deployment would support personnel readiness, incident staffing, deployment operations, and workforce accountability in a government-controlled cloud environment.
You will be responsible for maintaining the health of the application's production infrastructure, investigating operational incidents, improving observability, automating recovery procedures, and ensuring that the platform can continue supporting critical workflows during elevated demand or infrastructure failures.
DART's preliminary federal operating model proposes 99.9% monthly availability for agreed critical functions, a four-hour recovery time objective, and a recovery point objective of no more than 60 minutes. These targets require validation and customer approval before becoming production commitments.
You will help establish the engineering practices, operational safeguards, and recovery procedures needed to support those objectives.
This is a hands-on engineering role. You will investigate failures across application services, databases, cloud infrastructure, networking, external integrations, and background processing systems.
You will collaborate closely with GovCloud Platform Engineers, the Principal Software Architect, Forward Deployed Software Engineers, Federal Security & ATO Engineers, and application support personnel.
In this role, you will:
-
Own the operational reliability of DART's production application services, APIs, database infrastructure, background workers, integration services, and supporting cloud components.
-
Establish and maintain service-level indicators and objectives covering application availability, response latency, transaction success, error rates, processing delays, and critical operational workflows.
-
Develop centralized monitoring, alerting, operational dashboards, and health checks that provide actionable visibility into application and infrastructure behavior.
-
Implement monitoring for critical DART workflows, including personnel authentication, qualification decisions, incident staffing, deployment changes, notification processing, and external data synchronization.
-
Design alerting mechanisms that distinguish genuine operational incidents from routine changes in workload, avoiding excessive alert noise while maintaining visibility into critical failures.
-
Coordinate production incident response, including event detection, severity classification, technical investigation, mitigation, restoration, documentation, and follow-up engineering work.
-
Participate in a structured on-call rotation for critical incidents, working with additional trained engineers and approved support personnel to maintain primary and backup escalation coverage.
-
Establish incident response procedures aligned with the proposed DART support model, including priority-based acknowledgment, triage, escalation, and restoration targets.
-
Develop and maintain operational runbooks for common failures involving application services, database availability, authentication, integrations, job queues, notification delivery, and infrastructure resources.
-
Implement automation for recurring operational tasks, including service health verification, infrastructure recovery, deployment validation, log collection, and controlled maintenance procedures.
-
Design and test backup restoration, database recovery, service failover, and disaster recovery procedures against approved recovery objectives.
-
Conduct failure-injection and recovery exercises involving unavailable application instances, database disruptions, interrupted background workers, integration failures, and temporary network outages.
-
Monitor and optimize PostgreSQL performance, including query latency, connection utilization, locking behavior, storage growth, replication health, and resource exhaustion.
-
Investigate application performance degradation involving high concurrency, inefficient database operations, resource saturation, background processing delays, or external service dependencies.
-
Implement capacity planning and scaling procedures for large-scale workforce activity, incident staffing surges, bulk imports, reporting jobs, and high notification volumes.
-
Work with GovCloud Platform Engineers to establish reliable production deployment processes, including health-based rollout decisions, automated validation, safe rollback, and post-deployment monitoring.
-
Collaborate with cybersecurity personnel to integrate security monitoring, privileged-access controls, audit retention, vulnerability remediation, and incident evidence preservation into operating procedures.
-
Maintain operational change records, incident histories, root-cause analyses, recovery evidence, and service-performance reports.
-
Lead technical post-incident reviews, identify contributing system weaknesses, and implement corrective engineering work that reduces the likelihood or impact of recurring failures.
-
Work directly with application support personnel to investigate escalated user issues, identify systemic problems, and communicate expected restoration or mitigation steps.
-
Develop reusable observability, recovery, and production operations tooling that strengthens DART and USATII's broader enterprise software infrastructure.
You might thrive in this role if you:
-
Have 4+ years of professional experience in site reliability engineering, production infrastructure, DevOps, platform engineering, or backend operations.
-
Have operated production applications requiring high availability, controlled deployments, dependable monitoring, and effective incident response.
-
Have strong experience with AWS infrastructure, Linux systems, networking, containers, databases, and cloud-based application environments.
-
Understand observability systems, including metrics, logs, tracing, alerting, service-level indicators, and service-level objectives.
-
Have investigated and resolved serious production incidents involving unavailable services, database failures, network connectivity, degraded performance, or distributed system behavior.
-
Understand PostgreSQL operations, including performance monitoring, transactions, backups, replication, database connections, and recovery.
-
Have experience implementing automated deployment checks, infrastructure recovery tools, controlled rollouts, and rollback procedures.
-
Can develop operational automation using Python, TypeScript, Go, Bash, or comparable programming languages.
-
Understand distributed systems reliability, including timeouts, retries, idempotency, message processing, partial failures, and asynchronous workloads.
-
Have designed or participated in disaster recovery exercises and understand recovery time and recovery point objectives.
-
Can perform systematic root-cause analysis and translate incident findings into technical improvements.
-
Are comfortable participating in scheduled on-call duties and responding methodically to production incidents.
-
Can communicate operational incidents clearly to software engineers, government technical stakeholders, program managers, and customer support personnel.
-
Exercise sound engineering judgment under pressure and understand the importance of change control in sensitive production environments.
-
Hold a degree in computer science, software engineering, information technology, or a related discipline, or possess equivalent professional experience.
Experience with AWS GovCloud, federal production systems, FedRAMP-related infrastructure requirements, or mission-critical government software is particularly valuable.
Familiarity with NIST security controls, controlled production environments, and formal incident reporting procedures is beneficial.
About USATII
USATII is a U.S.-based enterprise software engineering company focused on designing, building, and deploying operational software and private AI systems.
We work directly with organizations to understand complex business processes, develop integrated technology solutions, and support those systems throughout their operational lifecycle.
Our approach emphasizes technical ownership, direct collaboration between engineers and customers, configurable enterprise architectures, and software that organizations can control and sustain over the long term.
USATII is privately held and operates from Rochester, New York, serving commercial and prospective public-sector customers.
USATII is an equal opportunity employer and considers qualified applicants without regard to legally protected characteristics. Reasonable accommodations are available during the application process.
Compensation
The proposed base salary for this position is $120,000 annually.
Additional compensation, employee benefits, and employment terms will be determined through an approved employment offer.
The position is expected to involve scheduled on-call responsibilities following production deployment. The applicable rotation, working arrangements, and compensation policies will be established before employment.
Full-time employment is contingent upon contract award, approved funding, and final staffing requirements.
Any customer-specific suitability, security-clearance, or work-location requirements will be communicated once established.