The Data Engineer will manage and manipulate data and data flows for both existing and new systems. Additionally, they will provide support in the areas of data extraction, transformation and load (ETL), data mapping, analytics, operations, databases, and maintenance of data and associated systems. As a member of the team, candidate will work in a multi-tasking, quick-paced, dynamic, process- improvement environment that requires experience with the principles of large-scale (terabytes) database development, large-scale file manipulation, data modeling, data mapping, data testing, data quality, and documentation preparation.
Provide support in the areas of data extraction, transformation and load (ETL), data mapping, analytics, operations, databases, and maintenance of data and associated systems.
Develop and manage complex data flows, or makes significant enhancements to existing pipelines.
Troubleshoot complex problems and provide customer support for the ETL process.
Advise hardware engineers on machine characteristics that affect software systems, such as storage capacity, processing speed, and input/output requirements.
Conduct investigations and tests of considerable complexity.
Provide ongoing maintenance, support, and enhancements in existing systems and platforms
Collaborate cross-functionally with software engineers, data scientists, analysts, project managers and other engineering groups.
Research emerging technologies to determine impact on application execution.
Communicate clearly and effectively with teammates, customers, and external partners.
Prepare written and verbal communications on analyses, findings, and project progress.
Write and update technical documentation such as user manuals, system documentation, training materials, processes and procedures.
Provide recommendations for continuous improvement.
EDUCATION AND EXPERIENCE
Bachelor's degree in Computer Science, Information Technology, or other related technical discipline, or equivalent combination of education, technical certifications, training, and work/military experience.
5+ years of experience in working with Big Data technology and using methods to ingest, process, clean, and analyze big data.
5+ years of implementation experience in Hadoop technologies and have worked in multiple Hadoop distributions like AWS and Cloudera.
Demonstrated experience with large data stores such as data lakes, data warehouses, and databases.
Demonstrated experience working with large and complex data sets as well as experience analyzing large volumes of data.
Demonstrated hands-on experience working with relational databases such as Postgres, MySQL, and Microsoft SQL Server.
Demonstrated experience with data ingest tools (e.g., Sqoop, Kafka, and Spark Streaming)
Intensive skill in scripting using Bash, Python and Shell.
Demonstrated experience in creating indexes and working with indexes using SOLR on Hadoop Distributed Platform.
Demonstrated experience in Kibana and Elasticsearch.
Demonstrated experience in ETL, data integration, and migration.
Experience with different file formats like ORC, Parquet, AVRO, JSON.
Experience in data cleansing scripts like Spark and MapReduce.
Orchestrated multiple Hadoop application jobs using Oozie.
Experienced in working with various IDE’s such as Eclipse, VS Code, and PyCharm.
Experience in system management expertise with monitoring, disaster recovery, backup, automated testing, automated schema migration, and continuous deployment.
Exposure in Infrastructure as Code languages like Chef, Puppet, Ansible, Cloudformation.
Experience in Agile software methodologies.
About Our Work
We are GDIT. The people supporting some of the most complex government, defense, and intelligence projects across the country. We deliver. Bringing the expertise needed to understand and advance critical missions. We transform. Shifting the ways clients invest in, integrate, and innovate technology solutions. We ensure today is safe and tomorrow is smarter. We are there. On the ground, beside our clients, in the lab, and everywhere in between. Offering the technology transformations, strategy, and mission services needed to get the job done.
GDIT does not have a vaccination mandate applicable to all employees. To protect the health and safety of its employees and to comply with customer requirements, however, GDIT may require employees in certain positions to be fully vaccinated against COVID-19. Vaccination requirements will depend on the status of the federal contractor mandate and customer site requirements.
GDIT is an Equal Opportunity/Affirmative Action employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status, or any other protected class.