Big Data QA Engineer
beaconhill

Job Info

---

This data/documentation software company has a need for a Big Data QA engineering professional who can assist with important test automation and QA tasks associated with mission critical projects.

Responsibilities

QA engineer will perform the following tasks:

• QA test automation
• Define a test automation strategy for new data pipelines or modify existing strategy to expand QA coverage of an existing pipeline
• Create data pipeline, specific input datasets, and expected datasets to implement QA automation
• Modify existing input datasets and expected datasets when business requirements for existing data pipelines change
• Create or modify Python scripts for triggering data pipeline specific QA tests and validating against the expected outputs

• Integrate with Jenkins CI/CD automation to run nightly QA tests automatically

• Perform manual testing where automation is not feasible, or QA tests need to be run on ad-hoc basis

Requirements:

The role requires an engineer who is data savvy and has an overall system quality mindset.

• Advanced Python scripting skills are must. For example, ability to work with ease using various data formats such as CSV, JSON, XML in Python and create modular/re-usable code. Should be able to write OO code and understand list comprehensions in Python.
• Comfortable working in Linux environment with bash, grep, awk, ssh, xargs etc.
• Critically think about corner cases in data pipelines and create test cases to simulate those conditions. Should understand the significance of test coverage. Should understand fault injection.
• Good understanding of basics of probability and statistics (working with large data). For example, ability to create representative smaller samples from larger datasets by applying various selection methods. Should know what a Normal distribution is and how to prove the randomness of a data sample. Able to apply descriptive statistics for data validation, troubleshooting.
• Understands the problems associated with processing large datasets (10's of TB) and is conceptually familiar with technologies available to solve those problems.

• Not expected to know Hadoop or Spark but would-be a plus.

Find Us on Facebook!
Follow Us on Twitter!

Beacon Hill is an Equal Opportunity Employer that values the strength diversity brings to the workplace. Individuals with Disabilities and Protected Veterans are encouraged to apply.

Company Profile:

Beacon Hill Technologies, a premier National Information Technology Staffing Group, provides world class technology talent across all industries utilizing a complete suite of staffing services. Beacon Hill Technologies' dedicated team of recruiting and staffing experts consistently delivers quality IT professionals to solve our customers' technical and business needs.

Beacon Hill Technologies covers a broad spectrum of IT positions, including Project Management and Business Analysis, Programming/Development, Database, Infrastructure, Quality Assurance, Production/Support and ERP roles.

Learn more about Beacon Hill Staffing Group and our specialty divisions, Beacon Hill Associates, Beacon Hill Financial, Beacon Hill HR, Beacon Hill Legal, Beacon Hill Life Sciences and Beacon Hill Technologies by visiting www.beaconhillstaffing.com.

We look forward to working with you.

Beacon Hill. Employing the Future™

This job has expired.