Data Engineer | Product

Per Google, data engineers “design, build, operationalize, secure, and monitor data processing systems with a particular emphasis on security and compliance; scalability and efficiency; reliability and fidelity; and flexibility and portability.”

Responsibilities

• Creates and maintains data pipelines
• Creates and maintains frameworks for data provenance
• Integrates external data stores, data lakes, data analysis frameworks and application database(s)
• Architects and maintains the database(s) used for data analysis
• Contributes to the design and maintenance of the application database(s)
• Wearing multiple hats as necessary

Skills & Competencies

In addition to shared-competencies for “managers of one” and Software Engineer Competencies Data Engineers are expected to demonstrate:

Technical Polyglot

• Understanding of the relative strengths of various databases, data warehouses/data lakes, management tools, etc. acquired through experience.
• Data engineering programming languages (R, Python, SQL, etc.)

Database and Data Warehousing Administration

• Set-up and maintenance of secure cloud environment
• Excellent communication skills; can bridge the gap between engineering lingo and human language
• Good schema design experience and understanding
• Strong command of:
  • Business intelligence, query, and reporting tools
  • Database design for read-only access
  • Data warehousing design issues such as star schema
  • Data warehousing technologies
  • Data transformation and conversion
  • Data quality issues
  • Data formats for loading and unloading of data
  • Data lakes