We are looking for a talented Senior Data Engineer to be part of our Data team. You will work alongside a growing multidisciplinary team of talented research scientists and machine learning engineers to improve and scale the efficiency within our models. You will contribute in building the data infrastructure that manages our data and powers the training of all Stability AI models. The position is based in Germany, remote.
Responsibilities:
Clean, normalize, and preprocess data in a scalable, parallelizable way to prepare it for ingestion into our machine learning model training pipelines while ensuring data quality
Design, implement, and maintain scalable data infrastructure for generative AI
Develop tools to search and serve the data at scale
Collaborate with cross-functional teams to understand and meet their data requirements
Develop and manage data processing pipelines to support machine learning teams
Maintain and improve data quality and integrity across various databases and data stores
Manage and organize large-scale unstructured data, including image, text, audio, video and 3D
Qualifications:
Proven background within large-scale distributed workloads
Experience with large-scale data loading for machine learning training runs
Experience with cloud storage and file systems. AWS (S3) is strongly preferred, but open to other cloud platforms
Experience with Python
Experience with large-scale data processing and software development for unstructured data
Expertise in database, data lake, and data warehouse technologies (Redshift, BigQuery, Snowflake)
Experience working with machine learning projects and ideally some deep learning / computer vision knowledge
Good teamwork and communication skills based on experience working with a distributed international team with timezone and cultural differences
Excellent communication skills to effectively collaborate with users, solve issues, and provide guidance
Attention to detail and the ability to document processes and solutions effectively
Equal Employment Opportunity:
We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses.
DanaRemoters