SanFranRecruiter Since 2001
the smart solution for San Francisco jobs

Data Engineer

Company: Granite Solutions Groupe
Location: San Francisco
Posted on: November 22, 2021

Job Description:

Job Category: Data Analysis Job Location: San Francisco, New York, Toronto, Omaha, Dallas Consultant, Data Analysis About Our Firm: Granite Solutions Groupe provides financial services decision makers with people and solutions that deliver. Our clients count on us to deliver the right talent at the right time to achieve critical business results. We leverage our industry knowledge and passion for client priorities to deliver human capital solutions. GSG was founded in 1998 as a consulting firm delivering project management and technology solutions for the financial services industry. GSG now has a global presence, with team members deployed across the U.S. at Fortune 1000 companies and high-performing Fintech firms. High-caliber consultants are the face of GSG at our client organizations and offices. We value the deep relationships we have with our consultants that enable us to place them in roles where their skills will have an immediate impact. The GSG consulting team is comprised of diverse, experienced, driven, and dynamic contributors who excel at getting things done. About the Opportunity: GSG is seeking a Cloud Data Engineer to join our dynamic team. The Data engineering team is entrusted to support the process, as enabler and force multiplier making data more useful and accessible, making it possible for consumers of data, such as analysts, data scientists and executives to inspect all of the data available reliably, quickly and securely. The Ideal candidate will spend time building and supporting multiple Data Pipelines to collect data from various sources, transform, clean, and aggregate to deliver data to various stake holders. You will work with cross functional Product management, advisory team, statistics, and data modelling experts. You are a person who loves to dig deeper to understand relationships among data and loves writing SQL queries or python programs to transform them to meaningful information. This is hands-on role and requires prior programming experience working with Data on cloud. Responsibilities

  • Design, develop, build, test and operationalize scalable data pipeline and data services.
    • Acquisition: Gather data requirements, research and design for data acquisition from different systems and various data sources.
    • --In particular, understanding existing systems, research and design new uses for existing data, explore ways to marry disparate datasets using automation.
    • Cleansing: Detecting and correcting errors and ensure data quality.
    • Aggregation, Disambiguation and De-duplication, Transformation.
    • Design for security and compliance.
    • Ensure scalability and efficiency.
    • Ensure reliability and fidelity.
    • Ensure flexibility and portability.
    • Integrate new data management technologies and software engineering tools into existing infrastructure
      • Mapping Storage system to business requirements.
      • Metadata management and data governance.
      • Data security.
      • Build a data platform, with the emphasis on self-service, sustainability and maintainability.
        • With clear separation of framework and application logic, abstract away details and automate as much as possible, so it is easy to set up and operate continuously with very little intervention. As a result, smart data pipelines are fast to build and deploy, fault tolerant, adaptive, and self-healing.
        • Operationalizing machine learning models - empower data scientist.
        • Collaborate with data scientists, researchers, data consumers, understand their need to develop algorithms including prototypes, predictive models and proof of concepts.
        • Take ownership end-to-end automation, understand data requirements from business analyst and product management, define mapping and implement data pipelines.
        • Implement Airflow or other orchestration programs and execution programs in Python, Jupyter Notebooks and/or Docker containers to be launched on AWS EKS platform.
        • Troubleshoot, support and enhance data pipelines jobs already built by other engineers and optimize them for better performance and deploy them on AWS cloud stack.
        • Collect data using AWS services like Lambda, SQL and Python programs from various sources like web services, sftp, AWS S3, RDBMS like Postgres/Redshift/MS SQL server.
        • Manage a portfolio of Data pipelines and support them, document them for Data operations/client support staff to maintain.
        • Ensure that processed data meets our high-quality standards. Requirements:
          • 4-6 years of experience working in data engineering using python on Cloud platform like AWS.
          • Proficient in programming in Python. With experience of packages like pandas, numpy, boto, s3fs, flask, fastapi, pyarrorw, pyspark etc.
          • Experience in SQL working with DBMS platforms like Postgres/MS SQL Server/ Redshift.
          • Experience or exposure working with Technologies like Apache Airflow, Kubeflow, EKS, Docker is plus.
          • Experience with Git/Bitbucket source control, Git Actions/circle-ci and shell scripting.
          • Experience working with AWS services: S3, Lambda, RDS, AWS SDK.
          • Able to communicate with business, data analysts and product management teams to understand their needs, change requests and issues to remediate.
          • Strong problem-solving skills to understand complex structured and/or unstructured datasets and their relationships to find issues and provide programmatic solution.
          • Prior Experience building and optimizing data pipelines, architectures and data sets is plus.
          • Collaborative attitude to comfortably work with cross-functional teams in a dynamic environment.
          • We need a candidate comfortable working with collaborative technologies like Jira, Confluence, Slack, Zoom, Outlook, GitHub etc. in remote work environment.
          • Creative Thinker - Data Engineers connect the dots and see the big picture to evaluate how data insights affect the business.
          • Collaborative - Ability to collaborate with both internal technology and business teams as well as integration partners and customers.
          • Curious - Data Engineers demonstrate a track record of learning and deep-diving into complex systems.
          • Statistical analysis and modeling.
          • Python, SQL, Java, Scala.
          • Big data processing and or streaming analytics - Spark, Kafka.
          • Model orchestration and workflow - Airflow.
          • Cloud Technology and related technical stack- AWS, GCP, Azure.
          • Experience with machine learning algorithms - big plus.
          • Data-driven approach to problem solving.
          • The ability to visualize and communicate complex concepts.
          • Strong sense of ownership. Employee Benefits: GSG is excited to offer its consultants excellent compensation and a robust option of benefits which include comprehensive medical, dental, vision and prescription coverage. You will receive company-paid life insurance. In addition, we hope you will take advantage of eligible commuter benefits-like paying for work-related public transit and parking with pre-tax dollars. Save for your retirement with pre-tax contributions that go directly into your 401K. We also care about your pets with several options for pet insurance, discount card and prescriptions. GSG's Commitment to Diversity: As a diverse-owned business, GSG is committed to creating a diverse workforce, and we are proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. Yes, I'd like to occasionally receive emails from Granite Solutions Groupe. Privacy Overview

Keywords: Granite Solutions Groupe, San Francisco , Data Engineer, Engineering , San Francisco, California

Click here to apply!

Didn't find what you're looking for? Search again!

I'm looking for
in category

Log In or Create An Account

Get the latest California jobs by following @recnetCA on Twitter!

San Francisco RSS job feeds