Senior Software Engineer-Spark

Last updated 5 days ago
Location:Sunnyvale, California
Job Type:Full Time

Azure Big Data Team is currently hiring for a Software Engineer to work on Spark core development in Mountain view, CA.

If you are a committer to the open source Spark project, we would love to talk with you at the earliest.

Azure Data team presents exciting challenges in technologies such as big-data, cloud, distributed computing, artificial intelligence, storage and open-source. We build reliable, highly scalable and highly performing distributed systems for data analytics on Azure, using popular open source technologies, such as Spark, Hadoop, Kafka, HBase and many others in a form that is easy for customers to use.

Data is being collected at an unprecedented scale, to support innovation and to optimize by learning from observations. International Data Corporation (IDC) estimates that data will grow from 0.8 to 163 ZBs (that’s zetta-bytes or 1021 bytes!) in this decade and that the Big Data business revenue will hit $260 billion by 2022. While everyone is building systems towards supporting this scale, the holy grail of Big Data is a system that is highly reliable and available and one that also delivers on performance without compromising the surrounding user experience.

Despite the amazing ideas we’ve had over the last few decades, Big Data still offers many wonderful opportunities for breaking the innovation barrier. However, doing this requires you to disassemble a modern analytical engine, figure out what every little bit does, and then put it back together into something quite a bit more awesome than the original, all without breaking user experiences and ensuring non-disruptive downtimes. If you love solving fun computing puzzles and are obsessed with why systems can’t be built to be better, cheaper and faster, the Spark Big Data team is looking for you!

Responsibilities

For this role, we are laser-focused on Spark. We will build on it, make contributions to it and make our offering the best the world has to offer. Our targets are both external and internal customers.

Our team has experience in building and running one of the world’s largest Big Data cluster which has more than 100s of thousands of machines, stored more than 10,000 PB data and run millions of jobs everyday across Bing Search, Ads, Office, Xbox, etc.

Knowledge in distributed systems, including high-performance distributed storage, databases or networking is a plus as Our code runs on hundreds of thousands of machines in multiple continents.

Qualifications

  • Bachelor’s or Master’s or Doctorate degree in Computer Science or Engineering or Mathematics or Physics

  • 5+ years of professional development experience in Java or C or C++ or Scala or C#
  • 5+ years of experience in developing and shipping complex and scalable software systems.

Prefered Qualifications:

  • Experience in database internals (e.g., B+ trees, query optimization) a major plus

  • Experience in modifying open source big data projects is an advantage

  • Open source committership
status in a popular Apache Spark project a major bonus

Additionally,

  • A self-driven individual will excel in this team.

  • Great design and problem-solving skills, great curiosity, and willingness to question.

  • Get excited by the challenge of hard technical problems.

  • Have a deep desire to work collaboratively, solve problems with groups, find win/win solutions and celebrate successes.

AZDAT #ENGGJOBS

Cloud Background Screening Requirement:

Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.

Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.