Big Data Engineer

 

As a Big Data Engineer at 9GAG you will help evaluate and build our data platform to sift and process terabytes of behavioral, textual, visual and unstructured data from billions of interactions with our global user base. As a member of the data team, you’ll have immense impact - literally shaping the data with which the company makes innumerable business decisions.

 

We’re looking for a detailed oriented Big Data Engineer who understands the value and impact of good data. You’ll help shape the entire lifecycle of data: from building robust scalable data pipelines in our backend, building dashboards to enable better decision making, to helping to implement a machine learning model serving real time personalized recommendations to our tens of million of users.

 

Building 9GAG’s data infrastructure is an enormous challenge. You’ll have to balance the need to move fast today while continuing to build for tomorrow. You understand the tradeoffs between different open source Big Data technologies and aren’t afraid to build it yourself if needed.

Responsibilities

  • Build end-to-end, high-performance, robust and scalable data processing systems.

  • Collaborate with others on high impact data projects.

  • Build tools to help make better business decisions.

  • Productionize machine learning models for serving real-time predictions and recommendations.

  • Identify the gaps and build the tools to increase the speed of analysis. For example, tools like automated AB testing or anomaly detection.

Requirement

  • Live our values - H2O: Hustle to go all-in to achieve the goal, work hard and work smart; Humble to embrace the better ideas of others; Open to welcome differences and possibilities.

  • Programming experience, preferably in PHP, Python or Scala.

  • Experience with designing and maintaining complex data pipelines.

  • Ability to work with varied forms of data infrastructure, including: RDBMS (PostgreSQL, MySQL); NoSQL datastores (Cassandra, ElasticSearch, Redis); MapReduce (Hadoop, Spark); Logging/messaging systems (Kafka, Scribe, Flume, RabbitMQ).

  • Experience with Cloud Computing (e.g. AWS, GCP).

  • Passion for building tools and automating everything.

  • Interest in machine learning is a plus.

  • You can teach us something new! We love to hire people who are way more knowledgeable than us in some area and can be good teachers and mentors.

  • No haters.

Desire

  • Contributions to open source projects.

  • Active user of 9GAG.