AWS re:Invent 2015

Dive deep into specific, common use cases for enterprise customers while stepping through the process of building a cloud and IT transformation strategy leveraging the AWS Cloud Adoption Framework. We will build a prescriptive roadmap for a cloud journey leveraging best practices, common techniques, and real-world examples from other AWS successes.

It's difficult to find off-the-shelf, open-source solutions for creating lean, simple, and language-agnostic data-processing pipelines for machine learning (ML). This session shows you how to use Amazon S3, Docker, Amazon EC2, Auto Scaling, and a number of open source libraries as cornerstones to build one. We also share our experience creating elastically scalable and robust ML infrastructure leveraging the Spot instance market.

Customer demand and competitive landscapes mean its not okay to move fast and break things. Whether you are following Agile, Continuous Integration (CI) or Continuous Deployment (CD), you cant afford to limit your releases, skip quality assurance (QA) testing, or let your customers beta-test your products. You need an on-demand QA platform that moves at the speed of development. Come learn how Rainforest leverages Mechanical Turk to build its network of 50,000 QA testers around the world and deliver on-demand QA through its API. We are able to deliver QA tests in 30 minutes that dont slow down development.

We are collecting tons of sensor data from billions of devices. How do you get the value from your IoT data sources? In this session, we will explore different strategies for collecting and ingesting data, understanding its frequency, and leveraging the potential of the cloud to analyze and predict trends and behavior to get most out of your deployed devices.

In this session, we show you how to use Amazon Route 53 to consolidate your DNS data and manage it centrally. Learn how to use Amazon Route 53 for public DNS and for private DNS in VPC, and also learn how to combine Amazon Route 53 private DNS with your own DNS infrastructure.

An end-to-end, over-the-top (OTT) video system is built of many interdependent architectural tiers, ranging from content preparation, content delivery, and subscriber and entitlement management, to analytics and recommendations. This talk will provide a detailed exploration of how to architect a media platform that allows for growth, scalability, security, and business changes at each tier, based on real-world experiences delivering over 100 Gbps of concurrent video traffic with 24/7/365 linear TV requirements. Finally, learn how Verizon uses AWS, including Amazon Redshift and Amazon Elastic MapReduce, to power its recently launched mobile video application Go90. Using a mixture of AWS services and native applications, we address the following scaling challenges: Content ingest, preparation, and distribution Operation of a 24x7x365 Linear OTT Playout Platform Common pitfalls with transcode and content preperation Multi-DRM and packaging to allow cross platform playback Efficient delivery and multi-CDN methodology to allow for a perfect experience globally Kinesis as a dual purpose system for both analytics and concurrency access management Integration with Machine Learning for an adaptive recommendation system, with real time integration between content history and advertising data User, entitlement, and content management General best practices for Cloud Architectures and their integration with Amazon Web Services; Infrastructure as Code, Disposable and immutable infrastructure, code deployment & release management, DevOps and Microservices Architectures This session is great for architects, engineers, and CTOs within media and entertainment or others simply interested in decoupled architectures.

Whether you are an automobile manufacturer, dispatch service, or drone manufacturer, your world is getting smarter. AWS provides all the building blocks required to build next-generation, Internet-connected devices that are in motion. In this session, we will share some best practices and architectural patterns device manufacturers can leverage for any connected device that is in motion to reliably and securely connect to AWS.

As data volumes grow, managing and scaling data pipelines for ETL and batch processing can be daunting. With more than 13.5 million learners worldwide, hundreds of courses, and thousands of instructors, Coursera manages over a hundred data pipelines for ETL, batch processing, and new product development. In this session, we dive deep into AWS Data Pipeline and Dataduct, an open source framework built at Coursera to manage pipelines and create reusable patterns to expedite developer productivity. We share the lessons learned during our journey: from basic ETL processes, such as loading data from Amazon RDS to Amazon Redshift, to more sophisticated pipelines to power recommendation engines and search services. Attendees learn: Do's and donts of Data Pipeline Using Dataduct to streamline your data pipelines How to use Data Pipeline to power other data products, such as recommendation systems Whats next for Dataduct

Adapting the capacity of your compute infrastructure to the demands of your applications is the domain of Auto Scaling. Adding and removing Amazon EC2 instances is only part of the story, though - there is more to it than first meets the eye. This session introduces the basics of how to use Auto Scaling before moving on to more advanced topics such as mixing Spot and On-Demand instances to optimize cost or strategies for blue/green deployments. If you have used Auto Scaling before, you can learn about useful new features like lifecycle hooks and step scaling policies that make Auto Scaling even more widely applicable.

Delivering a managed database-as-a-service in a highly secure and simple way can be a challenging problem, especially when your customers have many different network and access requirements. We went through many iterations trying to find a model that was easy to support, but also gave our customers control and visibility. In this session, we explore the incredibly flexible AWS networking solutions that we have used to deliver our services to customers with wildly different architectures. This is an advanced session for those who want deliver services into complex or divergent network architectures, while still maintaining control of the infrastructure that your services are deployed on.

AWS OpsWorks helps you deploy and operate applications of all shapes and sizes. With AWS OpsWorks, you can model your application stack with layers that define the building blocks of your application: load balancers, application servers, databases, etc. But did you know that you can also use AWS OpsWorks to run commands or scripts on your instances? Whether you need to perform a specific task or install a new software package, AWS OpsWorks gives you the tools to install and configure your instances consistently and help them evolve in an automated and predictable fashion. In this session, we dive into how you can create custom layers and a runtime system for your operational tooling, understand the lifecycle events, and show how to develop and test locally.

Two trends are driving app development: The shift from the server-based web to rich applications that run on a diverse set of mobile devices and modern browsers, and the growth of microservices running in the cloud that serve these clients. The results are "connected clients" - apps with the processing power of the device that are statefully connected and scaled to the cloud. In this session, you will learn about the architecture for Meteor's JavaScript app platform, Galaxy, which uses Amazon ECS, Elastic Load Balancing, and AWS CloudFormation to provide highly available, scalable, isolated environments for stateful apps across browsers and devices. We will discuss the essential characteristics of the platform, how those are provided for, and why we decided to use Amazon ECS instead of alternatives, such as Kubernetes. We will also demonstrate the Galaxy system in production.

You may already know that you can use Amazon CloudWatch to view graphs of your AWS resources like Amazon Elastic Compute Cloud instances or Amazon Simple Storage Service. But, did you know that you can monitor your on-premises servers with Amazon CloudWatch Logs? Or, that you can integrate CloudWatch Logs with Elasticsearch for powerful visualization and analysis? This session will offer a tour of the latest monitoring and automation capabilities that weve added, how you can get even more done with Amazon CloudWatch.

In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum as well as optimizing your overall capital expense can be challenging. This session presents AWS features and services along with Disaster Recovery architectures that you can leverage when building highly available and disaster resilient applications. We will provide recommendations on how to improve your Disaster Recovery plan and discuss example scenarios showing how to recover from a disaster.

This session explores building graph databases on AWS, examining common use cases, design patterns, and best practices. We then discuss the main options for running graph databases on AWS and go deeper into the Amazon DynamoDB storage backend plugin for Titan launched earlier this year. The Amazon Fulfillment team will share their story of running the Titan graph database on DynamoDB to track inventory going in and out of the company's fulfillment network. They provide best practices on running an efficient graph database at massive scale.

Learn how to monitor your database performance closely and troubleshoot database issues quickly using a variety of features provided by Amazon RDS and MySQL including database events, logs, and engine-specific features. You will also learn about the security best practices to use with Amazon RDS for MySQL as well as how to effectively move data between Amazon RDS and on-premises instances. Hear from Amazon RDS customer Airbnb about the best practices they have implemented in their RDS for MySQL architectures.

Do you want to analyze AWS CloudTrail events within minutes of them arriving in your Amazon S3 bucket? Would you like to learn how to run expressive queries over your CloudTrail logs? We will demonstrate Apache Spark and Apache Spark Streaming as two tools to analyze recent and historical security logs for your accounts. To do so, we will use Amazon Elastic MapReduce (EMR), your logs stored in S3, and Amazon SNS to generate alerts. With these tools at your fingertips, you will be the first to know about security events that require your attention, and you will be able to quickly identify and evaluate the relevant security log entries.

AWS Elastic Beanstalk provides an easy way for you to quickly deploy, manage, and scale applications in the AWS cloud. Through interactive demos and code samples, this session will teach you how to deploy your code to Elastic Beanstalk, provision and use additional AWS resources (for example, Amazon SNS, Amazon SQS, and Amazon DynamoDB), use your applications health metrics to tune performance, scale your application to handle millions of requests, and perform zero-downtime deployments with traffic routing. Demos and code samples will be available to all session attendees.

Amazon Cloud Drive's plans to provide a low cost, unlimited storage service presented a major engineering challenge. In this session, you learn how the Amazon Cloud Drive team designed and optimized the storage back-end, Amazon S3, to handle millions of users while containing infrastructure costs. In this session, the lead engineers share details of how they built the service for massive scale, and the regular steps they take to increase performance and efficiency. They also describe proven techniques for scaling and optimization, learned from experience.

The AWS SDK for Go is now live and ready for developers to build productive applications with using the Go programming language. This session will show you how to get started using the AWS SDK for Go. Well introduce the best practices for how to use the SDK, and how it integrates with your Go application. We will explore an example application which uses async programming with Amazon S3 and Amazon SQS to distribute tasks across multiple concurrent goroutines, and use Amazon Elastic Beanstalk to deploy and scale the application.