#puppetconf

Why did we think large scale distributed systems would be easy?

Let's imagine that your team needs to double the amount of servers for next year. Let's also assume that by 2015 you will need to increase your capacity to 15,000 hypervisors requiring somewhere between 100,000 to 300,000 virtual machines. You have an increasingly brittle infrastructure that has reached it's limits of scaling, and you've been told that you won't be able to increase any of your staff for a while. How would you possibly achieve all of this?

Tim Bell is an IT Manager from CERN who faced this exact situation, and his team decided to avoid using any proprietary solutions and instead adopt an open source toolchain to bootstrap their infrastructure. Puppet, OpenStack, and Foreman are at the center of their chosen solution, and these open source tools are helping scientists to make cutting-edge scientific discoveries by managing the infrastructure for some incredibly challenging data collection scenarios.

CERN is the European laboratory for particle physics that is famous for creating the Large Hadron Collider (LHC), which is a 17-mile ring that is buried 100 meters underneath the ground. One of it's main tasks has been is to create the conditions to observe the Higgs Boson particle and provide further evidence for the Standard Model of physics. CERN scientists are also investigating the open questions of:

Why do particles have mass?

What is 96% of the universe made out of?

And what is the nature of matter right after the Big Bang?

The LHC is one of the largest engineering feats of mankind, and it produces an equally impressive amount of scientific data. In order to detect evidence of the Higgs Boson, there are thousands of bunches of protons that are traveling around the collider at a rate of 11,000 times per second. Each bunch contains around 100 billion protons, which produces around 600 million collisions per second. In order to capture these collisions, there are four sensors that are equivalent to a 100 Megapixel camera that is able to take around 40 million pictures per second. This yields a total of around 1 Petabyte of data per second that needs to be filtered down by software to a more manageable stream of 5 to 25GB of data per second that can be captured to tape.

CERN is responsible for capturing and storing this data, and then making copies available to other Tier 1 storage centers. The Tier 2 computing grid then analyzes the data by providing 100,000 CPU days and executing 1 million jobs on an average day. The CERN IT department is tasked with capturing around 25 Petabytes of experimental data per year, and then storing it for up to 20 years. The data center has over 800 racks with nearly 12,000 servers, over 15,000 processors and 64,000 cores. 

Bell realized that even though CERN is on the cutting-edge of scientific research, they were no longer on the leading edge in terms of computer capacity. As Luke Kanies, CEO of Puppet Labs, has said, "If your infrastructure is special, you're doing it wrong." Bell and his team had come to this same conclusion and realized that it was time to look at what other people in the industry are doing rather than continuing to write and manage their own configuration management tools.

At PuppetConf 2012, Bell shared a diagram showing the building blocks of their open source infrastructure management toolchain, which includes Puppet, OpenStack and Foreman:

Bell talked about how OpenStack is very flexible and configurable, but that those options can be complex and somewhat overwhelming to set up out of the box. However, the Puppet Forge OpenStack module has made this process a lot easier to initially set up and reproduce the results.

Another benefit to moving to this open source toolchain is that training has moved from an extended 2-month process of one-on-one mentorship to a much quicker training process. Instead of waiting months before someone can be productive, it's now a matter of days after reading external training books or immediately if they've had previous experience with using Puppet. CERN also has many short-term contract employees that are returning home with marketable Puppet skills that are more transferable to other professional contexts as well as with other CERN-connected, computing grid locations.

Bell also commented that CERN has aggressive growth goals that they'd never be able to tackle on their own, especially without being able to increase their IT support staff. They are committed to contributing back to Puppet and to Puppet forge and he encouraged others to do the same. Not only does this participation help the Puppet project, but it is literally helping scientists to find the 96% of the Universe that we're missing.

You can watch Tim Bell's full presentation here:

You can also sift through his slide deck here:

State of the Puppet Community - PuppetConf 2012 from James Turnbull

James Turnbull wrote some books including "Pro Puppet," "Pulling Strings with Puppet: Configuration Management Made Easy," "Hardening Linux," "Pro Nagios 2.0," and "Pro Linux System Administration."

VP Technical Ops at Puppet Labs, education, training and community management. Australian accent. Film fan.

He's amazed that Puppet keeps growing. Puppet gatherings are great for being able to share the stories and to get real value out of their experiences, have great conversations and communicate. 106 employees at Puppet. Wore all of the hats except sales since the beginning. Pleased to have community management under him as well.

Ops tools world is a bit like Aliens. We're kind of like the colonial marines without the massive death toll and implanted parasites. Princess Bride is his favorite film, and will use moments from the film in the presentation.

Introductions

State of the community

Initiatives

Ask what your community can do for you

Questions

Introductions

Dawn Foster is the new community manager, and her 2nd day on the job.

Community lead at Intel. Lead Tizen and Meego, and worked at Jive. Started as a sysadmin back in 1995, and getting back to her roots. CS degree and MBA.

Will be lurking and learning for the first month. Work on publishing some of Puppet's community metrics. It's presented once a year, and wants to publish metrics more frequently.

Can be reached at @geekygirldawn or dawn at puppetlabs dot com

What does Puppet community does really well? What are things that it wants to do? Community Point of escalation. If it's getting ignored, then she can track people down and help get some movement on issues.

Andy Parker is the Puppet Team lead

Did some system administration, and used cfengine. Wrote a paper on configuration management. Does cycling and long-distance cycling. New to open source development, and is learning. Background is financial services software and it was a lot more locked down.

What do you have happen? What do you want to do? How can I help?

Andy at puppet labs and @zaphod42 on IRC

Eric Sorenson, Puppet Platform Platform Owner, and owns the roadmap.

SysAdmin at Apple. eric dot sorenson at pupplabs dot com and eric0. Talk to him about pull requests and features.

Metrics for the Puppet Community

Metrics from 2011

Puppet user mailing list membership was 3000

600 messages a month

#Puppet IRC channel averaged 400 people - One of the busiest IRC channels

Puppet on GitHub had 200 forks and 400 watchers

We had 50 modules on the Forge 

Metrics from 2012

5000 members on the mailing list

1200 a month Doubled the number of messages in a year 

#puppet IRC channel averages 800

Puppet on GitHub has 350 forks and 950 watchers

We had 500 modules on the Forge - How to contribute back features

Turnbull was blown away. Constantly amazed for how the community has organically grown together.

Engineering. What has Puppet Labs done in development? Needs to improve communication of what Puppet Labs is doing to the community, and will see more information.

Number of commits to Puppet per year:

2005 - 475

2006 - 1123

2007 - 1205

2008 - 1205

2009 - 930

2010 - 1254

2011 - 2454

2012 - 2447 (just year to date)

Will going to see a lot more commits in the future.

Ratio to core commuters to contributors has been growing over the years

Up to 99 contributors and 28 core contributors.

Don't have access to all of the platforms, and so there are some contributors who only contribute 1 to 2 commits, which is really helpful to get compatibility on all infrastructure platforms.

SLOC has gone from 21k to 141k since 2005.

What's next for the Puppet Community?

Questions and answers - Talk to Dawn any of your needs

Developing a Q&A Site Not everyone likes mailing list. People like forums, and putting together a QA site in the site of forums and leverage the success of sites like Stackoverflow. Mostly sysadmins, and getting better at design. Emphasize curated on practical questions and practical answers. Get questions indexed by Google instead of looking through layers of mailing lists.

Becoming more responsive

Sometimes we suck at communicating. Being in a start-up, it's easy to be heads down and not have the bandwidth for communication. 

Puppet Labs is the primary sponsor of the open source community, and will foster more interactions in the community.

We're getting better.

There has been some staff movement with the community management position. Mike and Jose did a great job at organizing the Puppet Conf and Puppet camps.

Call us out if Puppet needs to be more responsive or is not doing things that you want. We'll admin mistakes. Mistakes will be made. Honesty and transparency are key to fixing things.

More metrics

Dawn loves metrics. It's to understand who the community is, and what they are. We want to be transparent and accountable.

More awesome

Talk to us, complain to us, give us feedback that we can action and help us out.

State of the community

Puppet needs the community's help as well. Contributing to an open source project is personally and professionally rewarding. Turnbull used to work at a bank, and it was not satisfying. Worked on Puppet Labs in his spare time, and it got him a job. It's significantly changed his career, and he's much happier. Opportunity to give back to a community he cares about.

Needs help with Documentation

Wiki needs updating. Move info from Wiki into docs site: http://docs.puppetlabs.com. Put in pull requests for migrating documentation.

Documentation: http://github.com/puppetlabs/puppet-docs

Puppet Redmine has tickets: http://projects.puppetlabs.com/projects/puppet-docs

It makes people lives better and easy to get involved.

Facter

Have a platform Facter doesn't know about?

Have a resource or infrastructure component that you'd like exposed? 

Then help to expand the test coverage

http://github.com/puppetlabs/factor

http://projects.puppetlabs.com/projects/facter

Modules

Share and publish your modules. Workflow is not right. Bury us in modules and we'll get it right. Make it a pain point by contributing more modules

develop modules

There are module bounties that involves winning something. Graphite and log stash module bounties. These will be published on the blog. We will be featuring a module of a week. Help to write about a module

Forge is at http://forge.puppetlabs.com/

Code and Testing

Triage Puppet bugs and features

It's easier than it looks. Parts of Puppet are still a mystery to James, but things on the edges and extension points are easy to write and integrate with and to pick up Ruby. 

Provide version and reproducible information and write good tickets.

Puppet 3.0 fixes were because they were voted on

Write code, Write tests

Team will help to provide input.

Test releases -- Get as much feedback as we can for each release

Get some tips

Getting started with Contributing to Puppet and Factor

Joachim "Jok" Thuau, SpaceX talks about how SpaceX uses Puppet.

Linux system engineer, and came from the video game industry. Been at SpaceX for a year. It's a rocket company and first company to send a vehicle to low earth orbit. First to recover the vehicle. Visited the ISS. Designed the Falcon 1, 9 and Heavy rockets. It was too small to do anything and on a bigger scale and build scaled up rocket. And then used 9 engines -- Falcon 9.  Planning Falcon Heavy, and it will be 3 Falcon 9's stuck together. Rocket doesn't have any payload, and using a capsule named Dragon.

Puppet. Talked about Puppet for a long time. Problem: Need to deploy 10s or 100s of machines. Have Puppet, but haven't used it. It'd take hours instead of months. Getting lots of requests to deploy a lot of machines. Signed up for scaled university, and did a 3-hour Puppet training. Convinced them to use Puppet, and just got started. Sat around table for 3 hours and had only seen stories. Installed Puppet and started simple, and it was immediately useful.

What we have

Debian/Ubuntu

Kerberos (windows KDC/Active Directory)

LDAP (windows LDAP/Active Directory)

"Apt" mirror (with debian squeeze/ubuntu LTS)

Tons of scripts!

Deployment

Using existing scripts and replacing them with puppet.

Install packages from our local repo

Push configs with puppet

Using exec (in a couple of places only) to "fix" a couple of things.

This simplified things a lot for us

Where are we at now?

Plug desktop/server in (power/network/keyboard/video)

Boot "off the network" (pxe)

Pick on pxe menu what you want to install


We have a number of "presets" we use

Install starts and asks the user for hostname

Machine uses a pre-seed file to setup stuff initially

Installs puppet as part of that pre-seed from our local mirror

Generate and registers with puppet server

(login on server and sign cert)

Cool stuff to come

Foreman & Puppet server infrastructure overview

The Foreman

Really nice and easy to setup

Has "proxies" for handling stuff with DHCP/DNS/PXE automation (we’re planning on using this, and it’s a little more difficult for us, since that part of the infrastructure is windows based)

Nagios monitoring

Host lists/services/dependencies built by puppet


HPC

We have a compute cluster that we are starting to rebuild using puppet for automation and validation (test cluster with a couple of nodes, validate config, and push to production)

Accelerating science with Puppet from Tim Bell

CERN looks at universe to see what it's made of and how it works. Blue sky research. International Organization, and runs like an independent country. Own police and firemen. Spread over 20-25km.

Physicist worry about 4 things. Why do we have mass at all? Higgs Boson looks promising. Standard model

What is 96% of the universe made of? We've lost 96% of the universe. WHat is it? Why can't see it?

 Why isn't' there anti-matter in the universe? Nature should be symmetric.

What was the state of matter just after the Big Bang? What was the pre universal soup?

Community collaboration on an international scale

Four major experiments around the Large Hadron Collider. Searching for Higgs Boson. Protons go around 11k times a second. Neutrino experiment. Effects of cosmic rays on cloud formation. Look at anti-matter and trap anti-hydrogen and study properties of anti-matter.

LHC construction. Custom hardware because of the scale and size. Science and engineering now creating colliders in hospitals. 5 years of construction and major engineering feat. 

17 mile ring. Pressure inside in pipes 10x less than moon. Superconducting magnets requires lots cooling.

Have detectors at four places around. Takes 40million pictures per second. 1Petabyte of data per second that they have to deal with.

Send 2-3k bunch. 600 million collisions per second. 200 protons and ions.

1Petabyte a second, and can't record or store this data. Filtering data live and sending it to computing second. 5GB a second coming into data center. Make a copy and store it. Can't analyze it. Use data grids around the world, 200 centers to analyze it. 1 million jobs a day.

Whitebox, ethernet boxes. 64k disks with high failure rates. 200 disks date every year. Compared to Amazon and Google, seeing similar data rates.

25 Petabytes a year coming from experiments and want to store it for 20 years. 30-45 Petabytes a year. Up to 25GB of data per second. Lots of data infrastructure to look after data. Scary PEU numbers. Only get 6kW per rake. Open environment with 10k visitors. 80k people over one weekend during an outreach.

How to store data? Tape is affordable. 45k tapes 20% write and 80% read. 

3.5MW of energy, and needed a second computing center. A new site in Hungary and have same more capacity. Can't increase staff levels. 2x machines with the same amount of staff. CERN is not leading edge in terms of computing capacity. CERN was no longer special and look to see what others are doing.

Building blocks and follow Google toolchain model, including Puppet. Avoided running custom components. Build in 12 months. 

Training the support. Used to be guru training. But now go buy Pro Puppet, Managing Infrastructure with Puppet and Puppet 2.7 and start being useful and productive. Back to their homes and have marketable skills with Puppet experience.

Ask for tape. Take 5 minutes. Moving towards a cloud model to overcommit model to improve resource efficiency. 

Ask for machine in a reasonable model. Used to be 6 months to get a new machine and now within a matter of minutes.

Animal naming versus cattle. Animals get names and are nursed to health when they get sick. Cattle get numbers. When they get ill. You shoot them [laugh]. Move users towards a cattle model.

Open Stack. Open stacks many components and options make configuration complex out of the box. Puppet forge model from PuppetLabs. The Foreman adds OpenStack providing for user kiosk.

Scaling up with Puppet and OpenStack. LHC@Home is SETI like for simulating magnetics. Using puppet module puppet-boinc. 

Next steps: Heading towards 15k hypervisors in 2015. 100-300k virtual machines. 100 Cores, which is cheaper than people to overlook infrastructure. Working to get other grids to adopt Puppet. How to configure Macs, windows and Linux machines.

Final thoughts. Best decision that CERN made was to open source the WWW. Seeing similar behavior in Puppet community where people are solving similar issues. When you contribute to Puppet and Puppet forge, then you are helping to find the 96% of the Universe that we're missing.

Lifeblood of IT community that EMC tries to serve. Thanks for making it more predictable and consistent in the face of complexity with Puppet.

Will be talking Different layers of complexity

Hadoop Shop. Testbed for Apache cluster. 1,000 physical host, Intel processors, and community came together to provide Apache a testbed. 10k nodes in VM. 1k nodes w/ bare metal. Move the state of the art.

Allow everyone to participate, and pulled together with Zookeeper.

How to keep running? Puppet is a key part of the stack. Also use it internally, and move to prod soon. Lots of vibrancy in the Puppet community, and will be contributing soon.

Extending DevOps. 

EMC is a storage company. Produce and store data. Weather 1+ petabytes. Exo-scale. Talking more than a million hosts to get there. 1/10th of a power station. Driving into pure science. Information is driving the mobile experience and driving new insights into failure modes. Information will dominate mobile styles.

It's a complex system. Read across hundreds of data set to bring their data to the party. Def of Complexity. Massive number of connections. Not homogenous. Not clean boundaries. Interdependent. May be open. May have an memory that you have to respect states. Non-linear. Loops and feedback.  How to merge op platform to manage complexities.

Complexity management. Distill and partition. Hierarchy and create independence. How to make IT consistent. 

Manage 3 places of the portfolio. Big data. What's the workload of the underlying infrastructure? Lots of data coming in. Want data to live near it's produced or consumed. Changes paradigm, and mobile. 30% of info is internal and 70% external from company. Need orchestration layer provided by cloud

Service Providers. Enterprise IT will move to the cloud because performance isn't keeping up at data centers. How to allow self to orchestrate in service provider landscape. 

Next Gen Data Center. Build with specialized equipment. Emerging apps are impart independence in their frameworks, and there's a shift in the landscape. External control plane on top of the network to have independence of control of the data path. 

Emerge a new stack with a cloud stack hierarchy. Infra with shared services. Network compute and store resources with Razor domain with Puppet. Virtual information platform that's emerging. On top is the new application developer landscape. They want stability. J2EE is looking more like a mainframe. Looking for agility in infra with modern framework to describe complexity of interactions. Use declarative models to describe DevOps.

Where is app running? What's next to it? Who's on this box last? Important to security and availability. Common control plane that emerges in a hierarchical model.

Extended role for DevOps. Virtual platform has customized containers. Configure switches and storage and insight go into state manager.

EMC has some grand plans. Systemic state management. Cross cloud structural consistency. Common monitoring. Drive IT analytics to see how cluster is running so that you can improve. Provisioning. Reproducability and recovery

Information landscape. How to operate critical workload? Multi-language (polygon). Multi-form key-value/relation store on the backend. Multi-latency. multi-tenant, multi-view, multi-location, multi-policy

Example of "InfoOps." The Information Developer to orchestrate complex information services. Compositional operators. Assert controls and boundaries. Reproduce-ability. Audit.

Demo of Operationalize Bare Metal. Have Windows provisions running in Razor today. Run from catalog. Take windows ISO and drop it into the environment. Peeled away ISO and instantiate index and independent images in the catalog. Be very specific as to which Windows ISO to deploy. Establish and bind new policy. Defined windows model. Bind and apply policy to that node. Take policy and update it and hit enable. It's occurred. Policy is bound. System shuts self down under razor direction. Comes up via PXE bootstrapping. Get pre-boot environment and install process, and pull client frameworks from Razor. Define installation components. Normal install will complete and the reboot will occur and the system will come up and run. Windows lack installer that's consistent with open communities. Bringing more windows integration capabilities. Windows comes up. See pre and post install customers. Naming and networking structures in pre-boot and post-boot capability. Windows has Puppet window in it.

More details at https://github.com/puppetlabs/Razor