JANET Development Eye

It’s taken a bit longer than originally anticipated, but I’ve finally finished the minutes of the CGAMS BoF at TNC 2011. The original paper can be found here. Here’s an extract of the concluding text:

JH posed the following questions to the BoF:

Q9 -­‐ Is this implementation approach reasonable?

CP: Do we have sufficient clarity given the overall vision is still vague?

KO: Not sure we understand the question enough to be able to answer.

There following some discussion about the question – it was agreed to narrow it down to “as an initial direction”.

Yes – 23
No – 0
Abstain – 3

Q10 — If not, why not?

No responses.

JH walked through the “Proposed roadmap” section of the slide-set.

JH: Suggest we have a repeat of this meeting in 6-9 months to see if we’ve made progress in between; focus on getting some initial stakes in the ground.

CP: Also need to use feedback from today to refine roadmap, increase clarity on existing stuff (especially including use cases), etc.

JH: Do we need a mailing list

Room: General agreement.

JH: Do we need a shared space like a wiki?

Room: General agreement.

JH: Next meeting in about 6-8 months somewhere?

Room: General agreement.

JH thanked the BoF for their participation.

Some interesting analysis was presented by Pat Cain at the FIRST conference on trends from APWG (Anti-Phishing Working Group) data including their six-monthly surveys of domain names used in phishing campaigns.

There is evidence that concerted campaigns against phishing can be effective – the .hk domain used to be one of the most commonly used but is no longer in the top 10. However Government policies can also have unintended effects, for example one country that requires any recipient of public funds to have a website now has a high proportion of compromised servers hosting phishing campaigns.

Trends are a better measure than single statistics since a single phishing campaign (or the compromise of a registrar) can generate sufficient fake registrations to significantly alter a country’s registration figures. For example trends indicate that action to take down or block phishing domains has had the effect of making criminals change their tactics: free hosting sites used to be popular locations for phishing pages but as these got better at handling notifications the pages moved instead to cheap hosting sites, paid for with stolen credit cards, or compromised hosts. As browsers get better blocking tools, victims are increasingly asked to e-mail or phone their card details or even to upload forms to document sharing or survey systems.

In many ways phishing is showing the same trends as other types of eCrime, so APWG are investigating a more general classification of threats that countries or networks can use to benchmark themselves against aggregated global or regional statistics.

In a presentation to the FIRST Conference, Steve Purser of ENISA highlighted the difficulty of keeping security requirements up to date given the frequent changes in architecture that the computing world has experienced in the last twenty years. In this time we have gone from mainframes to simple networked systems to client/server and 3-tier models, to fully distributed computing (and, indeed, now on towards cloud architectures as well). Each of these systems architectures implies a different security architecture. This causes significant challenges for both designers and users, especially when organisations use more than one architecture at the same time, almost inevitably creating vulnerabilities at the joins between them. Users expect to be able to move seamlessly between intranet/extranet/internet, even though the security architecture(s) may depend on them behaving differently in each domain. Laptops and other mobile devices that move between domains can easily carry attacks and infections with them. Secure systems must recognise and accommodate these and other limitations of the environments in which they operate: unfortunately many of the most important limitations relate to human behaviour, rather than technological function, so are often missed when drawing up requirements. To be, and remain, secure, any system must be scalable, flexible and acceptable to its users. Unless these requirements are included from the start, technology alone cannot deliver a secure system.

I admit I was sceptical about World IPv6 Day.  Not because I thought it would cause a lot of problems, that fear didn’t come until much closer to the date, but more because I didn’t see the point and thought it was more likely to be both a public relations stunt from the companies involved, and also a tactic to delay deploying IPv6 for good.

I’m glad to say I was wrong.  Mainly.

As far as JANET(UK) is concerned, we didn’t receive a single World IPv6 Day related support call (I’d be interested to hear if you manage a JANET site and did receive calls).  Someone did notice that Juniper’s website fell off the Internet for a little while on the day, but that was due to a few errors Juniper made managing their DNS.

More than not having any reported disruption, we saw a substantial jump in native IPv6 traffic with our peers and providers.  This is a graph of the external (i.e. traffic coming across the interfaces to JANET’s transit providers and peers and not counting JANET to JANET traffic) IPv6 traffic the Monday before World IPv6 Day, an average day in the UK, not a public holiday, and this level is representative of what we’d been seeing recently.

External IPv6 Traffic, 2011-06-06

A maximum of around 30Mbit/s at a time when our overall external traffic is about 70Gbit/s,  or 0.04%.  Also note that it is very ‘peaky’ suggesting it is dominated by single transfers between a few hosts.  A few hours into World IPv6 Day, this is what the graph looked like.

External IPv6 Traffic during World IPv6 Day

When World IPv6 Day started at 00:00 UTC, 01:00 BST (the time on the graph is in BST), there was an immediate jump, and in contrast to the first graph, a clear diurnal pattern starts to emerge, far less dominated by the peaks.  The morning after World IPv6 Day, the graph looked like this.

External IPv6 Traffic covering all of World IPv6 Day

A maximum five-minute average of over 220Mbit/s!  In contrast to 70Gbit/s this is still small beer, but 0.3% is almost an order of magnitude more than we had been seeing, and after all, World IPv6 Day was not about enabling access networks, it was about enabling services, predominantly HTTP, and seeing what broke.  What this graph shows is that there are some networks out there in JANET with IPv6 enabled.  These are mainly at a handful of sites, but hopefully we can build on their experience.  To put it into context, here is a weekly graph of external IPv6 traffic, for World IPv6 Day and the six previous days.

External IPv6 traffic leading up to World IPv6 Day

The values here are averaged over a slightly longer period, so the maximum value is lower.

World IPv6 Day was just a one-day experiment, and the large content providers were always going to remove the ‘AAAA’ DNS records at the end of it.  Not everybody did though, and even some of the big guys left the DNS entries in for some of their services, such as the caches that serve user videos from YouTube.  What would that mean for the traffic come Thursday, Friday and the weekend?  This is what it meant.

External IPv6 traffic, week of World IPv6 Day

This is, of course, still much less than 1% of all our external traffic, but hopefully World IPv6 Day will have proven that IPv6 can be deployed for content delivery, and on a campus network.  This is also only a count of native traffic, there may have been some more that was tunnelled to external relays or tunnel brokers.

To answer the question at the top, World IPv6 Day was both a damp squib because nothing went wrong, and a roaring success for the very same reason.

The obvious question now is ‘what next?’  ISOC is considering another World IPv6 Day to allow content providers that missed the first one to jump on the bandwagon, and perhaps a World IPv6 Access Day.  Others are stating that content delivery is now a ’solved problem’ and the industry needs to move onto access networks.  If the content delivery is a solved problem, I’d like to see the content providers roll out IPv6 permanently, at which point the ‘killer app’ for IPv6 becomes IPv4 exhaustion and deployment of IPv6 will follow.  Perhaps.

Melissa Hathaway, speaking at the FIRST conference in Vienna, suggested that at age 40 the Internet may be experiencing a mid-life crisis. Although viruses, worms and cyber-crime have existed for nearly as long as the network, the benefits of open connectivity have seemed to far outweigh these disadvantages. Over 40 years the expansion in reach and use of the Internet has proceeded almost unchecked.

However the severity of incidents in the past few years has raised questions over this implicit assumption that Internet connectivity is always a good thing.  Compromises of on-line businesses can now cost hundreds of millions of dollars and affect millions of individuals; cyber-attacks have occurred alongside physical warfare, challenging traditional definitions of both “war” and “combattant”; cyber-espionage is a serious concern for both businesses and governments. Internet problems can also spread into other critical infrastructures, with concerns that devices controlling power distribution and other industrial processes may now be accessible from the Internet. Recent breaks in undersea cables in the Mediterranean the Pacific have revealed how much societies now depend on  communications networks.

Unfortunately this situation doesn’t seem to be something that any one group can fix. Attacks on governments and businesses are often facilitated by insecure behaviour by home users and their machines. Insecurities in business and government systems are likely to harm individuals. Somehow we have to explain Internet safety (and unsafety) to individuals, business, legislatures and governments, and motivate all of them to make the right choices. We may even need to admit that sometimes “the Internet” may not be the right answer. Will we achieve this by the Internet’s 45th birthday? Or its 50th? The speaker wasn’t willing to guess.

Nice slogans from SI-CERT’s Internet Safety Awareness campaign:

“Passwords are like toothbrushes – don’t share them and change them regularly”

“Personal information is like toothpaste – once you’ve let it out of the tube (onto the Internet) it’s really hard to put it back…”

They have been giving away visual aids in shopping centres and street markets

Since I posted an item a couple of weeks ago on World IPv6 Day, there have been several discussions both internal and external that suggest I may not have been entirely clear on the potential effects, so I am going to try and cover a few of those here.

First of all, and perhaps most importantly, nobody is going to be turning off IPv4 on June 8th.  The purpose of World IPv6 Day is to enable IPv6 access to websites, not to remove IPv4 access.  This means adding an ‘AAAA’ (quad-A) record in the DNS to popular websites such as Google and Facebook.  It should be noted that the JANET website has had both IPv4 and IPv6 access enabled for several years, so if you do not have problems accessing the JANET website, you should be fine on June 8th.  If, however, you or some of your users do have problems reaching the JANET website, that is an indication you should investigate a bit further before next Wednesday (I’d like to claim that the reason this blog is only available on IPv4 is to ensure it is still reachable next Wednesday, but the truth is that I don’t manage the system!).

Some operating systems ship (or have shipped) with some IPv6 transition mechanisms like 6to4 and Teredo enabled by default and not explicitly ‘depreferred.’  If these systems do not have native IPv6 access, they could be trying to use transition relays elsewhere.  Not only may those relays be slow to respond, access to them may be blocked by your firewall and the systems could take a long time to realise this and fall back to IPv4.  Again, if you can reach the JANET website, you should be fine, but use one of the connectivity tests I mentioned in the previous blog, such as Test IPv6.

Here at JANET we received a concerning email from Google earlier in the week.  It stated that some preliminary tests they have done suggest JANET is one of the top 10 networks worldwide for users that will have broken IPv6 access (access that will take longer than 15 seconds to fall back to IPv4), and they estimate there are 22,000 such users (Google also used a figure of 1.5% but admit the estimate could be very inaccurate) on JANET.  Those 22,000 users are almost certain to mean helpdesk calls next Wednesday.

[Update: I have received some more data from Google on what is broken, by and large it appears to be users of Mac OS X earlier than 10.6.5. JANET(UK) will be contacting the few sites that are directly mentioned, but if you have Macs with anything other than Mac OS X 10.6.5, 10.6.6 or 10.6.7 on campus and cannot upgrade you may be best off either disabling IPv6 entirely or using recent versions of Chrome (11.0.696.71) which try IPv4 and IPv6 in parallel, specific details are on the ARIN wiki.]

I can only guess that this is due to the historically open nature of campus networks compared to regimented corporate networks that are strictly controlled by a central IT support function.  If the 22,000 user figure is true, those users will be calling your helpdesks on Wednesday, so it is worth having some information ready, such as the IPv6 Test website, to diagnose the problems.

Some additional URLs that may be helpful are:

• Microsoft Knowledgebase item on how to depreference IPv6 for a short period.
• Microsoft Knowledgebase item on disabling certain IPv6 components.
• Microsoft TechNet blog on World IPv6 Day.
• RIPE IPv6 Eyechart.
• Comprehensive list of potential problems on the ARIN wiki.

As I mentioned at Networkshop, June 8th is “World IPv6 Day.”

This is not intended to be an event where IPv6 is enabled on access networks worldwide, instead it is a day where IPv6 is enabled by content providers and on which network engineers watch to see what problems it causes — or not.

Large content providers such as Google, Facebook and others will enable IPv6 on their main websites for all visitors, not just those that have been participating in limited trials up until now (i.e. through Google’s “whitelisting,” or http://www.v6.facebook.com/). JANET’s own website has been “dual-stacked” (i.e. available over IPv4 and IPv6) for some years and we have had little feedback on it, but there are some things that campus network managers and regional network operators will want to be aware of.

Campus networks that are only IPv4 should have no problems.  Similarly, networks that have deployed some managed IPv6 connectivity should have no problems either.  The potential pitfalls are where there is poor connectivity due to unmanaged IPv6 automatic tunnelling mechanisms such as Teredo and 6to4.  This may affect your users without your knowledge, so your helpdesk should be aware that connectivity problems reported on June 8th could be caused by something other than the usual set of issues.  Some estimates put the level of users that can expect problems reaching dual-stacked networks at about 0.05%.  Others put that much lower, some slightly higher.  The aim of World IPv6 Day is to get a handle on those numbers and start the work towards solving the problems.

What is happening on June 8th?

When you go to “www.google.com” in your web browser, a lookup is performed in the DNS (Domain Name System) that answers with an IP address.  Your web browser then connects to that IP address to get the content.  On June 8th, the major content providers will not only answer the DNS lookup with an ‘A’ (IPv4 address) record, but also with an ‘AAAA’ (quad-A or IPv6 address) record.

As software prefers an IPv6 address if one is present, this means that your web browser will try to use IPv6 to connect to the website and retrieve the content.  Most of the time this will not be a problem, you will still also receive an “A” record and if you don’t have IPv6, you will continue to use IPv4 as happily as normal.

However, due to some dubious engineering decisions made in the past, some operating systems shipped with various translation technologies enabled by default, of particular concern are those called “Teredo” and “6to4.”  These attempt to give you IPv6 connectivity even when you don’t have it natively by tunnelling across the IPv4 network to reach an IPv6 relay.

You should try to be aware of these as a matter of course as any tunnelling technology can potentially bypass your firewalls if it isn’t blocked or managed, but this is also part of the potential problem when it comes to World IPv6 Day.  Browsers and operating systems detect broken IPv6 connectivity (as opposed to non-existent IPv6 connectivity) with various degrees of success, and as a result may attempt to connect to a website using IPv6 on June 8th, then pause for some minutes before realising all is not well and falling back to IPv4.  What may be even worse is when a PC or laptop that has such broken IPv6 connectivity also turns on some form of “Internet Connection Sharing.”  It may then tell all the other computers on the local LAN that it is an IPv6 router and they may have broken connectivity through it.

So, before June 8th it would be worth testing IPv6 connectivity from various parts of your campus LAN using one of the following resources:

• http://test-ipv6.com/ This performs a number of tests of IPv4 and IPv6 connectivity and tries to diagnose potential problems, such as the use of auto-tunnelling mechanisms.
• http://www.testmyipv6.com/ A little bit simpler.
• http://omgipv6day.com/ Simpler still!

If you have a few minutes, it is also worth reading a presentation given by Dave Freedman from Claranet (video here) at the recent RIPE meeting.  Another version of Dave’s slides, with a bit more detail, were presented at UKNOF.  Also, Tim Chown’s presentation from Networkshop includes a thorough list of security issues you need to think about when deploying IPv6 — even if you don’t think you have any IPv6 already!

Incidentally, there is a bit of work progressing through the IETF at the moment called “Happy Eyeballs.”  This suggests that instead of software trying IPv6 first and falling back to IPv4 if that fails, it starts both connections at the same time and uses whichever one replies first.

What can you expect on June 8th?

In an ideal world, nothing.  However, you may get calls from your users about not being able to reach Google, Facebook or other sites.  Ask them what the output is from one of the IPv6 test sites mentioned above, find out if they are using an auto-tunnelling mechanism and if they are, have some steps on how to disable it.

What else can you do on June 8th?

If you have content, such as a web or mail server, make it available over IPv6!  Watch the logs, see if there are any problems, note them, and see what you need to do to make the services available over IPv6 in the longer term.  Like it or not, IPv4 stocks are running low, and for worldwide end-to-end connectivity that does not rely on multiple levels of IPv4 NAT, IPv6 is the only other solution on the table.

If you’re using Google Analytics, then APNIC has some JavaScript that allows you to perform IPv6 measurements, which is described in some detail over on Geoff Huston’s blog.

We’ll be watching the levels of IPv6 traffic on JANET, which if you saw my presentation mentioned at the start of this item are woefully small.

What can you do after June 8th?

If nothing went wrong, look into making your services permanently available over IPv6.  Look at what it will take to roll out IPv6 to your campus network so all your end-users will be able to use it.  JANET has some documentation including an IPv6 Technical Guide and is starting an IPv6 Fundamentals training course.

Click to download paper

Parties who are interested in tracking and discussing this proposal are encouraged to subscribe to cgams-announce@jiscmail.ac.uk at http://www.jiscmail.ac.uk. This proposal will also be the subject of discussion at a BoF at the TERENA Networking Conference 2011 on Thursday 19 May at 1400 CEST.

An extract from the “Executive Summary” of Proposal for a common global access management system (click to download the paper).

In recent years, user requirements have driven the development of a set of access management systems to support inter-organisational activities within the global research and education community. These systems have, in general, addressed separate and often quite different application domains. Viewed independently, these access management systems have generally been highly successful. However these systems have, in general, addressed separate and often quite different application domains and, when considered as an ensemble, two issues become apparent:

• Their multiplicity imposes significant complexity and costs on organisations and users, by requiring them to interact with a number of dissimilar access management systems.

• Despite their multiplicity, these systems fail to address the inter-organisational access management requirements of many applications, resulting in significant opportunity costs.

An access management system that not only provided an access management solution for these other use-cases, but also for those of existing applications, would clearly be highly desirable. A new access management technology, ABFAB, developed primarily by the education and research community and undergoing standardisation within IETF, may be able to deliver this.

The technology has already been substantially implemented by Project Moonshot, a JANET(UK)-led initiative in partnership with the GEANT project and others; the planned work will be completed during Q3 2011. The technology has been demonstrated with a number of applications; little or no software modifications were required.

This paper argues that an access management system that delivered value to significant parts of the global research and education community could be implemented within months, and at relatively little effort and cost, and without ‘re-inventing the wheel’. This would be achieved through re-use of the global RADIUS infrastructure that currently supports eduroam: it presently incorporates over forty countries on every continent, connecting thousands of organisations and many millions of users. Some minor modifications to this infrastructure would be required but these could be managed almost entirely through upgrades to existing software. Most of the required effort would need to be directed at modifying existing the RADIUS infrastructure’s policies to address the new applications.

This access management system could confer a number of benefits to the community, including:

• Lower operational costs for service providers and their customers.
• Increased opportunities for collaboration and revenue generation.
• Improved user experience leading to greater adoption and use of services.

The purpose of this document is twofold:

1. To brief the reader of a proposed strategy for establishing a common global access management system by April 2012, that will hopefully lead to discussion and consensus. This paper does not not seek a single access management system; it is instead advocating a common access management system that is available to those that wish to use it.
2. To ensure that the community is aware of the potential opportunities and other consequences of the technology, so that informed action can be taken.

There is no Internet presence for this proposal at present; the manner in which this proposal is taken forwards – if at all – is a matter for the community. However, a temporary mailing list has been created to support co-ordination in the interim. Parties who are interested in tracking and discussing this proposal are encouraged to subscribe to cgams-announce@jiscmail.ac.uk at http://www.jiscmail.ac.uk.

Matt Cook’s talk at Networkshop explained Loughborough University’s thinking on how virtualisation might be used to provide both resilience and flexibility by allowing services to be moved between different locations in both internal and external clouds.

Rather than virtualising a single server, this involves creating a virtual container holding the various components required to deliver a particular service. For example a virtualised VLE container would also need to include the underlying database, a DNS resolver giving a consistent view of the world (especially if DNS views differ for ‘internal’ and ‘external’ requests), a mirrored copy of at least relevant parts of the authentication/authorisation system, and a network firewall. Such a container can then be moved relatively easily between data centre hosts, whether in response to load, system faults or simply changes in contracts.

Such flexibility does, however, create significant demands on Internet routing to ensure that the container can be ‘found’ wherever on the network it happens to be (re-)located and, indeed, no matter whether the user is connected to the campus network, elsewhere on JANET, or elsewhere on the Internet. Although this should, in theory, be possible using IPv4 addressing, the near-exhaustion of that address space means it may be hard to find enough contiguous public addresses. Loughborough are therefore planning an initial trial of this approach using IPv6, where it will be much easier to obtain the required addresses while preserving a hierarchical allocation of address blocks (and therefore simple routing tables). Matt also observed that moving a major service such as a VLE, document store or e-mail to a different network location can have significant effects on traffic flows. For example moving e-mail from on-site to off-site added 20-40Mbps to the traffic on Loughborough’s JANET link: organisations need to include this effect and any impact on network components in their out/in-sourcing plans.