ICANN - Nairobi DNSSEC Workshop 10 March 2010 >>JULIE HEDLUND: Hello, this is Julie Hedlund, we're at the DNSSEC workshop in Nairobi, Kenya. I just wanted to say hello to those who are on the telephone and let you know that we'll be starting in just maybe five minutes or so. We're just working out a few technical things here. Who do I have on the phone right now? >> (inaudible). >>JULIE HEDLUND: Thank you. And who else? >>JIM GALVIN: Jim Galvin. >>JULIE HEDLUND: Good morning, Jim. Who else is on the line? >>JIM GALVIN: Good morning. >>JULIE HEDLUND: Hello. Good morning. >> (inaudible). >>JULIE HEDLUND: Russ. Great. Can you hear me okay? >>RUSS MUNDY: Yeah. >>JULIE HEDLUND: Could you confirm that you can hear me okay? >> Yes. >>JULIE HEDLUND: Thank you. I just want to make sure that the sound was okay from this end. >> Yes, we hear you great. >>JULIE HEDLUND: Wonderful. >> You look good on camera, too. >>JULIE HEDLUND: Gee, thanks. Yeah, this -- we have a rather sophisticated setup this morning. So bear with us. And next time you hear from me, we should be getting going. But thanks very much. >> For folks who want to follow along with the slide set, I believe Julie just sent them to the mail list. >>JULIE HEDLUND: Jim Galvin on the line, was that just you typing into -- or who just typed in for the screen? >> Not this moment, I didn't type. But -- >>JULIE HEDLUND: I was wondering if you were typing into the chat room. Are you in the chat room by any chance, Jim? >>JIM GALVIN: Yes. I'm in both the Adobe chat room and also the DNSSEC deployment one. >>JULIE HEDLUND: Oh, thank you very much. I really appreciate that. We are just at the moment getting into the Adobe chat room. And the slides for the Adobe Connect will be up shortly. Hold on for one moment. >>STEVE CROCKER: Yeah, I think in the interests of time and with respect to -- and with the people who are participating remotely, we should move ahead. We have a relatively full schedule. So let's just crank it up. >>JULIE HEDLUND: Good morning, everyone. And hello to everyone on the phone. and who may be joining us through audio through Adobe. This is Julie Hedlund. I'm director of SSAC support. And I want to welcome you to the DNSSEC workshop in Nairobi, Kenya, on March 10th, 2010. I just wanted to take care of a few administrative details before we begin this session. And then I will turn it over to Steve Crocker. We are -- we have several methods that people are participating in this workshop. We have several people here in Nairobi in the room. We also have access via Adobe Connect through -- that you can connect to through the schedule, the ICANN meeting schedule online. If you click on the workshop. The presentation materials, the slides, are ordered according to as they will appear in this workshop. And they are being loaded into the schedule online so you can access them there. They also have been sent in hard copy to the DNSSEC deployment list, and when you receive them there, you may access them there as well. When we run through these slides at the -- when we ask for questions, we want to make sure that we entertain questions from the room here, but also from people on the telephone and people in the audio room. And I would ask that when you do speak, please use the microphone. And if you're in the room, please introduce yourself and give your affiliation. This session is being scribed. We have live scribes. And it will be very helpful to them to be able to get your names correctly for the scribing. And also, I would ask, if you are on the telephone, if you're not speaking, to please mute your phone so we don't pick up any background noise. So, anyway, I think that is all that I have for now. And I would like to turn the session over to Steve Crocker, who is the co-chair of the DNSSEC deployment initiative and also the chair of the Security and Stability Advisory Committee at ICANN. >>STEVE CROCKER: Thank you very much, Julie. It's a pleasure to be here in Nairobi, and it's a pleasure to see everybody here and to know that we have a significant participation remotely. This session has become a regular feature of the ICANN weeks. We run a DNSSEC session traditionally, now, every Wednesday morning. And it's been quite exciting to watch the evolution of DNSSEC through its deployment process from a period when it was questionable as to whether there would be any traction to a period now where there's no question about it moving forward, and we now have even enough data to track it quantitatively. I want to recognize my colleagues in the planning process here. Julie Hedlund, whom you just heard from, has been amazingly effective at pulling all of the pieces together. Markus Travaille, from the Dutch registry, SIDN, and Russ Mundy from Sparta in the U.S., and Julie and myself have been regularly putting the pieces of this program together. I particularly want to thank Markus, who approached us at the end of the prior session that we had in Seoul and, at some risk, put forth the bold idea that maybe we could improve these sessions and suggested that we might focus a bit more on deeper content and (inaudible) topics as opposed to a general fast-paced briefings by everybody across the whole spectrum. And it resonated quite a bit with me. It matched some of the thoughts that I had had in my background. And so we did the obvious thing, we told Markus he was now unavoidably involved in the planning process, and he graciously agreed and has been very active. So I'm appreciative, and I think it's been effective. So this session here will include a kind of deep dive into a particular technical area having to do with the combination of key rollovers on the one hand, and transfer of registrations and DNS operations across registries and across operators. We're fortunate to have Roy Arends and Olafur Gudmundsson here to make these presentations. So -- and that's kind of in addition to the usual spectrum of presentations that we have. So with that, let me move rapidly into the program that we have. And I'll make some other remarks along the way as we go along. Yeah. >>JULIE HEDLUND: I'm sorry, Steve, this is Julie. I have one more administrative thing. Could I ask the people on the telephone, for the purpose of the describes, to announce themselves, since the scribes can't see you and we don't have your names, we would like to be able to capture them for the scribing. Please, each of you, announce yourselves. And do speak loudly, because I gather that some of the telephone sound comes in rather lightly for the scribes. Those on the phone, could you announce yourselves? If you're on mute, you can take yourselves off mute. >>JIM GALVIN: Jim Galvin here. >>SUZANNE WOOLF: Suzanne Woolf. >>RUSS MUNDY: Russ Mundy. >> (saying name). >>JULIE HEDLUND: Anyone else? Could the last person who spoke repeat a bit louder? The scribes did not catch your name, unfortunately. >> Thierry Moreau. >> (inaudible). >>JULIE HEDLUND: All right. Could you -- was that -- could you repeat yourself one more time? The scribes didn't catch the name. >> Samuel Weiler. >>JULIE HEDLUND: Thank you, everyone, I appreciate your patience. Since we do have remote participation and scribes, this is quite helpful. >>STEVE CROCKER: Good. So we've now completed the first item on the agenda, which is my brief talk, probably not brief enough. And so let me turn things over to Kim Davies, who will be speaking -- let's see, how do we do this delicately? -- are you speaking on behalf? Speaking with full authority and on behalf of three organizations: ICANN, VeriSign, and NTIA? Or are you just going to try to skate through this as smoothly as possible? The last, no doubt? >>KIM DAVIES: I am not actually a member of the DNSSEC deployment team between the three organizations. So I think I can safely do the latter. But I hopefully know enough to answer all of your questions. >>STEVE CROCKER: Yeah. >>KIM DAVIES: So as an introduction, my name is Kim Davies. I'm responsible for the root zone at ICANN. I work within the (inaudible) department. And I'm here to present to you on behalf of three different organizations: ICANN, the U.S. Department of Commerce, and VeriSign. The three organizations have been involved together in deploying DNSSEC at the root zone. As I just intimated, I am closely involved in this process, but I'm not directly on the design team. So I think I know 95% of what I need to know. But I apologize if I don't know some of the specific details if there's any questions. Next slide. So the design of the root zone signing process has been a collaboration. We've been supported by the U.S. Department of Commerce. and the design itself has been collaborative between -- First I'll talk about the design. There's a few key words that went into the design process, which I'll go through. It would be helpful -- would it be helpful to switch it to mine? Okay. So the first key word in the design is "transparency." In order to -- for the community to trust a signed root zone, it was felt that transparency was key. And all the processes and procedures are designed to be as open as possible, to be as transparent as possible, so the community gets as much confidence as it can in the processes we use and the procedures we utilize. Next. The idea is that the processes be audited. We're trying to use industry standards in deploying a signed root zone. We want to be able to have a third party come in, verify what we're doing, make sure we comply with their own procedures. Next. And, of course, being DNSSEC and being a security-related application, it needs to be high security. There's a governmental standard in the U.S. that it's aiming to meet, the high-impact system qualification. So everything in the design has been designed to cater for security. Next. So there's different organizations involved in root signing, obviously. So I'll go through the roles and responsibilities of each of the organizations. Next. Firstly is ICANN. ICANN is responsible for the root zone in the context of being operator of the IANA functions. The IANA functions involve maintaining the contents of the root zone, day-to-day processing, (inaudible) DNSSEC. Whenever there's a change to the root zone, it's submitted to ICANN. ICANN staff processes that change. We validate that change. Once we're satisfied it meets policy, we transmit it to the U.S. government for authorization and then to VeriSign for implementation. So our role with respect to root zone signing is the same. But, obviously, there's additional tasks associated with DNSSEC. Now, one aspect that is new is managing the KSK, the key-signing key for DNSSEC. So ICANN, under this framework, will take on responsibility for managing that master key and is involved in generation of the KSK and so forth. Also new is accepting DS records. DS records are what enable top- level domains that have deployed DNSSEC to be entered into the chain of trust. So TLD operators will submit their DS records to ICANN, and ICANN will list them in the root zone, following similar procedures for current root zone changes. Next slide. The U.S. Department of Commerce is responsible for authorizing any changes to the root zone. In deploying DNSSEC, they will be involved in authorizing DS records for implementation in the root. They will be responsible for authorizing the KSKs. And they will process changes to the root zone very much a similar way that they do today. Every time ICANN makes a change that's in compliance with its policy, it will be submitted to NTIA, who will then authorize that change. Next. VeriSign's role is as the root zone maintainer. They take various changes that have gone through ICANN and NTIA and implement them in the actual file itself. They then are responsible for distributing that file to the root server operators. In the DNSSEC model, their responsibility is for maintaining the ZSK. They also use the ZSK, obviously, to sign the root zone. And they take those DS records that have been submitted to them by ICANN, authorized by NTIA, and implement them in the root zone. Next. So there's a diagram on this slide which I think is a little too small to read. But it's the basic process flow for DNSSEC in the root zone. So TLD operator that uses DNSSEC submits their records to ICANN. ICANN uses its processes to validate that information, transmits that approved information through to doc. Doc authorizes the change and transmits it to VeriSign. VeriSign generates an unsigned root zone. They -- they sign that root zone using a ZSK, and then that signed root is propagated to the root servers. And that ZSK is signed by the KSK, which is managed by ICANN. Next. So what's the approach that ICANN is taking to protecting the KSK? In the root -- in the DNSSEC model, the KSK for the root zone is arguably sort of the most critical part to protect. With a fully propagated DNSSEC-signed infrastructure, most DNSSEC deployers will be able to simply roll the DS records in the event of some kind of key compromise or key change. At the root zone, we don't have that liberty. So it's extra crucial that the KSK for the root zone be protected. Next. Another diagram. I won't go into the details here. But, in summary, there's a great deal of physical security surrounding management of the KSK. We have two redundant facilities and very -- in two very dispersed locations. Each one is in a data center which has a variety of physical security, hand scans, ID checks, and so forth. You have to get through multiple layers of physical security to get into sort of ICANN's facility. Then once you're inside of ICANN's facility, there's multiple levels of security inside there as well. No one can access this room by themselves. There needs to be multiple people to gain access to the room. Within the room, there's multiple safes, there's multiple -- multiple levels of security. And I think it's spelled out in great more detail in some of the documents I'll refer to later on. Next. One of the keys to deploying DNSSEC was the development of what we've called the DNSSEC practice statement, DPS. This is comparable to something that X.509 certification authorities already do. They call it a CPS. What it is is a very detailed description of exactly how DNSSEC will be deployed, what the key management policies are, and so forth. The idea is that this document, we can be verified against, we're very transparent about how we do things. It's all written down. And the community should feel comfortable that we're processing things in the right way and that they can verify that we're processing correctly according to this document. That's a document in draft, but it's available to view now. I'll come to the link a bit later. Next. One of the absolute keys to deploying DNSSEC is having community trust. And community trust in one way is being implemented by having trusted community representatives involved in management of the KSK. This idea is current in development, and we just published the first draft on how it might work. But, in essence, the idea is that representatives of the Internet community are involved in key-signing ceremonies, they're involved in being crypto officers, so these representatives actually have to be physically present in order for a KSK to be generated. These community representatives are also involved in recovering our keys as well. So it brings in third parties that can witness and verify that it's being performed correctly and enables the situation where, acting by itself, ICANN can't do anything that would be surprising for the community. Next. We're bringing in auditors to -- third-party auditors to check that we operate according to that DPS. And we also expect that external witnesses, above and beyond the ones I just mentioned, but other third parties will be able to come and watch key signing ceremonies in order to verify it was all done correctly. Next. So what are the DNSSEC protocol parameters that we are using for a signed root zone. Firstly, for the KSK, using a 2048-bit RSA key, the idea is this will be rolled every two to five years. We utilize the methodology of RFC 5011 to automate key rollovers, and the signatures will be based on SHA-256. Zone signing key is 1024-bit RSA, rolled every quarter, so four times a year, and the zone will be signed with NSEC, and the signatures will also be based on SHA-256. DNSkey covering RRSIG ability is 15 days. New signatures will be published every 10 days. Other RRSIG validity is seven days. And the zone is generated and re-signed twice per day, which is the same schedule that we use today. So there will need to be key ceremonies for the generation of that KSK. That will happen, obviously, when we initially deploy DNSSEC, and then every two to five years after that. With the ZSK, VeriSign will generate the ZSK, send its ZSK signing request to ICANN, and ICANN will then sign that ZSK every quarter. Next. The root trust anchor itself will be published on a Web site by ICANN. It will be in a variety of formats, including an ability for third-party CAs to sign it to improve trust level in that key. Next. So that's the description of how it's intended to be deployed. Or how are we deploying it. Next. The goal, of course, I think it's stating the obvious, is to deploy a signed root zone, using as I said earlier, transparent processes, audited procedures. And obviously to help spur general DNSSEC deployment in terms of validators, registries, registrars and so forth. And of course another goal that we have is to communicate regularly, communicate early on what we are doing, communicate often. Next. So there are a few anticipated issues that I will go through that we are keeping a close eye on. Next. Firstly is that from the beginning, we expect a significant proportion of DNS clients to set the DO bit. This means they will actually ask for DNSSEC information in their queries, even if they are not necessarily doing anything with it. We anticipate a large number of clients unable to receive these large responses, so serving signed responses might break those clients. The other issue with deploying DNSSEC in the root is that if we have people relying on DNSSEC and we need, for some reason, to stop deploying DNSSEC, it's actually very difficult because those that are relying on DNSSEC existing will no longer have functioning DNS. So unsigning the root zone at some point after launch will be very difficult to do. Next. So in order to address those two issues we identified, we have taken approach to staged deployment. Next. Firstly, we started an approach of signing the root zone, and then it's being rolled out incrementally to the different root zones. So initially, the idea is to have the (inaudible) available just on the L root, follow that with the A root, and then incrementally the remainder of the root servers with J root being last. The goal here is that initially we will have some of the root servers returning signed data and some not. It will allow the client population that have difficulty with DNSSEC to sort of round robin to a different server and get some nonsigned data if necessary. This of course relies on resolvers randomizing or somehow alternating between the root servers it uses. Next. The way we have tried to address that second problem of not being able to do a roll-back is by key employing what we have called a deliberately unvalidatable root zone. What this is is, as we're signing the root zone, at least in the initial deployment, there are not actually valid keys in the zone. They are deliberately blinded so validation cannot be performed. This gives us the benefit of testing things like response sizes so we can see how software interacts with these new larger response sizes, without the risk that the community now relies on DNSSEC being there. So while we are deploying the zone, we can swiftly roll back in case some unforeseen event happens. Next. This is what the DURZ looks like. It's basically we take a signed zone, we apply a filter to it and strip the key out. Internally within ICANN and VeriSign, we have a version that has that key available, so we can perform validation as part of our testing procedures. Next. So the general aim of this is to deploy conservatively. After all, this is the root zone. It's essential that it remain stable. And to obviously prevent that community of validators forming. So we don't have a number of people in the community relying on the signed root zone so if we need to roll back, it's not an issue. Next. One of the things that is being carefully done as well is measuring the root servers. The root servers are being instrumented with full packet captures and subsequent analysis is being performed on those packet captures. There's also an ongoing dialogue with the operator communities to assess the impact of the changes. Next. So as part of the testing, it involves testing widely deployed resolvers, testing against the DURZ, testing with clients behind broken networks, and so forth. So how do we interact with the top-level domain operators? Next. As I mentioned earlier, top-level domain operators will need to submit DS keys, DS records to ICANN in order to have their signed zones in the root zone. The idea is that the approach we are taking will be very similar to the current root zone change process. So TLD operators are familiar with how to change their NS records. The approach is very similar with DS records. We anticipate they will be able to accept DS requests at least once two months before the signed root zone is in production. It's currently a topic of discussion within the design team, but I think it's fair to say we more or less have the approach settled. We are just going through the final aspects of implementation. Next. So how are we communicating this? Well, one key thing we are doing is we have deployed a new Web site, root-dnssec.org. This is a joint Web site maintained by the three parties, and on there you will find status updates, all of the documents associated with deploying, all the presentations that we are giving archived on there, also various links and so forth. So that's designed to be a one-stop shop for everything you need to know about how the root is being signed. With regards to communicating to nontechnical audiences, this is something I think we are ramping up now. The aim is we want to try to transcend the usual audiences that know quite well what DNSSEC is, and raise awareness with non- technical audiences about what this means. So I think this is an area that more work will be done with over the next few months, but certainly our goal is to communicate with non-technical audiences as well as technical audiences. In terms of technical audiences, obviously I am sitting in one right here, I imagine. And also, you know, there's a lot of discussion on various mailing lists, be it IETF lists, non-IETF lists, specialist lists, general NOG lists and so forth. So what's the timeline? Well, some of this is in the past already. Around December 1st, the root started being signed, so we now have a history of several months of root signing already behind us. Initially, when that root was signed it was only visible internally to us but it was being signed in essentially a production manner. We certainly -- (inaudible) KSK and ZSK processing in a way that reflects how it will be done in production. Throughout January to July, there's been an incremental rollout of the signed root with that deliberately unvalidatable root zone. That's obviously a six-month process. And then the aim is come July 1st this year, we will finally have a fully signed, fully deployed root zone with a KSK generated in the public fashion I mentioned. There will be no more blinding, there will be no more temporary measures. It will be fully signed and ready for production. I want to indicate this is the draft timeline. We are deliberately not committing to these dates concretely because the whole purpose of us being deliberate in the deployment and being slow in the deployment is we want to make sure there is no unexpected consequences that are significant to instability. So it could very well be that there is some issue in the intervening months that might make us reconsider these dates. But we are hopeful that there won't be. Next. So what is the deployment status? And I will qualify this was a slide from a couple of weeks ago. These slides take a while to go through the approval process so my apologies. Next. The requirements documentation has been posted, and we have documents on the high-level architecture. We have those DPS's, those policy and practice statements. We have information on trust anchor publication. There's a bunch of detailed documentation on that Web site I mentioned earlier. We have also documented the ceremony, KSK facility requirements. There's a few documents remaining but I think in substance, most of them have been published. Again, they are at that root-dnssec.org Web site. Next. Several rounds of testing have been complete with root server operators, and we have obviously exercised that signing process between VeriSign and ICANN, so we signed the ZSK with KSK. So testing is obviously ongoing, but some initial testing has been done and it's been positive. Next. This is out of date. We now have four root servers running the DURZ, L, A, M and I, with the remainder tentatively to be deployed in April and in May. Next. So that's kind of my somewhat scatter shot summary of how we're signing the root zone. Certainly feedback on this proposal is welcome. I wouldn't say any aspect of it is set in stone, but obviously we are getting closer to deployment. But nonetheless, feedback is very welcome. And if you send an e-mail to rootsign@icann.org, it will go to the team members and also some additional ICANN staff involved in rolling out the signed root zone. Next. I just wanted to finally emphasize that this is a joint effort between three organizations, and it involves a great deal of expertise from sort of a wide variety of people. And, you know, I haven't worked with such a team that has been really involved in the process and brings so many skills to the table. So I just want to recognize all these people who have been working diligently over the last year or more on this process. It's really been a joint effort between the organizations. So with that, thank you very much, and I hope it was illuminating. [ Applause ] >>STEVE CROCKER: So do we have questions, either in the room or on the -- Let's see. I have got to look both directions here. People in the audience here? No? Do we have questions from remote participants? >>OLOF NORDLING: None in the chat room. >>STEVE CROCKER: None in the chat room. You don't get off that easily. I have a question. Go ahead, Markus. Sorry. >> >>MARKUS TRAVAILLE: Hi Kim. Thank you for the very clear presentation. I have a question with respect to the unscheduled KSK rollover from TLD operators. Because it seems that your processes are ready for scheduled rollovers, but if there is something like an emergency, what is your plan there? >>KIM DAVIES: The process for unscheduled KSK rollovers at the TLD level is obviously a new settle of DS records needs to be published in the root zone. Right now, ICANN has a 24/7 emergency number available for TLD operators. This is designed precisely for the scenario that we need to make an emergency root zone change. We will be increasing our effort to remind TLD operators what that number is in the coming months before we deploy this. But our intention is you would use that number if you need to make an emergency change. I just want to put a caveat on that, which is that ICANN as an organization can commit to having 24/7 availability. And by that I mean if you call this number, it will go to a call center. That call center has all the (inaudible) numbers, phone numbers of all the relevant staff who can do something for you, and they will keep trying to reach someone until somebody wakes up or responds. That person will then get on the case. That's how a 24/7 service works. But as I mentioned in the beginning, root zone changes involve multiple parties. (Audio cutting in and out) Those other parties will respond as quickly as we do. But nonetheless, we will do everything in our power to try to get them to be involved and to do a root zone change as quickly as possible. >>SIMON McCALLA: Hi, Kim. Simon McCalla from Nominet. Thanks for DEC. Really useful. We used that very DEC ourselves to deploy DNSSEC in the U.K. just recently so it's great. Thank you. Quick question. Have you seen any -- since you are deploying to the root servers, have you seen any change in traffic patterns? >>KIM DAVIES: I will give a very general answer. Duane Wessels, who is on the deployment team, has given a very excellent detailed presentation on exactly that topic which is available on that root- dnssec.org Web site with graphs and so forth, but I don't believe there has been a significant up tick. Obviously response sizes have increased substantially, but there has not been any concerning trends. >>SIMON McCALLA: Have you seen much of a difference -- have you seen an increase in TCP traffic as a result? >>KIM DAVIES: Off the top of my head I can't answer, but to be clear, I know L root and certainly other root servers publish live statistics in terms of utilization, so if you go to rootservers.org you can see the statistics of many of the root servers. That's public information. And with respect to the changeover, as I mentioned Dwayne has done an analysis of that time window and exactly what happened when the signed root was deployed. >>SIMON McCALLA: Great. Thanks. >>OLOF NORDLING: And we do have a question from the chat room. More specifically, from Bret Carr, asking, "What criteria would cause to you decide that things were broken enough that you would stop or roll back deployment?" >>KIM DAVIES: It's a very good question. I don't think that there's a specific criteria that we have in mind except to say obviously we have a diverse number of people on the deployment team for a start. And I'm sure if that was being considered, there would be some kind of consultation with thought leaders on the matter. I can't imagine it would be a decision taken lightly, but it's kind of one of those things we will know it when we see it, I suspect. But it's hard to quantify. >>STEVE CROCKER: Let me follow up on that because that touches very much on the question that I had in mind. You have four of the 13 root servers are now implementing this. That means that when a resolver has a DO bit turned on in its request, it's going to get back a signed but not validatable request. So it's going to see the full size and shape of a signed response. And those, as we said, are -- four of the 13 are already doing that. So it stands to reason that there's a significant amount of traffic that is already in play, and that if there were going to be something that was broken in a significant way, it likely would have shown up. And so I think it's fair to ask sort of what do we know about that process so far. >>KIM DAVIES: Well, I think with respect to when things might break in the deployment schedule, I think you're right, that we would have seen a lot in the first root signed. I think we are also looking particularly towards when the last root is signed, because right now if there is some client that doesn't accept the large responses and throws them away, it might round robin to another root server and try again. And right now it can do that and get an answer it can deal with. I think when you have more and more root servers deployed, it's more and more likely that those clients will start breaking. >>STEVE CROCKER: That's intriguing. It would be interesting to see if there's any way to observe whether or not there are clients that are, in fact, doing that so that we're not surprised on the day the last one is signed. I don't have an instant suggestion for how to do that, but it's worth thinking about, I think. What is the method for publishing the public part of the KSK? This is the key that generally will be referred to by everybody as the root key for DNSSEC and will have to be implanted in resolvers all over the world. And there's more subtle questions following up from that, but that, I think, is the first question along that path that's important. >>KIM DAVIES: Right. So I think in one of those documents on the root DNSSEC Web site, it goes into this in some -- much more detail. But generally speaking, sort of our primary place will be a specific URL, something like rootkey.iana.org. On that page will be the key. It will be, I think, PGP signed and so forth as well. It will be available for other parties to co-sign, to improve the trust in it. You know, in some respects, sharing the key is similar to sharing the (inaudible) file, I imagine, which we already do on the Web site. But I think the design team is open to additional methods that might be appropriate to sharing the key as well. But that will be the primary method. >>STEVE CROCKER: Thank you. Any other questions? >>JULIE HEDLUND: Do we have -- this is Julie. Do we have any questions from the people who are on the telephone or in the audio? >>STEVE CROCKER: All right. Thank you very much, Kim. This is remarkable development. We have one of the most significant parts of the DNSSEC process and ,indeed, one of the most significant changes in the Internet under way, and it's proceeding quietly and nicely enough to suggest that the impact will be modest, and we're on our way for a clean launch in July. I think that's just a -- the lack of hubbub, I think, speaks louder than everything else. So I think that's really quite extraordinary. Thank you very much. >>KIM DAVIES: Thank you. >>STEVE CROCKER: Markus. >> Markus: I did see a question in the chat room popping up. The question is from Jim Galvin. Is SHA-256 tested with the use of DURZ? >>KIM DAVIES: Sorry. I missed that. >>STEVE CROCKER: Is SHA-256 tested with DURZ. >>KIM DAVIES: I believe we are using SHA-256. Yes. So my assumption is yes. >>STEVE CROCKER: And another one. We're not done yet. >>OLOF NORDLING: Yeah, this is Doug barton asking is there somewhere where I can download a copy of the DURZ signed soon? >>KIM DAVIES: I believe the answer is no, actually. If that would be useful, I can't imagine why we couldn't necessarily provide it. >>STEVE CROCKER: This is as distinct from just making individual queries, whether you can get the whole zone at once. Don't you mail the root zone in an FTP-able file as a matter of course ordinarily? >>KIM DAVIES: We do, and that file today has the unsigned version, and presumably on July 1st it will have the full version. >>STEVE CROCKER: Right. But I guess it will not be too unreasonable to put a parallel file somewhere. >>KIM DAVIES: I will check into that, certainly. It doesn't occur to me that there's a problem doing that. >>STEVE CROCKER: Okay. Have we drained the queue? Then we'll shut the spigot, and that's the end. Thank you again. All right. We now move to a pair of presentations on operational issues with DNSSEC, transfers by -- a talk on transfers by Olafur Gudmundsson and a talk on key rollovers by Roy Arends at Nominet. Olafur Gudmundsson works in my company, full disclosure here, Shinkuro. You guys are presenting in the other order, though; right? No. Olafur first. These two guys have been cooperating. I'm delighted to see that. Olafur, the floor is yours. >>OLAFUR GUDMUNDSSON: Hi. I'm Olafur Gudmundsson. I -- they are bringing up my slides on the screen. Yeah. Okay. I've been working on this for about a year now, and with a number of other people. And the issue is, when we have to change domain to what is called a -- do what is called a domain name transfer, people want to know, how can I do that with DNSSEC. It is not a straightforward problem. And that's why we have the subtitle "are they compatible?" As we are going down, exploring various ways of doing this, we have run into a number of stumbling blocks. We have had a number of iterations on the design. And this presentation is the first time that we are relatively comfortable that we have got all the details lined up in the right order to be able to tell people how to develop their processes around this. So the things that we have run into is, we -- the big issue is not technical. It has been linguistical and confusion on what the different roles and responsibilities people have. And that is what I am going to try in here. So I am going to be talking a little bit first about what people have to do depending on what they're doing. And I -- this presentation is (inaudible). It is from the perspective of the DNS protocol and what you have to do so the DNS protocol and DNS answers work correctly and properly. And at the same time, we are taking into account all of the things that we see on the network, how different components of the DNS system behave. And this is designed to work around some of these behaviors. This is the goal that we had going into here. And, basically, we want to find processes that work for -- regularly for DNS and DNSSEC that you don't have any verification or resolution errors. And, secondly, when there is a change taking place, the approach we take or wanted to take is to minimize the work by the old guys, because they have the least incentive to do anything. But we tried very hard to make sure they didn't do anything. But it turns out they have to do something. From that, we have to go with the assumption that everybody is willing to cooperate. Of course that's not going to happen. But the cost of that is going to be seen on the wire. But what we need to do is to motivate people to be good citizens and participate. And also, when I'm talking in here, we're only talking about changes to DNS. All other services that may be affected in the changes are outside the scope of these presentations and our talk. And that will be handled differently. Once we started bundling that, it just got too complicated. These are the three major changes that I'm looking at in this presentation. First of all, it is the changing the DNS operator. This is a big change. This -- when I'm talking here about the DNS operator change, we're talking about changing all the name servers from one entity to another entity. A registrar transfer is when you change who your registrar is. And the third we have to look at is how do you change -- roll a DNSkey. That is the so-called KSK. And what we have taken -- we have taken into account all the things that are listed here below on the slides. And we tried to highlight in here what are the effects of each one and how they influence the timeline. Nothing in DNS can be done instantly. Nothing in DNSSEC can be done instantly when it involves changes. So for this presentation, I am going to go through what the -- what -- the notations that Steve and I have come up with. And I apologize to the people who are watching this on the PDF. I'm scrolling through this slide one bullet at a time so that people here are in anticipation to see it, but you see it also. The first entity that is going to be involved is a domain name holder or registrant as it's frequently called. But the procedures in here work both in the -- on the registration of regular domains and will also work inside organizations when you have administrative responsibility split over certain subdomains. The DNS operator, we have two of them involved in this discussion most of the time. There is the old one and there is the new one. And we assume that the operators are operating or somehow contracting out all the DNS servers while they are the official operator. The registrar is the party that you send changes to the parent. From a domain name holder, this is as far as they see into the system. They give something to the party we call a registrar. Then the changes show up in the DNS system in the parent zone. So from the registrar -- from the registrant perspective, there is no registry, because they don't know it exists. So I won't be saying the word "registry" too many times. The parent, this is from the -- also from the perspective of the domain name holder, there is some zone there where my information for my name server is (inaudible). And that's it. Then there is the content providers, that is mail services, Web services, et cetera. And we totally ignore it. In my slides, I also have color coding. And I hope that makes it out on the remote sites. The lines that are in red are indicating an error. The blue ones are steps that can be taken but are optional. If they're taken, it's helpful. Orange slides is -- well, sometimes you may not be able to avoid that. They may only affect a subset. So it is not the perfect state, but not a bad state. And, most importantly, we have green states. And those are when you have to wait for something to happen -- to take place before you can proceed. And this denotes what the condition is. Now I'm going to into the highly technical detail for the next few slides. Don't worry. I have an animation after my slides that hopefully makes it all clear. So in the DNS, we have what I call the DNS control plane. For traditional DNS, there's only one real set to worry about. That's the NS set. And that is supposed to list all the servers that are authorized for a zone. Unfortunately, DNS has a design quirk, slash, defect, slash feature, that the NS set appears in two places. They don't have to be identical. They frequently are not. The one that appears in the parent is just a hint, and it will continue to be a hint. And when you get it in DNSSEC, it's going to be unsigned. The authoritative set resides in the child. If the child is signed, then it will be signed. The DNSSEC key set record says what keys can sign the zone. And it -- and, of course, it resides in the child. But we have a problem of transferring trust from a parent to child, and that is done by the DS record, which resides at the parent, signed by the parent, and it gives a cryptographic hash of the key that the child has told the parent it will be using to sign its own DNSkey set. So far, so good. If we do look at the DNS operator chains, the new operator will create and loads a zone. The data is available, but it's absolutely not visible anywhere, because nobody knows about it. The important point in the DNS is the change -- what point can we say the DNS has changed. That is when the parent changes the (inaudible). That means the new operator becomes visible. But -- here's a big "but" -- DNS records have a GTL on them. That means a resolver that has seen the record can keep it and use it for a certain number of seconds. During that time, they can just use it and use it. So if you have a -- if a resolver has an NS already in the (inaudible), it will happily use that until it times out. And same thing. And if a resolver has an NS record for a domain, there is no reason for it to go and ask the parent until the NS record times out. So for a long time, resolvers will just keep on asking. So now let's look at this graphically. In the beginning, there is only the parent and the old operator that are in there. The parent points to the old one. Now resolver wants to talk about -- find the domain in question. And it gets NS (inaudible) and it keeps happily talking to the old operator. The owner of the domain gets a new operator signed up, the new operator instantiates the zone, and everything is hunky dunky, but nobody knows about it. So suddenly, parent flips. As you noticed, the arrow just flipped over. But the (inaudible) domain keeps happily talking to the old operator until their data starts timing out. So over time, what we are going to see is the resolver will move over to the new operator, the resolvers that were ignorant of this particular domain, when they asked them for the change, they went directly to the new operator. And when everybody has moved over, the old operator can just fade away. Well, it's not that simple. What you have is, because NS set resides in two places, some resolvers use the one from the parent, and some use the one from the child. Second thing that the resolvers do, which is very interesting, is if you get -- if they somehow learn a new copy -- get a new copy of the record -- error set, they -- that is identical, and it has the same credibility as the old one, they say, "Okay. Let me put the new TTL on the set I have in memory." So, basically, they stretch the time it can be in there. So in the case of a resolver that uses the child set and is talking to a zone where the NS settle has a relatively high TTL and there are some records in the zone of a relatively small TTL, but the resolver needs to check frequently, let's say NS records, it can -- and the name servers for the zone happen to hand out the NS set in answers just because it's helpful, this resolver can become sticky to the old operator. And it happily keeps extending the TT- -- the NS set in there, it keeps asking, even after the moment of transfer. The second -- Another complication is, after -- if you have an NS set and not a single one of the resolver -- of the name servers listed in the NS set answers, the resolver is not supposed to go and ask the parent again what is the new version of the NS set, because it's very likely it's still using NS set that it got from the parent one second ago. So going to the parent, walking through the list, it will get back in an infinite loop. So they keep the NS set until it times out. Now, let me walk you through a -- how we would change the DNS operator. This is for a normal DNS operation, not one where you are doing any DNSSEC at all. And the two parties in the beginning is the domain holder is using O as an operator. Here's a new one. Somehow, they work together to get a version of the zone for this domain up and running. We may have O involved in this process. O doesn't have to be. When the domain is ready, or the zone is ready, I should say, N gives the NS set to the old domain holder. The domain holder asks the registrar to change the NS set, and that propagates up the system and shows up. At the same time or a little bit later, H asks O to change the NS set. So we break the child sticky resolvers stretching. This is optional, and it is nice if O does this. O can also do other things to help this process that I will talk about a little later. Now, H has to sit and wait. He needs to wait for the old NS set, both the parent one and the child one, to expire from the system. And the -- how long does he have to wait? He has to wait until the TTL of the old set from the parent, the second before it was changed, expires. And he has to check -- wait for H -- for the old version to -- from O to expire also after O change it. If O didn't change it, he has to wait even longer and hope the resolvers switch over. Eventually, H has to ask O to stop the DNS service. And for the child sticky resolvers, at this point, there may be errors. But that's.... Okay. There are a number of things that can go wrong. Well, if O takes the message of saying that the DNS server is going to be moved away from him at some point in the future as saying, I can turn off service now, well, we have a total DNS failure. This happens frequently today. If O stops the service before all the resolvers have migrated over, there will be outages somewhere, not everywhere. And also it's going to be very hard to diagnose this, because to diagnose this, you have to know the state of the resolvers that the particular client is talking to. So this causes lots of follow-up calls. Well, if O doesn't stop the service, what is the impact of that? Well, this becomes a probabilistic thing. Eventually, all child sticky resolvers will migrate over. But it will take time. Similarly, if the NS set in the -- (No audio) so here are the preconditions that we have to apply. The DS set in the parent must contain the records pointing at both DS key sets -- both the operators must have the zone signing key for each other in their set. And both of these must be globally visible before the changes to the NS set in the parents are made. And what do I mean by "globally visible"? That means that all copies of the zones -- of the set must have been in the servers long enough that all old copies have been timed -- have timed out. And one of the important parts here is, data becomes -- the timer for measuring when something becomes globally visible is when the last authoritative name server starts serving up that piece of data, not when the first one. So when to start the timer has to be done by observation, not by when the command is issued. Okay. Before the operator change, this is very similar. The changes here is, we have now keys. We have to wait for them to become visible. And here at the bottom in very small letters is the formula for it. And when you take -- and when these timers start depends on operational conditions, like I said before. Now, when we change the set -- let me go back here for a slide -- a second. I'm back on 17. The old operator, we assume, is willing to accept the zone signing key. Back on slide 18. When we change the NS set to point, we can also go back in the set of asking O to change NS set. We assume that O is still signing the zone and he will keep on signing the zone, keeping signatures live, until he is told to stop service. Once the NS set has been changed, we wait for the old ones to expire. Now we can stop the service. This is where the old transfer plan ended. But here are a few extra steps. We have to wait for the resolvers to all have moved over. And after that has happened, we can do the cleanup features of throwing out the key information relating to the old operator. And these operations don't have to be done in order. They can be done at any time, in any order. Now, on slide 19 is how can the change go wrong? Well, if O refuses to (inaudible) this key for the new one, we can't transfer the signed zone. It's simple. That's that. We can still transfer the zone, but we have to do it via unsigned state. O turns off service before. Well, we have, as before, resolution failures. And when I talk about turn on service -- sorry. I heard myself in a delay in the room here. Turns off service is both stopping the name servers and also stop signing the zone. So O is always giving out expired signatures. If H cannot update the DS records, we have a really, really bad condition. There is no way the operators can be changed, either signed or unsigned. And Roy will tell you what happens today if you try this. >>ROY ARENDS: I will. >>OLAFUR GUDMUNDSSON: Okay. On slide 20. H is the integral part of this whole process. It is their decision to want to do the change. It is their desire to get things done smoothly. So we have placed the work and effort on them. And if they make a wrong choice, they're the ones causing problems. We -- O is an integral part of this process. We hope they are going to be cooperative. We ask them to change the NS set to reflect N. They don't have to do that. There are other things they can do. The easiest and best thing they can do is to slave from it. But this is a very complicated and administrative difficult task for them to do. The other thing that O can do is to just decide that to help this process is to change the TTL on these two -- on the NS set and the DNSkey set, and no other set in the zone. So the TTL of these is smaller than the negative caching TTL in the zone. This forces all resolvers basically to switch over to the new operator very quickly. And maybe this is what we should be range for operators to do when the zone is being transferred from them, is to just change the TTL on these two important error sets. Now, let's go back to the real world. Now I've been in the world of cooperation. And we assume H that knew what he was doing, which is not always the case. Given that it is in N's interest that this gets done well and hopefully has the same or higher level of clue than the domain name holder, it would be nice to be able to give N the authority to do the task that involved talking to other parties. We -- this is not necessarily possible, but it would be nice. It is quite common that domain names -- domain name holders are using an entity known as registrar to also provide the DNS service. Trying to change the registrar and DNS operator at the same time is really hard. It's almost impossible. So what -- we have come to the conclusion that the only safe and sane way to do this is to tell people, "Change your DNS operator first. Then change your registrar." The big issue is that if -- there are a number of issue in here. This means that if you -- for a bundle -- registrar that operator -- that also offer DNS service as a bundled service is that the time they start being a DNS provider and become a registrar is going to be different. So they have to adjust their systems and processes to reflect this. Okay. Now we are going to start talking about the registry requirements. And this comes a little bit back to what Markus asked about of Kim earlier, how long does it take the parent to process a request when it comes from the child? Well, we would like them to do it in near real time. But it's not going to be the case. We -- for most domains, we would like them to accept the DS records with EPP or some other automated mechanism. The registries must accept multiple DS records. If they don't do that, then a key rollover is almost impossible. It's much easier to do key rollovers with using multiple DS records than using multiple key records. If you can use both, that's fine. So we have been working with Orick (phonetic) on a number of DNSSEC issues, and we worked on getting them to raise their limit. Finally, after they gave up on having a low number, they (inaudible) up to 12, which is a very reasonable number. Also for another argument for having multiple DS records is if the children are using multiple algorithms to support their zone, we need to be able to support that. The registries can either accept the DNSkey or the DNS record. I personally am strongly in favor of the DS records. There are other people who have different religious view on this. Now let's look at the requirement for the registrars. Well, this is -- when talking about them strictly in the role of being registrars, no other role involved, they must be able to process the DNSSEC EPP extensions, the interfaces, the client or the domain name holder must be able to provide DS records, that's both at the add and delete operations. The registrar may accept the DNSkey records. It may look them up. There are a number of ways they can do this. They can calculate the DS records for the registrant. That is totally a business decision what they want to do in this. A very nice feature would be if registrar allowed a separate account attached to each domain that is for the technical contact that allows the technical contact to maintain the DS and NS records without having the actual domain name holder do this. Not everybody would use this. This account could be operating for multiple domains. But many of the communications I talked about before could be simplified, and you would get the domain name holder out of the loop. And that speeds up the processes and would get rid of many possible error cases. Okay. Here are the minimum requirements for the DNS operators in the DNSSEC context. They must accept the DNSkey records from the domain name holder if the domain holder tells them I want it. When they are asked to change the NS records, they should do it. And they should turn off service when they are told to, but not before. It would be very useful for the domain name holder to be able to call up the operator and say turn off my service, the 1123, tomorrow night. While we have been looking at this in theoretical ways and doing experiments, we have been helping the early registrars or orgs that want to be DNSSEC compliant with their systems and operations. We have been working with Names Beyond and DynDNS. We have designed a whole suite of tests that try every reasonable combination of DNS operator and registrar transfers. And so there are about 16 -- up to 16 different tests that are applied to each registrar. It depends on whether the registrar is offering a bundling service or not how many tests actually apply. And the registrars are tested both as the source registrar, that means they are the registrar of record when the test starts, and the destination registrar, meaning they are going to be the registrar of record at the end of the test. We have generated test sheets that we are now debugging through our test process, and they describe which one is in what role. And we have a very easy to remember domain names that you can use for these tests, and people can monitor from them. And the five-digit number and there is the registrar number of the registrar. And you see here that the step one of this particular test is H, meaning the holder, S which is the future DNS operator. And then we see in step two what the D is doing for himself in step two. And on the next page, 27, you see the rest of the 20 actions or 20 steps that this process has. The white boxes you see on this sheet is where the times and results are recorded. What have we discovered so far? Well, when we have been working with the registrars and the dot org, we have identified issues that should be fixed or corrected before it goes into production. All the issues have been minor. And it's just a question of having the processes go through. Future registrars and registries can benefit from looking at the results of this, and they will know what to do so they will not have the same problems. We, Shinkuro and Sparta, are acting in the roles of the outsiders and we are running the tests. One of my colleagues, Mark Feldman, is the domain name holder the DNS operator, and Sparta is providing the second set of DNS servers and I am the test monitor. It would be very nice to run these tests quickly. Unfortunately, that's not possible because org can't change it's TTL of one day. So when we change the DS record, we have to wait for one day, even though we have TTLs in our zones are 15 minutes or 16 minutes or something like that. We just totally (inaudible) off the parents TTL. The tests are being performed. We are probably in wait periods right now because we do changes in the morning and then we wait until the next day until we can go to the next steps. Now I am going into a slightly different space, and one that I may not be qualified to be in. Going from the technical world to the policy world. Everything that is in here should not be totally justified by what I said before. The registrars that are just DNS registrars, all they need to do is update their UI or interface APIs to domain name holders so they can process the DS slash DNSkey information, they have to start talking with the parent. This is relatively straightforward and is a very reasonable first step for registrars who want to be DNS compliant to only provide for external DNS operations. If a registrar wants to be a bundled DNS registrar, as a value- added service, they have to realize and reorganize their processes so that they are act being in the two separate roles that I have identified before. You. And when we get to the really important part is, what should be the transfer policies from a registrar's point of view. If a domain name holder asks for an authorization to do a transfer and this is a bundled DNSSEC operation, I think it is perfectly reasonable for a registrar to say no, you have to move the DNS service away first. Or they can say that's fine, and we will operate the DNS service for a certain time after registration is transferred. And there are other possibilities. But these are all business decisions and choices that fit within your models. Okay. What are the policy issues from the DNS protocol I think the registry should be asking itself? Again, when can the DNS signed domain be transfer. SE, for example, has the policy that you can only transfer between two DNSSEC capable registrars. Okay. Very reasonable policy. Other policy is if you want to transfer to another one, you have to go and sign first. They (inaudible) ask, say how many DNS records are allowed? Some registries are going to do sanity checks on the DS records; others are not. Simple question you can ask the registries. During a change, can we have the TTLs on the affected NS and DS set lowered? Registries can say no. They might say yes. They might say yes, but it's going to cost you more money. All of these are, from a technical point of view, acceptable answers. They have to expressly tell us what is going to be the certification process. What kind of test something going to be involved. And finally, is the registry going to accept DS records, are they going to accept DNSkey records, or are they going to accept both? You and at this point, I will accept questions. DNSSEC transfers are not as horrible as they look like when you break them down into their logical components, but if you try to do things you bundled together, things get so hairy that it is not funny. Thank you. >>STEVE CROCKER: Olafur, in the interest of time and partly because there's interplay between the two presentations, let me ask people to hold their questions for you, move on with Roy, and then we will take a combined set of questions. Thanks. >>ROY ARENDS: Thanks. Wow. Markus, if you want to be technical, you got deep technical here. >>STEVE CROCKER: We did. And before you begin, while the slides are coming up for you, let me say a word of introduction. Roy Arends with Nominet, been a major contributor in the DNSSEC over a long period of time, and has been spending time looking at the intricacies of the key rollover process. And I am very much looking forward to this talk. >>ROY ARENDS: Thanks, Steve. So earlier this year, operators of the RIPE NCC witnessed a massive increase in traffic to the name servers. What they thought was an apparent attack turns out to be something -- turns out to be a phenomenon that has the potential to grow into a perfect storm. And if it does, it has a similar effect to the China syndrome. Now, I'm not an alarmist. This is not theory. This is actually happening right now. And yes, we are at Nominet just deployed DNSSEC. I did this work together with George, Geoff and Patrik. My name is Roy Arends and I work with Nominet U.K. So this is what they saw. On the left-hand side of this graph, you will see a diagonal pattern about 12 megabits per second coming into the name servers. And then instantly, mid-December, it more than doubles. You have 20 megabits per second with 30 megabit per second peaks. So when they look at the traffic, it turns out to be DNSkeys. DNSkey queries coming in, DNSkey responses coming back. This is a fantastic DNSSEC amplification attack. If you think about DNS amplification attacks, it makes use of the fact that responses are larger than queries. And DNSkey is among the largest that you can get. In the RIPE case, the response is about a thousand bytes. If you take the minimum of this graph, which is about 2,000 queries per second and you multiply the two, you get a whopping 16 megabits per second. And this is additional loads to the load that they already had. If you take this an order larger, you can actually wipe small TLDs off the planet. Now, who does this? So they took ten minutes of query traffic, noted down every individual IP address, and plotted this into this graph. In blue you will see the amount of IP addresses. And on the left- hand side you will see a very large blue bar. This is what you expect. This is a lot of name servers -- sorry, this is a lot of resolvers asking for one, maybe two queries per ten minutes. And on the right-hand side, you see an enormous bar and that bar will tell you at about 60 -- you see the small blue bar here, about 60 IP addresses asking individually more than 2100 queries in this ten-minute time frame. More than 2100 queries per individual IP address is massive amount. Now, what was so special about that day? If you type the name in or if you type the date into the RIPE Web page, you will instantly see the DC page. And the DC page is the RIPE's DNSSEC deployment page. And it has this nice message. On the 16th of December, current deprecated keys are removed. There is one key in use. So this might not be an attack. This might simply be a misconfiguration. So why were there so many clients? Where 60 instances in one instant, did everything completely wrong. The answer came one month later. This is one month after the roll. There is one operator who filed a message through the fedora bug reported to the site. He got 240,000 log messages in 24 hours. You think about this, this is about 10,000 log messages in one hour. That's about three per second. I can't fight that fast. And if you get this in your log, that means that Bind is working very, very hard. He also mentions the CPU time was massive. CPU consumption time was massive. The memory was swapping. The traffic was massive to the upstream name servers. And it turns out to be when you look deeper there was a small tool called DNSSEC Conf which is basically a bunch of scripts and it came with a trust anchor file, and this is the same trust anchor file that RIPE distributes. But obviously, RIPE distributes this file and you need to update it regularly. But this tool didn't update that. So you have a static file that on the 16th was unusual. It completely expired. You. So obviously we found a fix for that. Just switch off this tool. And we were hoping that traffic load went down. And it did went down a little bit. This is a graph from just last week. It went from February the 3rd when we found this problem to March the 3rd last week. And you see that it declines a little bit. But it just doesn't decline fast enough. Why doesn't it decline fast enough? We have another rollover in June; right? And if you look at the past history, you see that this was not the first time -- December was not the first time that we see a massive increase in traffic. If you remember that very first slide that had 12 megabits per second incoming stream. This is from a different perspective. This is from perspective from Japan into the ethnic name servers who also run part of the RIPE DNS server (inaudible). If you look at this traffic, the same incident happened in June, and the 12 megabits per second is probably an enormous amount of DNS queries coming in from the June. So this is the key rollover in June. They removed the key there. This is the key rollover in December. And in September they added a key. The key rollover works as follows. In September they add a new key, in December they remove the old key, In march they add a new key and June they remove the old key. This is how the rollover happens. You add a new key and somewhat later you remove the old key. Was this a one-off event? Obviously there are more deployments in DNSSEC than just in (inaudible) and ARPA, the reverse tree. So we asked Sweden and, yes, Patrik came up with this slide. Photograph this is Sweden in June 2009. This is Sweden in June 2008. So this happens every year when they do a key roll. Now the traffic here is not so massive. It's about an instant increase to 60, to 80 queries per second. 60 to 80 queries per second translates depending on the key size, translates to about one to one and a half megabit per second. Now, that's not so -- I mean, for Sweden, they have a massive Anycast deployment. So one and a half meg built per second to them is noise. But what is so significant in this slide, this traffic is caused by only two resolvers. The first resolver was fixed here in the end of June, and the second resolver was fixed here in July 11th. Now, this massive amount of traffic was caused by only two resolvers. So how is it possible, how is it possible that one resolver or two resolvers can send so many queries? Resolvers are supposed to be nice guys. They are not supposed to be aggressive. They are supposed to cache their data. The reason is if there are a massive amount of resolvers out there and only a very few servers for TLDs in comparison. So we researched this, and early February we found that there is a bug in all versions of Bind 9. And we call this bug the depth first search problem. And it goes something like this. You have a train of trust; right? And the train of trust starts with the trust anchor that you have configured. And in this example, I will use root. So you have a DNSSEC configured in your resolver for root. In comes a query for www.dnssec.se. You query it, you do your subsequent resolving and you get this address with the signature. You want to validate the signature. How do you do that? You just fetch the key. But you can't trust the key yet. So you get a referral from the parent to the key. It's called a DS record. The DS record blesses the key from the child. That DS record is signed as well with the DNSSEC key which resides on the same name server, and the same for root. It blesses the SE key, which is signed by the root key, and the root key, and here we are in the end of the validation chain, the root key should be equal to the trust anchor. And when all this is equal, Bind is a very happy camper. When this is not equal, when the trust anchor is different than the key -- and by the way, this also holds when the DS record mismatches the key or when the signature is expired, it will cause a depth first search problem. The numbers you see here on the slide is the number of name servers that this information resides on. Notice that for root you actually have 20 name servers. This is because root has 13 IPv4 addresses and 7 IPv6 addresses. Combined is 20. For Sweden, the same. They have 13 name servers. And for dnssec.se there are three name servers. Even though this is an example, the numbers are right. When Bind tries to validate this, it goes completely berserk. Because there's a mismatch in the trust anchor, it will try all 20 root name servers for the right key. And if it can't find the right key, it will take one step back. It will go to the second name server for root to find a proper DS record. When it has, it will do it again, all 20. Of course, still the same problem. It will go one step back. The third name server for root asking for the proper DS record, and it tracks all the way back. And if it's done, you get a whopping 600,000 queries to the root name server set. Of course, you will never get your name server to create 600,000. There's some limit you hit first. Bandwidth, CPU time, memory swapping, locks in disk. So we reported this problem to the ISC, the writers of the Bind code in early February, and they acknowledged the problem immediately. And they told us this needs a fundamental fix. This touches the core of Bind. This needs a fundamental fix. This is an all Bind 9 resolvers, all Bind 9 versions. So when we patch this, we are going to send out our patches in one big bunch, and we need thorough testing. Completely understandable, very good approach. A few days later, this is because they already committed to a release schedule, a few days later they release Bind 9.7 with this bug, and there is one slight issue here because this is the very, very first version of Bind that can validate the root. When the root becomes available, and this is because this is because it's RSA SSH2, Bind 9.7 can validate that. But luckily, there's trust anchor rollover. 5011 deploys in 9.7.0. Now, the message that came with that was a little bit naive. There's a message in that code that says exercise caution because you might trigger a problem. Exercise caution. When you put this on the gun, when you put this label on a gun, exercise caution, obviously we have no more accidental death. I think we should put this message on a car. If you are in a traffic accident, you probably haven't read the label. Anyway, what surprised me was last week, only last week, a few days ago, they released Bind 9.6.2 and it has the same bug. It has also the same warning. And 9.6.2 is special because they have the root validation, RSA SSH2, that ported into 9.6.2. So it can validate the root, and it has this bug that has absolutely no 5011 implementation. You have to hand configure that key. You have to, by hand, roll this over. This is a time bomb and I think it was highly irresponsible to release that code. They announced the patch as soon as possible. We're still waiting for it. Meanwhile people deploy 9.7.0 and 9.6.2. By the way, the 9.6 branch is among validators, among people who have implemented DNSSEC in their resolver setup, the most popular server. So let's talk about this perfect storm that I mentioned in the introduction. The perfect storm basically requires a few things to happen at the same time. But at least one or a combined set of what I am about to say will cause this perfect storm. This is when you configure accidentally a DURZ key in your resolver. You see a massive amount of queries instantly going out and this is a small fraction that you show on the screen. Then we have no proper automatic trust anchor rollover. There is a bug 9.7.0. The bug won't be triggered currently, but if you use views it will be triggered, and they promise a fix in 9.7.1, but they have no automatic trust anchor rollover planned for 9.6.2. Then we have DLV mishaps and DS record mishaps. Remember that Puerto Rico, they also deployed DNSSEC, and at one point in time, ISCs, DLV decided to use the keys embedded in the ITAR in their DLV setup. I know this is a lot of acronyms, but basically they were -- they were provisioning their DLV system using (inaudible) in the ITAR. Now, P.R. roll over their key, and it was present in the ITAR. The ISC's DLVs pick this up only once a week, and there were five days between the key rollover in P.R., and thus the key rollover in the ITAR and DLV setup. So users of DLV did not see P.R. for about five days. Now, that's significant, especially when you are Puerto Rican. [ Speaking too quickly ] What is more significant to me currently is that this DLV problem, we did it in lab setup, caused also a massive, massive packet storm, because DLV is just another trust anchor. DLV is just another chain of trust. And then we have a multiple trust anchor problem. And it gets a little bit complicated here but it basically means if you configure a DLV trust anchor in your resolver, that it trumps the trust anchor for root in your resolver. It thinks that trust anchor for your TLD, for instance, for Sweden, or for the U.K., is more important than the root. But this also means that an invalid trust anchor for a TLD to a linked trust anchor for, for instance, Sweden or the U.K. is more important than a valid trust anchor for root. Now, if you think about it a little bit more, you have a doomed scenario; right? Currently, we have DLV deployment of DNSSEC. Sweden has it -- Okay. I will slow down a little bit. Thanks. That was Bill Manning, by the way. So this is my doom scenario. You have TLD deployment right now. You have .SE keys being configured, we have (inaudible) being configured, sometimes goes wrong. And, hopefully, in due time, everyone will have resolver -- everyone will have trust anchors in their resolver. Now, imagine this. You have configured a Swedish trust anchor. And then root becomes available. You configure that trust anchor. Naturally, it's the logical thing to do. And then the Swedes decide to use the rollover through DS records. So they're not going to announce that they're going to roll over the key. Why should they? They can do it now up there the root. But, of course, if you don't remove the trust anchor for Sweden, it will automatically become stale. And if it becomes stale and the root key doesn't work, you have triggered the same problem. So let me go over this series of unfortunate events. We have buggy software. I don't mean bind 9 here. What I mean is things like DNSSEC (inaudible). All well meant, but not well thought through. And then we have DNSSEC deployment at root. Now, in itself, this is, of course, not an unfortunate event. It's a very fortunate event. But it will amplify this problem. We have the multiple trust anchor problem. This is actually a protocol spec. But it's going to be very hard to change that. And then we have no proper 5011 deployment almost anywhere except for the root, and they're not really signed yet. And then we have opportunistic (inaudible) scraping by DLV registries. Now, this all combined can cause the perfect storm. But there's another thing I really want to talk about. That's the necessity of rolling over a key itself. When Markus and Steve asked me to do this presentation about key rolling, right, this was before we realized that there was a real problem. And we did some thinking about the necessity of -- the necessity of rolling over a key. And we've actually talked to a few cryptographers and a few people who really, really know their crypto. So it turns out that a lot of people suffer from frequent key rollover syndrome. Right? This is the urge to roll over your keys every day, every week, every month, twice a year, right. It's just roll over because you can. I call that frequent rollover syndrome. And the advice seems to be rollover as often as you can. The advice was actually misguided, because it stems from the idea that if you generate enough signatures with a single key, you might actually leak the material. Now, for RSA, this is not true, and in fact it's proven not true. So with RSA, you can send -- you can generate enough -- as much signatures as you want. You won't be able to recover the key from the signature (inaudible). Now, when you explain that, you get another reason. The intention is to mitigate fallout due to a compromised key. Now, think about this. You roll your key once every year. You roll your key once every year. Then if the key is compromised, let's say, three months before that, it's okay, because three months later, we're going to roll over the key anyway. There are three months long a problem. But this is not true, because there is no perfect forward security. In 5011 or in trust anchor rollover, you use the (inaudible) key to bless the new key. The old key signs the new key. So if the old key is compromised, you don't have to roll over. The guy who compromised your key can do the rollover for you. He will introduce new keys for you. So this is actually fairly bad. And the last thing is, if you compromise the key, (inaudible) one year, you can compromise the key in half a year for twice the cost. Now, in general, this statement doesn't scale that well. It has to do with the -- with the way that crypto analysis works. But for this number, for this time frame, it's this. I mean, if you can do it in one year, you can do it in half a year for twice the cost. And when you explain that to people, you get -- slowly, you get, like, other reasons that are really crazy. You want to practice key rollover, because, otherwise, your operators might forget how to do that. Now, this is crazy. You don't go into a nuclear power plant and just switch over to plutonium just to help operators. What you do is you test this in a virtualized environment. If you really want to exercise, for instance, things like emergency rollover procedure, you get your guys in when you do a ZSK rollover. ZSK rollover is something that has no outside dependence. ZSK rollover, you can do that in the privacy of your own registry. In fact, never, never actually mess with a critical production system. So never mess with a critical production system. In fact, -- is this working? This is working. Sorry. In fact, at Nominet, we have a very stable -- I think we have a very stable production (inaudible). We see it as highly critical. No one messes with that. Next to that, we have a (inaudible) system where we can test stuff and we have a preproduction system where we can test new software. This is how you should deploy DNSSEC. Don't just overnight put changes in there. Don't have operators test their own knowledge on the new system. So what's the solution to this? Fix the buggy software already, all right? And stop -- ISC -- Suzanne, if you're on the phone, stop releasing bind 9 on with bugs. And meanwhile, this is to help ISC, help fund bind 10, which is a new version of bind. We support it, with a number of other registries and number of other people. It's a fantastic product. I've seen the light of it a little bit. And it works very well. Actually, that -- don't roll keys too often. Be practical. For instance, if you look at browser certificates, know you can compare this one to one. You can also compare this one to one. But if you look at browser root certificates, they're in there for ten, 20 years. It works for them. And that's more often used than DNSSEC at this moment. And do not endorse (inaudible) of trust anchors when your parent is signed. When the root is signed and you're in the same political environment, you don't have a problem with the root being signed, right, just (inaudible) keys there as a TLD. Don't put them on Web pages. Don't use 5011, don't use ITAR. Don't use DLV. (inaudible) it all through a signed parent. Because when stuff goes wrong, you can blame mom and dad. Right? Then the last thing, when the parent is not signed or when you don't agree with the way it's deployed or you can't, in your own political environment (inaudible) for that, only then use proper 5011 and use ISC's DLV. One more thing I want to cover here. Of course, at Nominet, we do roll our keys once every three years. But that's completely a different reason. This has nothing to do with the cryptographic aspects. This is because we use hardware signer modules, HSM. This is a piece of software in any production environment when you buy a piece of software, you need to write it off. We write it off, like any other hardware, in about three to four years. The thing with an HSM is, you have a cryptographic key in there you don't want to ever get off. We don't migrate the (inaudible) keep to the new HSM, whatever that is. It might not even be possible by that time. So we decided that we roll over the hardware, we roll over the key at that time. Now, this was my presentation. If you ever -- if you have deployed DNSSEC as a TLD or as a large operator, you have kept graphs during your key rollover, I'd love to see them. You can actually find research on the URL on the screen. And if we still have time, I don't know -- if we still have time, I think we now take questions for Olafur's presentation and my presentation. >>STEVE CROCKER: Thank you very much. Speaking personally, I'm delighted with this presentation. I think this is a very specific and important topic. I have to hold myself back from diving into this with my technical hat on and play chair here. But I -- if we don't have time here, you will definitely hear me go into all the details with you offline. What -- Do we have questions from people here for either of these presentations? The chat room maybe? >>ROY ARENDS: Steve, I think I put everybody to sleep. >>STEVE CROCKER: Not me. Let me pose one question to you. You make the point that there's not necessarily cryptographic necessity for rolling keys and that if there were a cryptographic breakdown, that doing a 5011 rollover wouldn't be the right thing to do, you'd want to do some sort of emergency rollover. My intuition in this area isn't driven by exposing the key or the fact that it gets used a number of times. I assume we have a huge amount of headroom in that from a cryptographic point of view. But I have a much simpler and more intuitive issue. It seems to me that if one is ever going to roll a key, then one has to do it often enough to get some practice so that at the time you do it, it works. So in my mind, I have a -- either frequently enough or never, but not so infrequently that when you have to do it, it doesn't work. And that's the question. Now, if you argue that we will never, ever have to roll the key, then that's a consistent point of view. But I - - I have real trouble with that for a number of reasons. For example, eventually, we're going to want to change algorithms. So if we're in the position that we're going to have to eventually roll the key for any number of operational reasons, not necessarily just the cryptographic lifetime, then the other factors, operational factors, human factors, and so forth, memory of the people involved, seem to me that one is going to have to do it -- and it's also related to product lifetime and deployments and so forth. But there has to be a nominal rate so that it's a practiced and regular process and doesn't fall into this trap of somebody configured the key, it's stale, it's baked in, they've left, and nobody knows what to do about it and all of a sudden things break when the day comes. >>ROY ARENDS: Thank you, Steve. I'm going to use the other mike. So I didn't actually want to imply that you should never roll the key. I think key rollover has its uses. The advice that currently exists is key -- is, for instance, in RC 4641, I think it was something like roll your key monthly for -- for this size and roll your key yearly for this size. Now, that, to me, is crazy talk. I think you need to roll the key when you need to, not when you can. And, of course, I do agree with your point of operator experience. But I would suggest to stay off the production system as long as you can and practice in a virtual environment. I also urge that when you -- And, of course, just like in a medical hospital, for instance, you want to test the diesel engine often enough so you get power when you need to. So in that sense, you actually do already key rollovers every so often with your ZSKs. With your ZSKs, that's actually the same operation from a protocol perspective. Because the protocol doesn't make the difference between a ZSK and a KSK. It's just that the operator uses them differently. If you do a key rollover, right, the same variables, the same properties appear for a ZSK and a KSK. The only difference is, the KSK has outside dependencies. And the outside dependencies, that is what is causing this bug, what is triggering this bug and what is causing this problem. So, in general, my advice is, be practical. Roll over when you need to, right. Take into account the size of your key, right. Educate your operators, educate your internal staff by using a highly virtualized system. And have them witness an actual key rollover when you do them. So I hope this answers your question a little bit. But I'm very happy to take this discussion offline at some other point. Thanks, Steve. >> Steve, Simon from Nominet. I'd like to add to what Roy said, which is, one thing we did quite a lot of, set up a virtual environment so we could practice this, we could try and test. It's not expensive. It's not difficult to do. And we'd be very, very happy to share that experience if people would like to understand how to set up an environment where they could practice key rollovers, practice setting up the signing. I strongly urge people to consider making sure they've got a really solid test environment before they embark on this. Thanks. >>STEVE CROCKER: Thank you. I think the idea of setting up an environment is a very important and might even say should come as a standard part of the package, another personal opinion. There's a question in the chat room. Or is that what you were going to -- Okay. Let's see. Actually, I should ask you to do it, because this is a Dutch name. >>MARKUS TRAVAILLE: This is Markus, I have been watching the chat room and there's been a lot of discussion about both the transfers and the -- Roy's presentation, and there are also some questions arising from it. One is from (saying name) in the Netherlands. He says that Roy mentions all the issues as being caused by unintentional misconfiguration of resolvers. But what if this is exploited with malicious intent? >>ROY ARENDS: I think the problem doesn't discriminate between malicious intent and misconfiguration. So you will see the exact same traffic. I think it's actually very difficult to exploit this. To maliciously misconfigure it. Because you have to hack into a resolver. Because if you hack into a resolver and misconfigure that key maliciously, and you can do much more than harm than -- you can actually do much more harm then, you can take over all the traffic and just do the misconfiguration. So unless you, of course, want to have a denial of service attack on the roots. So the only thing I can think of right now -- and this is from the top of my head -- that if there is a known resolver out there and the I.P. address is highly published, it has a known misconfigured key, then I can see easily a botnet sending queries to that resolver, which then, in turn, denial of service other DNSSEC deployments. I hope this answers your question a little bit. >>STEVE CROCKER: Are there questions on the phone? >> There's another question from the chat room for Olafur. On slide 14, you -- that's available to put it on the screen? >>STEVE CROCKER: The question is from Rick Wilhelm at Network Solutions, I believe, on slide 14, you mention the distribution of TTLs, the variability of them, the median, et cetera. Is that data available? Do you want to take that, Olafur? >>OLAFUR GUDMUNDSSON: Yeah. I pulled that data down by querying all the TLDs yesterday. I will put up a slide after we are done with that somewhere and post it to the mailing list. If you give me a few minutes, I can probably bring the chart up right here on the screen. >>STEVE CROCKER: I think that we ought to do this in the -- you know, sort of continuing mode rather than try to get everything resolved here right away. It's also the case that the -- it's the maximum TTLs in a given situation that govern this behavior. Are there other questions? >>MARKUS TRAVAILLE: There's another one from the chat room. Olafur, how do you expect to get buy-in from the registrars? >>STEVE CROCKER: And that's from a registrar. >>MARKUS TRAVAILLE: That's from a registrar. >>STEVE CROCKER: Let me take you off the hook. Because Olafur and I have had some very strong discussions about this. And Olafur's perhaps more skeptical than I am. The -- so here's the plan. We'll operate with full transparency. First step is to lay out, from a technical perspective, a feasible scheme by which these transfer processes can work, and work in what we call a ripple-free fashion, so that there's no loss of resolution and no loss of validation. That's a very strong goal. It's a very high standard. And we believe we have worked out how to do that. And we believe we have worked out what some of the obligations fall on the different parties in order to make that work. As (saying name) asks and as this question implies, well, that's fine if we lay all that out. But how are we going to get the buy-in from the various parties? In my mind, we are going to move through a phase in which we're going to document this more clearly, get this more thoroughly shaken down and provide information. That's the next step. Then, as part of that, but also continuing as a separate venture of its own, if you will, is to go through the socialization process. I can't guarantee. I don't have 100% expectations we'll be successful. But I think that if we lay out what this behavior is, get it documented, and get it implemented in ways that really are not too onerous and establish the corresponding labeling that goes with it so that users and managers and policymakers and so forth can see quickly whether or not -- it's not just registrars, but it's also DNS operators -- are or are not adhering to the specific set of rules, that has the potential of being widely adopted. And I would say it's not entirely an onerous process. Each operator, each registrar, has to -- has to implement something when faced with losing a customer, which is basically what we're talking about. And in our experience so far, we are seeing a variety of implementations. And I would say, with no disrespect -- it's just an opinion -- that many of those implementations are not with a great deal of thought. They seem to be sort of reasonable from a local perspective, but not driven by deep conviction that this is the way to do it. And I suspect, and I'm hopeful that if we lay out a -- a specific regimen for how to shut down smoothly, how to transfer smoothly, that that will be viewed as a contribution and not just an imposition. So that can trigger some debate, I would imagine. >>MARKUS TRAVAILLE: I have a comment from (saying name). He cannot speak, so I will speak on behalf of him. He says why. The question is why. So he says basically there's no demand for it. So that was the main question he wants to ask, if I am correct. >>STEVE CROCKER: We'll stimulate the demand by saying to the people who are dependent upon this, that is, customers, look for the following capability when you are looking for an operator or looking for a registrar that is supporting DNSSEC. Whether that will be successful or not, we'll have to see. But that will be an intimate part of the process. Good. All right. We're short on time. We had scheduled a break. But I think that rather than take a formal break, people who are here in Nairobi can drift out, get coffee as they need. For people who participate remotely, we won't pay any attention to what you're doing. With that, let me turn the floor over to my long-standing colleague and friend, Russ Mundy, from Sparta, overview of open source tools for DNSSEC. And I hope I have given him enough time to take his microphone off mute. We're not hearing you, Russ. Let me ask you to hold up, Russ, for a minute, while we get the audio taken care of. Meanwhile, I have overlooked a request from Suzanne Woolf at ISC to speak. Suzanne, can you speak? >>SUZANNE WOOLF : Sure. Can you hear me okay? >>STEVE CROCKER: Yes, perfectly. >>SUZANNE WOOLF : Terrific. I just wanted to make a brief comment, and I don't think this will surprise Roy, since he called me out by name. But just to sort of close the loop. ISC is aware of the concerns Roy has been discussing today and in other venues. And as he knows, the fix for the problems he is describing is under test now, will be available well before the final signing of the root. In fact, will be available in the short term. Just will not be available until it's been thoroughly tested. We don't want to be in a position of hastily releasing fixes to longstanding problems that might further risk making things worse. We have to respectfully disagree with his assessment that this is irresponsible, given the sequence of errors or malicious activities that have to occur to trigger the described problems and the limited deployment of DNSSEC to date. But the take-away that I don't think people are getting is that we are working to make patch code available as soon as we reasonably can without delaying the other features and bug fixes our user base also needs. If anybody wants to test that code, it is, we hope, ready at this point, and we are happy to discuss further participation and testing in helping us get that code out there. Thank you. >>STEVE CROCKER: Thank you, Suzanne. >>ROY ARENDS: Thanks, Suzanne. Yes, I absolutely realize that ISC is working very, very hard. And I completely understand the necessity of testing these patches. And I know you guys are doing absolutely your best to get this problem fixed as soon as you can. I remember in early February that you made an announcement that you really are working hard on this. And that within a few weeks, patches should be available. Now, I'm not going to discuss here, because -- I mean, I am not going to discuss here the impact of the problem. I already did that during the presentation, and I feel it's a little bit inappropriate now to go deep into the presentation again. However, I do want to say that the issue that I have is releasing -- sorry, sticking to your release schedule and then releasing, for instance, 9.6.2 and 9.7.0 knowing that it has a bug inside. I think it would be very, very easy to just hold off releasing those two versions while fixing this bug. I think it's just a matter of priorities. And I think we have maybe a fundamental disagreement on that. But let's just disagree on that. Thank you. >>STEVE CROCKER: Okay. I think that we have had what the diplomats call a full and frank exchange here. And I have known Roy and I have known Suzanne for long enough that I can't quickly recall how long and have a high respect for both. So this is an important conversation, and I'm sure it will continue in other venues. Russ, can we hear you now? Try again. >>RUSS MUNDY: Okay. Test one, two, three, four, five. How is that? You are soft but very clear. What can we do about the volume? Okay. Plunge into it, Russ. Speak as loudly, but as clearly and slowly, as you can. And now the floor is yours for an open view of open source tools for DNSSEC. >>RUSS MUNDY: Thank you. For those who know me, a volume increase is not a problem for me. So I will turn it up. So let's just dive right in. Slide two, please. What we have here is just a simple example of a DNS query, feeding the zone with data and then a query to get information out of the zone. This is what, in many ways, DNS itself is all about. And what I'll be talking about today is how various tools exist to fit in, that permit you to do DNSSEC in this environment. Now, one of the things that you see on the next slide, slide three, is when you look at just the query, and one Web page you think generates one query. So let's see what happens when the slide goes to the next illustration here. You get a picture of how many queries actually get generated to fill one Web page. And so this is really rather dramatic. And unless a person has actually sat down and looked at what happens when you use various applications, you're completely unaware in almost every case how much work DNS is actually doing behind the scenes. And of course when DNSSEC is involved, it must be fully integrated into that. Next slide, please. And so when you look at what all the pieces are when you are dealing with DNS, this is the different view of just what has to happen with one zone. And the first view with the pictures of the people and the content and the servers. This is another way to look at it, where on the left side of the slide, the data, the information that actually has to go into a zone gets put in. Somebody says, "I have this data. It needs to go into a zone. It's my zone. I am going to put the content in." When I get the content inside, it goes into -- as a regular user, it often will go through a registrar, then it will get to a registry, then it will get to a name server, and then applications make use of it. And golly, there's this human being way at the other side. So if we hit the space bar on that slide, please, and you see the arrows show up. And it has to pass through many pieces of software. And, often, a number of different people and processes. So DNSSEC, again, gets integrated into all of this. Next slide, please. So when you get DNSSEC integrated with the people in boxes from the first pictures, in the simplest form what you are really doing is you are adding signed signature verification information that is used eventually by either a validating recursive name server or a client to ensure that they got the correct information that the owner of the zone wanted to go in. And that's really what DNSSEC is all about; to make sure that you, in fact, get the information out that the provider of the information put in in the first place. So if we go to the next slide, please. Today, there are a wide, wide range of tools available. I have got a group of categories that I will just go through very, very quickly. Won't really talk about any of the tools. Will flip through the slides quickly. But I will try to point out where the pieces fit and how they fit together. And so with all of the tools that exist, most of the holes are covered with something. There are places where more tools would be helpful. But for folks that are deploying it today, they are finding that for the getting the zones fed part, it's looking good. The other parts, we still have progress to make. So let's go to the next slide, please. Next slide, please. Thank you. Name servers. This section is really dealing with primarily authoritative name servers, the name servers that provide data to the Internet, the running machinery part. That information has to get all fed into it, but the name servers themselves, in some ways, are the heart of the data source for DNS. There are several there. There are two on this slide that aren't open source, those are the Nominum products, but the others are, indeed, open source. Next slide, please. Thank you. Key generation. This is a set of tools needed to actually create the DNSSEC. The next one is rollover, a set of tools for changing the keys and rolling forward with your tools -- with your, I'm sorry, DNSSEC information. Next, please. Hardware interface. There's several of those in existence. Oh, we jumped to troubleshooting. That's fine. We will keep rolling here and push on through them. Anybody that has operated a DNS operation knows that once you get things up and running, you have to do troubleshooting. These are a group of tools that are available for both troubleshooting with DNSSEC and also we found there's some very good use in the general DNS world. Next slide, please. So the next area is when you have -- when you are dealing with a parent, almost every zone in the Internet, with the exception of the root, has a parent, and there has to be relationships between the parent and child. Next slide, please. And so data has to be exchanged between parent and child. And so there are some tools for the exchange of information in this space. This is a very, if you will, loose space. There is nothing from a standards perspective that tightly defines how you go about doing it. And so there are various ways that it can be done and the information itself can be different. But you do have to exchange information. Next, please. Update to parent, more the following rolling of your information in DNSSEC. Next please. Next slide, please. Thank you. Validation. This is for your validating resolvers, that folks like Comcast are out there running today. The fetching of key information has become less important since there's been the big emphasis being placed on getting the root signed, but there are tools available for obtaining information that exists out there. So these tools do exist and are freely available for use if desired. Next slide, please. Next, please. Again, troubleshooting in the resolution side. When you are trying to identify why something isn't validating properly, you must have some set of tools to help work through that. That's what this collection is. Next slide, please. And now we get to applications. Next slide, please. And there are a range of applications that have been instrumented to be DNSSEC capable. Actually, open SSH was the first application out there, and it's been distributed for quite a while, and there are a number of other extensions out there. There are Firefox extensions, there are also Web browsers available, mail is available, mail transfer agent. And the list does continue to grow. At this point it's largely used by folks that are used to dealing with emerging software, but it's becoming more and more mature as we go along. Next slide, please. Developer resources. I will just ask Julie to flip through these. There are a range of them for doing validation and development tools, software development kits, APIs and so forth. And so those are available. There's also some testing resources. And Roy has some support in that space from some work that he has done over time as well as others. Next slide, please. And deployment aids aren't exactly software, per se, but there's a set of documentation that does help provide guidance in how you should go about actually deploying and making use of this. And in the spirit of open source software, it is all open, freely available, can be picked up and used as you desire. Next slide, please. Okay. So how will some of these fit in? I have used a number of the tools that come from the tool kit that Sparta has developed and distributed as all open source, all free. And these are some places where the holes are plugged with those tools. I use that one just simply as an example of where you can pick up existing software and make use of it to actually accomplish DNSSEC for both the authoritative server, the recursive server, and clear down to the application, if you desire. Next slide, please. The survey of existing tools, and we try to keep this up-to-date, it's a long URL. We promise to try to do a better job of making it shorter, but for now that's where the survey information is and all data that's contained in the tables in this presentation is in this survey. So if you want to see more details, you can look at that URL, or you can pull these slides. Next slide, please. And a final set of resources that you can look at and collect information from. And with that, I will jump to the end. And I doubt that there's time for questions, but if there are, certainly willing to take any. >>STEVE CROCKER: Thank you, Russ. Questions from the floor here? >>ROY ARENDS: Not specifically a question, but just a comment. I have used the tools from Sparta before, and I have to say they work very, very well. Thank you. >>STEVE CROCKER: Thank you, Roy. >>RUSS MUNDY: Thank you, Roy. I appreciate that. >>STEVE CROCKER: I'm not seeing any questions in the chat room. Any questions on the phone? Russ, thank you very much. As you know, the work that you and your team are doing with these tools and that in general the community is doing providing these tools is having a pervasive and substantial effect in providing a basis for the rest of the community moving forward. So that's great. We'll move now back to Roy Arends from Nominet on open DNSSEC. >>ROY ARENDS: Thank you, Stephen. I will try to keep this as short as possible. My name is still Roy Arends. I still work at Nominet, I hope. [ Laughter ] >>ROY ARENDS: And this presentation is about open DNSSEC. Next slide. So we've built open DNSSEC together with partners as a complete DNSSEC solution. It's basically a black box, right, in comes signed zone, out comes signed zones. There's a piece of software that you can install on most popular (inaudible) distributions. And in the middle of the file, you will see open DNSSEC, which does the automatic zone signing. It's basically install, configure, walk away. Next slide, please. It has three major components. I will go over them in more detail. But, basically, there's an HSM, which is the hardware security module. There is a KASP, key and signing policy. And it has the signer itself. It's actually that simple. Next slide. So what's an HSM? We really want to use an HSM because you can store the keys in hardware. Basically, how it works is you ask the hardware to generate a key for you. Subsequently, you ask the hardware to sign RR sets for you with a key that you just created. So why would you use one? The cool thing about HSMs is they're designed to never let the key go. Even though this thing gets physically ripped out of a system, it still -- still guaranteed that a key doesn't come off. These things are certified, certified on different levels. So you can get them level 1, which is not really that secure, level 4, even if you point at it, it will delete the key, that's how secure it is. Other uses of HSM, if you -- it's used in the banking industry. But it's also used, for instance, in SSL termination. You really want secure environment and speedy cryptographic processing. HSM is the thing you can use for that. So next slide. We go to the KASP, the key and signing policy. I don't know where -- That's the key and signing policy. What is KASP? KASP is basically nothing more than a configuration file that's been read once in a while. In that configuration file, you basically state, which zones do we want to sign? How often do we want to sign? What do we want to sign them with? When do we want to roll over the keys? How often do we want to roll over the keys? All that kind of stuff. If you don't know what to do, we have a default configuration that works for us that works for SIDN, that works for SE. We use slightly different configurations, but it has a very, very secure default. We're actually very, very conservative on creating this default. Next slide, please. Then we have the signer engine. The signer engine actually does the following task: It basically sorts the RR sets in a file. It creates the necessary NSEC or NSEC3 chains. It signs all the data you need to have signed and keeps the signatures up-to-date before they expire. Next slide, please. This is actually a slide from the architecture of open DNSSEC. I see on the screen it's very, very hard to read. Download the slides, if you can. In -- going through slight detail, the green box that you see there that's the signer component. The black box on the top right, that's the HSM stuff. It talks using -- it's -- sorry. Open DNSSEC communicates with the black box, HSM, using a PKCS11 library. The yellow parts are basically the configuration files, even though we have them distributed on the screen, in fact, you can use either one or two. The blue stuff is basically the inputs and outputs. It's the unsigned zone coming in and the signed zone coming out while you're having your coffee. Who's done this work? We did, Nominet, together with dot SE, together with (saying name), (Listing names) I'll go into a little bit more detail. We did the policy stuff, Nominet did. And (saying name) implemented the signer. Sweden did most work on the software HSM part -- I think I missed that slide; right? Oh, okay. Sweden did the soft HSM part. I will switch back to the slide in a moment because I think it's important to talk about it. (saying names) did consulting on the project. And SIDN has been massive supportive in testing open DNSSEC. I think in the future, they want to use it as well. I'm not speaking for SIDN at this moment. But they did so well that will we at Nominet felt comfortable to use it in our deployment. Please, Julie, if you can go back to the slide of the soft HSM. What is that? It's a virtualized HSM. We've built this especially so you don't have to buy a hardware HSM. So you can use a HSM in a virtualized environment. You can use it for testing. We think it's pretty much secure. And, in general, if you look at this from a 30- mile distance or from a 30-mile-high perspective, the soft HSM is actually secure enough in some people's mind. The reason for that is because if you can hack a system, regardless if it is software or hardware HSM, you can actually change the input to that HSM; right? So if you can change the input of the HSM, it's almost the equivalent of hacking the HSM itself. So soft HSM is actually, in a sense, just as secure as using hardware HSM. Thank you. This is my presentation. You can get soft HSM -- sorry, you can get open DNSSEC from the URL on the screen is www.openDNSSEC.org. And if you're interested in participating, we have mailing list called openDNSSEC user. We have a development list called openDNSSEC develop. That's a read-only list. The usual list and write and read list. If you're a top-level domain or a registrar or an open -- or a DNSSEC hosting provider, if you really want to use this, please talk to us, tell us your needs. If it's interesting enough, if it's good enough, we can change this on the spot. Where we are for version 1 is, currently, we're done with version 1. We have AXFR with -- I mean, we're using it at Nominet. It works really, really well. For (inaudible), it's something different, it's very large zone, highly dynamic zone. We will use the features in version 2 for a highly dynamic zone. The features in version 2 will basically be dynamic update, incremental zone transfer and that kind of thing. And, really, this is the end, after saying that for the third time, of my presentation. Thank you. >>STEVE CROCKER: Thank you, Roy, very much. One point of clarification. What does Botan refer to? >>ROY ARENDS: Cryptographic library, similar to open SSL. >>STEVE CROCKER: The -- one very technical question. Where does the randomness, the crypt -- the entropy come from? >>ROY ARENDS: We leave that to the device, the HSM or soft HSM, to provide that for you. When you generate keys, that's where you need the randomness. >>STEVE CROCKER: I'll push on that later. Thank you. Do we have questions from the floor here? Do we have questions from -- on the phone? Jaap Akkerhuis says there's -- >> I've tried most of the DNSSEC tools out there, and I found open DNSSEC by far to be the most sort of buy-it-and-forget solution. Installing it was a little bit of work, but after that, it runs itself. >>ROY ARENDS: Thank you, Warren. >>STEVE CROCKER: That's a great endorsement. Okay. So I think -- I think that completes it. I was just checking the chat room here. There was a comment about needing randomness, which is similar to what I was asking about. Let's see. Is Eric here to make this presentation on AfTLD? Introduce yourself, and make the -- and then talk about the DNSSEC survey through the African top-level domains. A small change of schedule here. We're going to skip to the next section and then come back. Survey of DNSSEC deployment in Europe. Wim Degezelle, from CENTR. >>WIM DEGEZELLE: Well, as we're on schedule, I will really run -- behind schedule, I will really run through the first couple of slides. Because I first gave a little introduction about CENTR, because not everybody in the room will know what CENTR is. You can already go to the next one. CENTR is the European ccTLD organization. It is an organization that brings together the ccTLD registries to exchange information. Can go to the next slide already. To discuss and to -- inform each other and discuss and exchange information and best practices. In our organization, we have -- you can go to the next slide -- we have a survey tool set up. We use it very, very often, because it is the way to get some (inaudible) from the mailing list, but in the meantime have a good archive of what is being discussed. One of the recent surveys was our DNSSEC survey. It was made by us because we wanted to have a kind of overview of what is happening in the -- in the CENTR membership. And it was set up with input from our members. Questions were asked between the 1st of February and the 2nd of March of this year. So it's quite recent. Next slide. 21 CENTR members have answered the survey. I have put them there. The total number can be interesting to note, total number of all the -- of domain names under all the ccTLDs that answered the survey is 27,3797,000 and something. So it is not -- not just a few ccTLDs. One remark I would like to make at the beginning: I know one or two or three CENTR members that are actually already really preparing or just launched DNSSEC didn't answer the survey. So it is not a complete overview of what is happening under the CENTR membership. It's only what's happened or the answers of those 21 members. But, on the other hand, I'm also (inaudible) CENTR mailing list, it's not because you're not interested in DNSSEC that you shouldn't answer the survey, because my basic aim was to have an overview of who is interested, who is doing what. You can go to the next slide. First -- and maybe you have to click a second time. Okay. The first question was, what's the current state of DNSSEC in your country. So we have three countries in the survey that say, okay, DNSSEC is up and running at this moment. More importantly than CENTR members that say, okay, DNSSEC will be up and running within the next 12 months. Then there is another three members that say, okay, we will need a little bit more time, but we have a clear plan already there. It will be ready, DNSSEC will be there within one and two years. And another three CENTR members say we will have DNSSEC operational within two and five years. Only two members answered the survey saying, okay, at this moment, we don't have any concrete plans to have DNSSEC. An open question we asked in the survey was, yeah, but what were the most important reasons for you as a registry to say, okay, now we start concretely with the plan to have DNSSEC. You can click I think a couple of times, four times or so. The most important is, of course, safer Internet. But that's an answer everybody will give. But we were more interested in, okay, but what made you take the decision? A large number of registries answered, yeah, we see it as a ccTLD registry as our task to help build out a safe and more stable DNS. A number of registries also said, yeah, but it was -- in the context of our local Internet community, that they would hear talking about the DNSSEC, and we got some input, some suggestions, okay, maybe as a registry, you should think about it, and we got some support from there. Also, a number of registries -- you can click -- said, okay, we got some support from the government. And there was one or two registries said, okay, we saw that ICANN is pushing and putting DNSSEC in the contract for new gTLDs. So that's for us as ccTLDs, well, it would be good if we start to prepare DNSSEC or start to be ready, because if all the new gTLDs have to do it, then it will be good that we, as ccTLD registries, are ready, too. Okay. Next slide, please. We had a question, is there any ccTLD registry that at this moment or has plans to say, okay, we have no DNSSEC domain name. Will that be -- will be charged for that or not? There's no extra cost. All registries said, we provide it as a free service, as an additional service to the domain holder. Second question on that side is, yeah, but what was the number of domain names signed at the end of January? That was, of course, really, really low. It's less than -- less than a half percent. Of course, here, I have to make the remark, only three registries in the survey already very many DNSSEC, and two of them only started at the - - well, at the beginning of January or in January. But, still, it is a really, really low figure. Next slide, please. Well, one of the questions that was also asked is, yeah, but how do you work together with your registrars? Was it easy to prepare them? Did you have any problems with them? If you can click. Well, the answer mostly was, well, at this moment, or when we started with our DNSSEC plans, there was no real demand from the registrars. Sometimes we had the feeling that they were not ready for DNSSEC or for the complete deployment. Next, please. Some registrars answered, okay, DNSSEC is an extra burden, both technically or financial or administrative. Some registrars have a lack of experience. And I think there is one left. One of our members also -- and that can be important -- say, okay, we have the feeling that registrars are just waiting until dot com decides, okay, we implement DNSSEC, and then it will be worth for them, it will be worth for them to do the investment, and maybe then they will be more interested. Next slide. Well, the question wasn't asked, well, do you involve -- how do you involve and form your registrars about DNSSEC deployment? Well, I think the answers are quite obvious. They -- most of our members organized seminars, information sessions, sent e-mails. It is more or less the normal way in which they communicate on technical issues with registrars. Next slide. Question we asked was, did you communicate, did you consult with your registrars before the implementation of DNSSEC? All participants to the survey said yes. But one or two made the remark, if you can click, that, okay, we consulted them, but we didn't really get substantial feedback. So we wanted -- Okay. Next slide. The next two or three questions are a little bit more, well, technical or just asking what kind of solution the registries took or are planning to take. If you can click -- the question was asked, what solution did you choose for key management? Eight answered, we went for the -- or we want to go for the open DNSSEC solution. Roy was talking about that. Four say, we have homemade solution. One member said, yeah, we choose a kind of a DNSSEC box, but don't have further information. And one member answered, we came up with another solution. Next, please. Same question was asked whether they went for NSEC, or NSEC3, or NSEC3 opt-out. Eight go for NSEC3 with opt-out, four for NSEC3, and one says, okay, we selected the NSEC option. There was a question asked, what do you -- how do you deal with the supply and update of the DS keys to the registry. First of all, what interface is used for the registrants to supply or update the keys? For answers, we have a Web-based solutions -- four answered, we have a Web-based solution, three others, we have another solution. Same question was asked about the interface for the registrar, to supply an update to the DS keys. Four choose Web-based solution. Three registries answered, okay, we choose EPP. One registry answered, okay, we have another solution. Next slide, please. The same question. For the interface for the key holder to supply updated keys, well, the answer is equally divided. Two go for EPP, two answered a Web-based solution, and two said another solution. Then our final question was a rather open question, yeah, what would you suggest to registries that deploy DNSSEC or want to deploy DNSSEC? An answer that came back with a lot of people was, work and plan together with your registrars. Try to involve them from the beginning. Make sure that there is a lot of information available, a lot of also easy, accessible information and also useful information available. Provide easy-to-read documentation. And manage the different aspects of DNSSEC in different processes and see them as different -- different parts. Okay. And that was more or less overview of my presentation. I did my best to go as quick as possible. I hope it was.... >>STEVE CROCKER: That was a great presentation. I liked it. Thank you. Any questions? >> Thanks. I thought it was a great presentation. Really useful figures. Maybe it's a question. Maybe it's a comment to the floor. One of the things we're wrestling with at the moment as we're looking at signing (inaudible), is how much of the effort that we put into doing it is the technical and how much of it is the marketing and awareness campaign with registrars and with other parties and (inaudible) commented at least along the line on that. >>WIM DEGEZELLE: Well, I think it's difficult to answer. My first question would be, maybe we should set up a new CENTR survey and dot UK can come back in the Brussels meeting. >>STEVE CROCKER: Let me comment that in planning for this meeting, we've also done some look-ahead planning for the Brussels meeting. And there will be a considerable amount of focus on sort of what I might refer to loosely as the next layer of the registrars and the ISPs. And I don't -- we haven't gotten all the pieces in place yet. I don't know how mature that's going to be. But we're clearly headed there. Good. Any other questions? >> I also would like to comment that what I have been seeing within CENTR, too, is that DNSSEC was, until, let's say, one year ago, something that was exclusively discussed within the technical workshop, and now you start to see it being discussed in -- more from an administrative and also marketing perspective. So I think the amount of time spent in working groups represents also the money and the effort that is spent within the registries. >>STEVE CROCKER: Good. I'm a little unsettled about missing the African presentation. Let me ask here, would there be any objection to just flipping through the slides? Any of the folks from -- yeah? Okay. I don't know what the authority is, but we'll just take that as a good -- All right. So as I say, I'm quite curious. I haven't seen these slides before. So this is unrehearsed. Title slide, AfTLD DNSSEC survey. Next slide. Aim of the survey, to access readiness and plans of the African ccTLDs to adopt DNSSEC. Next slide. Ask the ccTLDs their level of adoption, surveys done in cooperation with the Security and Stability Advisory Committee, online tools, and the results were collated. And let's see what the results are. The Oscars were Sunday night. So this is -- okay, outcome of the survey, number of respondents was 7 to questions that date internal experimentation began. All had no plans. Tanzania and Kenya plan to start experimentation soon. Date you announced, no definite plan by most respondents, Tanzania had a date. No date by respondents to partial or full operation. Next slide. Method -- survey not very targeted. Many of the registries in Africa are small and the priority is making the registry run properly. I would have to say that is a very good thing to do, to make the registries run properly. Way forward. Many ccTLDs not sensitized enough on DNSSEC. And moving toward the end, planned capacity-building with SSAC on deployment of DNSSEC. And speaking on behalf of SSAC, I would say not only with respect to SSAC, but there are other groups that I think will be happy to supply assistance and help build capacity. Next slide. Great. Okay. Well, again, with apologies, I think that's helpful to have that there. That segues into the next presentation, which is my own. I have been running my own survey of ccTLD and also gTLD, but the focus here is on ccTLD deployment process. I have been asking the same four questions: What is the date experimentation began, what is the date an announcement was made formally that there would be DNSSEC deployment in the future, what is the date that there was partial operation, and what is the date of full operation. These answers are plotted on a map where the colors are keyed to the state of development. And as of the date at the top of the chart. So as of end of last year, 31 December 2009, this map shows that there were nine countries that were fully operational: Brazil, Namibia, Sweden, and others around the globe. Seven that were in partial operation, seven that had announced, and five more that had been in some state of experimental operation. And then the next slides are going to march through this quarterly for 2010, and then skip a full year ahead to 2011. So this shows the expected state of development at the end of this month, moving up to 12 countries in operation. Julie, next one. End of June, one more. End of September, 14. End of this year, jumps up to 19. And by the time we get to 2011, it goes up to 20. Let me say I believe the actual numbers when we get into that range will be higher. This based on information provided by these ccTLD operators. And I know that there are plans that are not yet announced and not yet being conveyed. Clearly, one of the things that is driving, it's spurring efforts forward is the signing of the root in the middle of this year, the signing of com and net. Net at the end of this year and com at the beginning of next year. Along the way, although this is the formal survey activity, I have been acquiring a lot of other bits of information. I had a very nice exchange with Andre Phillip, who runs the Czech Republic TLD dot CZ. He says they have 87,000 signed zones under dot CZ -- Oh, you are here. You speak. Sorry. >>ANDRE: Yeah, that's true, we have about 87,000 signed domain names and it is rising every day. You can look at the Web site www.nic.cz. Basically, it's done by some registrar that decided that the DNSSEC is just a feature of the domain, so they ultimately sign every zone that is registered and is run on their DNS servers. And that's basically the majority of the domains because they are also the major DNS operators in the country. Thank you. >>STEVE CROCKER: This is a big step forward. And I think that we're going to see comparable kinds of large numbers of zones being signed, principally because they are all by registrars or other operators who are hosting large numbers of zones. And I'd like to find a way to track that. It's been on my mind for a couple of years that this will be a large step function. And this will be the way in which we will have a large step function in the number of signed zones. So thank you. So as I say, this mapping process, I will attempt to keep up and evolve over a period of time. I am expecting that by the time we get together in Brussels, the numbers will have changed, not just because of the passage of time as shown on these maps but because of other announcements that show significantly more activity. And most particularly, I'm hoping that this region, Africa and particularly east Africa, begins to show up and provide some color to the map. Any questions? >> On these slides, does that include -- is that just CCs or does that include the impact of upcoming gTLDs? >>STEVE CROCKER: This map is only the ccTLDs. I have the data for the gTLDs. I haven't been pursuing it as aggressively. I hadn't figured out a good way to display it. And anybody who wants to jump in on this, I am happy to provide the data. We are putting these maps and the data up on DNSSEC deployment, and happy to distribute it, or turn this into a broader effort. I have been doing this out of my back pocket. And I know these maps could be made more interesting and different ways. >> Thanks. >>STEVE CROCKER: Is there anything in the chat room on this? I don't see any. Next. Presentation on the status of dot org DNSSEC activity. Markus, you wanted to chime in? >>MARKUS TRAVAILLE: It's just a comment from the chat room. Dot dk claims to be ready in July, and I don't see them on the map. >>STEVE CROCKER: Now I see the same question. So Christian, I apologize. Send me e-mail, and I will send you my little survey and you will be on very next version of the map. Jim Galvin, I hope you are still there. >>JIM GALVIN: Yes. >>STEVE CROCKER: You are there. >>JIM GALVIN: Can you hear me okay? >>STEVE CROCKER: More than okay. We need to tone it down just a bit. Step back, maybe, a bit. Okay. Jim Galvin from Afilias talking about the rollout of dot org DNSSEC activity. Take it away. >>JIM GALVIN: Thank you, Steve. As you say, I am Jim Galvin. I am from Afilias. Afilias is the back-end registry services provider for dot org, and the Public Interest Registry, PIR, is the registry for dot org. While we're on the title slide, let me take a minute and just recap what we reported on in the past here. Dot org was signed on June 2nd, 2009, but we were not offering signed delegations at that time. Over the past eight months, we have been conducting friends and family testing, and we are now preparing for the final step in the launch of DNSSEC, which is allowing of signed delegations. At the time that dot org signed, they were the largest TLD to sign and the first gTLD to sign. Next slide. We have, in our testing phase, ten registrars that have passed the certification test. What that means is DNSSEC will be optional for registrars. They are not required to offer that, but if they do offer it, they will be required to complete a certification test. We have been actively running a registrar education program. We have a series of Webinars. We have a few that we have done and a few that are coming. And we have been conducting the operator transfer testing. Olafur has already done an excellent job of describing what's going on there. And the Shinkuro team has actually previously prepared a registrar crib sheet that was also one of the bases for one of the Webinars we have had in the past for registrars. And you can see a URL there for folks to grab that. Next slide, please. We are now wrapping up our internal testing. A critical part of that, of course, is the work that Olafur and the rest of the team at Shinkuro are currently conducting with the transfer testing. We certainly want to thank publicly everyone who has participated in that testing. And the brand-new news of the day is that we are about to send out the 90-day notice to registrars. This 90-day notice is required by ICANN, and you have to give it at least 90 days in advance of when you are going to do your product launch and launching of signed delegations. That notice should go out -- you will see that today or tomorrow within a company press release, et cetera. Regular, typical sort of marketing fanfare. But that is our news of the day. And soon, we will be opening up a certification environment for additional registrars who want to prepare for being able to offer signed delegations. And that's actually it. Next slide, please. We are open to any questions or comments from the floor. >>STEVE CROCKER: No questions? So just to repeat the news of the day, you are saying that you are announcing today the beginning of the 90-day period after which you will be open for general registration through DNSSEC compliant registrars? >>JIM GALVIN: That's correct. >>STEVE CROCKER: And this is March 10. That leaves you -- I can't count 90 days exactly, but roughly June 10th. That will be excellent. Warren. >>WARREN: Jim, are you going to be publishing a list of DNSSEC capable registrars so that people can easily figure out who to use to get a signed (inaudible) in dot org? >>JIM GALVIN: We expect that PIR will include on their Web site a list of those registrars that are known to be DNSSEC enabled. >>STEVE CROCKER: Ram. >>RAM MOHAN: Jim, this is Ram. I am wondering if you can list for the group here registrars who are already in the program so that folks can start look at it. This is not private data at this point. >>JIM GALVIN: The only two that we're publicly stating are DynDNS and Names Beyond. As Olafur had described in the transfer testing. The rest of those that have certified and have been participating have not made any public announcements themselves, and so we're not at liberty to announce for them. >>STEVE CROCKER: So the important element of this is that we know now that there will at least be two registrars that are ready, and so it won't be an empty table. >>JIM GALVIN: That's correct. >>STEVE CROCKER: Any other comments or questions? Thank you. We have, amazingly, come to the end. Let me thank you everybody for sitting through this extended session. We are very interested in feedback. As I said at the beginning, we have tried to make an adjustment where we put substantial content on technical matters into the program. It was clear from some of the dialogue that it was very spirited and engaged a number of people. Don't be bashful. Send feedback. We are very much looking forward to it. We have planning -- We are expecting that a number of DNSSEC- related activities will come together almost with a crescendo in Brussels. It will be on the eve, if you will, of the root being signed. You have just heard that dot org will be open for general registration even before Brussels. I am expecting that there will be many other announcements, or at least a notable number of other announcements in the process. We are going to try to have experience from ISPs who are running recursive resolvers, and, in general, expand the range of information that's available. As a consequence, we're going to try to run an even longer session than we usually do, and I believe we're going to try to provide lunch as part of that. Something to look forward to. So with that, let me thank my colleagues who we have labored considerably every week to put the pieces of this in place. Let me thank every one of the presenters. There is a tremendous amount of work represented here, and all of this I think is making an important difference in laying a firm architectural plank in improving the security of the Internet. Thank you.