By Nick Butler
Tags: Agile , Development
This cloud migration case study shows how DigitalNZ increased the flexibility, security, reliability and cost transparency of their web applications by moving them to the cloud. By cutting the time spent managing and maintaining their platform, they’ve freed up time to work on improvements and extensions.
The move has boosted their ability to achieve their mission: making it easy to find, share and use New Zealand’s digital content. By opening up our digital treasure trove, they’re helping bring Kiwi culture to life, now and for years to come.
In the case study, DigitalNZ Systems Manager James Robertson looks at:
These lessons include:
DigitalNZ is a one-stop search site for New Zealand digital content. Run by the National Library of New Zealand, the site gives easy access to over 30 million items — photos, videos, news stories and more — from over 200 content partners. Enabling this digital collection aggregation is a complex set of applications, databases and supporting infrastructure.
Learn more about the development of the DigitalNZ web applications.
James Robertson says there were two main drivers for DigitalNZ’s cloud migration.
“Our physical servers were aging beyond their economic lifespan and even starting to fail. We knew we had to make a change. At the same time, the government introduced a cloud-first direction, so all agencies were required to at least investigate cloud as an option,” he says.
DigitalNZ chose Amazon Web Services as their cloud provider.
“AWS had the widest range of offerings, and the offerings that were aligned with our existing architecture,” says James.
“It’s also quite well-supported in New Zealand. We have direct access to AWS solution architects. They’ve been very helpful, both with helping to resolve specific problems or questions, and also for general training to the devs and the wider audience who need to understand the cloud.”
Boost’s developers have certainly found the support invaluable. And the feedback from the AWS support team has been heartening, with the consultants noting that the team’s level of expertise allows for in-depth technical discussions and innovative approaches.
“We were able to take an Agile approach, which meant starting with things that were easy, so that people got their feet under the table, learned the basics. Then we moved onto things with higher uncertainty.
“That meant not appearing to deliver so much in the early phases, but it makes sure that when we do come to deliver, everything falls into place a lot more smoothly.”
Not having a project with set milestones helped.
“We’ve been able to migrate at the pace that we were comfortable with, learn our lessons as we go.
“That does mean that we’ve had a bit of a split focus. We’ve had to maintain our existing systems at the same time as migrating them to the cloud.
“We’ve had to do a bit of patching on the old servers and that sort of thing, but apart from that, we haven’t had to invest a lot in the existing infrastructure,” he says. “Once we’re actually moving an application, we try and get it in production in the cloud in a fairly short timeframe.”
James sees their cloud migration falling into four stages.
“These stages won’t be relevant for everyone. They work for us because of the Agile nature of our work.”
“For us, stage one is more of an exploration — learning about the cloud, trying the obvious stuff, getting comfortable with the different offerings and different solutions.”
“The second stage is focusing on the bigger unknowns and building out some infrastructure. You’re making sure that you’ve got good security in place, your network’s set up right, your VPC’s [Virtual Private Cloud] right. You’ve got all the major components that you need and you’re confident that they can communicate with one another and with the outside world, or external networks.
“During both of those phases, you’re probably also migrating some simple workloads, some simple web applications, for example, or database content. That way you’ve got some wins under your belt too. You can say, ‘Hey, look, we cloudified this thing, and it hasn’t fallen over.’”
“We’re probably in the third phase now, which is just working through cloudifying each of our individual applications and sets of data.”
“The fourth stage will be decommissioning our old hardware and making sure we’ve got archives of things we need archives of.
“It will also involve re-architecting some things. As we’ve gone, we’ve changed the way we do some things to better fit the cloud, but we haven’t done a lot of in-depth re-architecting. We’re not making the most of cloud-native services for example. The next thing to do will be to look more closely at some of our big pieces of software infrastructure and how they could be more effective and more efficient in the cloud.”
An Agile approach also helps you deal with the range and rate of change of cloud offerings.
“It’s quite overwhelming to start with, particularly from a provider like AWS. They have hundreds of different service offerings. Working out which are relevant for you is a bit of a challenge, but there’s usually a core of services that you can make a start with.”
On top of that, the cloud is always evolving.
“The rate of change — that’s definitely something you need to accept.”
It can be frustrating when a new feature arrives just after you decide to go down a different route. But it’s also a good thing, because it means there are always going to be new opportunities.
Reducing dependence on aging servers has increased reliability.
“We have fewer outages. That’s also partly because, now that we’re with one (cloud) vendor, we’re not reliant on so many other moving parts between our servers and serving the public.
“And if a part of our infrastructure were to die, for whatever reason, or we needed to recreate it somewhere else, it’d be way simpler than if some physical server was dying somewhere.”
The cloud migration has made the DigitalNZ infrastructure more flexible. It’s both more adaptable and more resilient. It can easily be scaled up or down as demand or new services require.
“It makes it much easier for us to experiment with new ways of providing services or managing our existing infrastructure, so we can be more adaptable,” says James.
There was some initial concern that additional security measures would be needed for the move to the cloud.
“We learned that the security options that are available to you out of the box, cheaply or easily, are pretty comprehensive. You should be able to satisfy any set of security policies that your organisation may need.
“AWS provides some great tools. Trusted Advisor basically scans, if you like, your infrastructure and identifies potential weaknesses,” he says. “Guard Duty is another tool, an active threat identification service from AWS.”
“The other side of security, I guess, is logging and deleting. Again, AWS has lots of tools for that, for making sure that you can audit. If an incident does occur, you can find where and when that happened and who the likely actors were.”
As a result of the cloud migration, the cost of the DigitalNZ infrastructure is more transparent.
“Cost is much more at the front of my mind than it used to be, when we had our physical servers that we’d already paid for upfront,” says James.
“I’ve done a bit of research and it pays not to over-invest in trying to estimate what your costs are going to be ahead of time. Because it’s pretty hard to predict, unless you’ve had some solid cloud experience.
“We did a rough estimate for cost, based on a like-for-like comparison between the server setup and management, maintenance costs that we had, versus something similar in the cloud. There’s plenty of calculators and big lists of pricing for each region,” James says. “We’ve reacted as we’ve gone. We needed to get some production workloads into the cloud before we focused on cost reduction.”
Other cost management lessons:
“Quite a lot of the stories that we do, the development team do, are investigations or implementations of cost-reduction measures,” says James.
“One of the biggest benefits of the cloud is the number of managed services that they offer. Managed services take away all of the maintenance burden, security patching, upgrades all of that sort of stuff.
“We’re able to spend less money on managing and maintaining our service and more on new developments,” says James.
James found it took a while to get the required security and risk assessments and architectural sign-offs.
“I was fortunate in being supported by my manager to go ahead and start getting some work done, albeit prototype, non-production-mode-type work, prior to those sign-offs happening.”
Sign-offs may also include data sovereignty. Some New Zealand organisations, for example, will need to consider Māori data sovereignty.
And, as more organisations track their carbon footprint, it can pay to be clear up front on the impact your cloud migration will have. AWS, for example, has quite a high carbon footprint, especially in the Sydney centre that serves New Zealand.
Exciting and challenging, taking on a cloud migration is also a great opportunity.
For DigitalNZ, it has increased reliability, security and transparency of costs. The new set up is also much more flexible. It can easily be scaled up or down as demand or new services require. And, importantly, it’s freed up time to focus on improving services.
“As Systems Manager, I’m better able to support the rest of DigitalNZ team in developing new features and serving new audiences in new ways,” says James.
Top cloud providers — Canalys Cloud Channels Analysis
