Go-live of WCT 3 successful at National Libraries of the Netherlands and New Zealand

2020-12-07T00:00:00+00:00

The National Library of the Netherlands (KBNL) is the first organization to take the brand new version 3 of the Web Curator Tool (WCT) into production. As a faithful user of the WCT starting in 2007 the KBNL has since then harvested more than 18,000 websites successfully with the WCT as a workflow management solution for scheduling, harvesting and quality analysis. With WCT 3 into production, the KBNL has improved reliability of the WCT environment and is ready for future enhancements. Jeffrey van der Hoeven, head of the Digital Preservation dept at KBNL: “I’m very proud that our cooperation with NLNZ is working out so well. WCT version 3 is the result of that joint development effort.”

The National Library of New Zealand (NLNZ) has also successfully put version 3 of the WCT into production. As one of the institutions that first developed the WCT in 2006, NLNZ has since that time been using the WCT to manage its selective web harvesting. Having migrated to version 3, NLNZ now has up-to-date tools at the heart of its web archiving programme. Andrea Goethals, Manager of Digital Preservation at NLNZ: “It was very fortunate for us that the KBNL had very similar needs as us for a web archiving tool that can be put in the hands of our curators. By collaborating on developing WCT with us, we all end up with a much better solution.”

What’s new in version 3?

With version 3 you get a much easier installation process, well documented and tested, and an up-to-date technical framework behind the scenes. With this solid foundation in place, you will also be in a much better position to receive future updates with new functionality.

A significant change in version 3 that System Administrators should be aware of is that WCT no longer requires an application container, such as Tomcat to run it. Instead, WCT has now been migrated to run with Spring Boot, which uses an embedded Tomcat server.

Why choose WCT?

Jeffrey: “Reliability and full control over our web harvesting process is very important to us. Meanwhile, we believe that collection specialists are often not techies and need easy-to-use tools at hand. Therefore, the WCT is an excellent piece of equipment for us.”

In 2018, version 2 of the WCT was released publicly containing long-awaited improvements on crawling techniques by integrating the latest Heritrix crawl engine H3 into WCT. But under the hood, a lot of the technical libraries and scripts of WCT were outdated due to lack of maintenance for several years before. Also, installation and migration could be a quite challenging process.

Meanwhile the WCT still delivered strong crawling results and has shown to be a powerful tool in performing orchestrated selective web harvests. Both National Library of New Zealand and KBNL recognized that investing in the WCT is worthwhile. In 2017 this resulted in the start of a strong cooperation between the two organizations to revamp the WCT and bring the software a major uplift, first technically and later on functionally.

Trienka Rohrbach, service administrator at KBNL states the usefulness of WCT for her library: “With the WCT I’m capable of supporting our collection specialists at the library in such a way that they can schedule crawls and perform in-depth analysis on their own without the need to dive into a lot of technical details.”

Since the successful relaunch of WCT with version 2, it took about two years of development to upgrade the whole software stack of WCT into a modern foundation while not compromising reliability and backwards compatibility with previous WCT-versions currently in use across the web archiving community.

What is to be expected next?

The WCT development team has already started development on huge functional improvements such as a neat integration with the Webrecorder, graph-like visualization of web crawls and better screenshot capturing. We expect next releases to be more lightweight and more frequent so please stay in touch with the WCT community. Hanna Koppelaar, software developer at KBNL: “We’re still continuously improving the WCT code base and reducing technical debt, but the focus for version 4 will be adding new features and improvements requested by our users.”

Click here to start downloading the new WCT release 3.

Want to try it first locally? Download our demo-version ready for use in Virtual Box.

Welcome to our new website!

2020-11-26T00:00:00+00:00

Welcome to the new website for the Web Curator Tool (WCT).

On the new site you will find important links for new and existing WCT users, such as guides for getting up and running, documentation, source code, the features of WCT and our development roadmap.

Our core communcation channels remain the same - Github and Slack, with the addition of a general email address () for contact with the teams at the National Library of the Netherlands (KB-NL) and the National Library of New Zealand (NLNZ).

Our hope is to also share more frequent updates covering the development, the use of, and promotion of WCT.

Please be aware that all Github activity for the project has been shifted to the https://github.com/WebCuratorTool organisation. This is to better align with the collaborative development between KB-NL and NLNZ. The previous Github repository https://github.com/DIA-NZ/webcurator, has been transferred to https://github.com/WebCuratorTool/webcurator-v2-legacy, and will be archived.

If you have any comments or questions about WCT please get in touch with us, we would love to hear from you.

Slack

Github

@WebCuratorTool