Go to the content
RSS Feed

Bits Debian

No one following this article yet.

Bits.Debian is an official Debian blog. This blog is under the same license and copyright as the Debian website, see Debian WWW pages license.

If you want to contact us, please send an email to the debian-publicity mailing list. This is a publically archived list.

salsa.debian.org: Postmortem of failed Docker registry move

0 no comments yet No one following this article yet. 0 views

The Salsa admin team provides the following report about the failed migration of the Docker container registry. The Docker container registry stores Docker images, which are for example used in the Salsa CI toolset. This migration would have moved all data off to Google Cloud Storage (GCS) and would have lowered the used file system space on Debian systems significantly.

The Docker container registry is part of the Docker distribution toolset. This system supports multiple backends for file storage: local, Amazon Simple Storage Service (Amazon S3) and Google Cloud Storage (GCS). As Salsa already uses GCS for data storage, the Salsa admin team decided to move all the Docker registry data off to GCS too.

Migration and rollback

On 2019-08-06 the migration process was started. The migration itself went fine, although it took a bit longer than anticipated. However, as not all parts of the migration had been properly tested, a test of the garbage collection triggered a bug in the software.

On 2019-08-10 the Salsa admins started to see problems with garbage collection. The job running it timed out after one hour. Within this timeframe it not even managed to collect information about all used layers to see what it can cleanup. A source code analysis showed that this design flaw can't be fixed.

On 2019-08-13 the change was rolled back to storing data on the file system.

Docker registry data storage

The Docker registry stores all of the data sans indexing or reverse references in a file system-like structure comprised of 4 separate types of information: Manifests of images and contents, tags for the manifests, deduplicaed layers (or blobs) which store the actual data, and lastly links which show which deduplicated blogs belong to their respective images, all of this does not allow for easy searching within the data.

The file system structure is built as append-only which allows for adding blobs and manifests, addition, modification, or deletion of tags. However cleanup of items other than tags is not achievable within the maintenance tools.

There is a garbage collection process which can be used to clean up unreferenced blobs, however according to the documentation the process can only be used while the registry is set to read-only and unfortunately it cannot be used to clean up unused links.

Docker registry garbage collection on external storage

For the garbage collection the registry tool needs to read a lot of information as there is no indexing of the data. The tool connects to the storage medium and proceeds to download … everything, every single manifest and information about the referenced blobs, which now takes up over 1 second to process a single manifest. This process will take up a significant amount of time, which in the current configuration of external storage would make the clean up nearly impossible.

Leasons learned

The Docker registry is a data storage tool that can only properly be used in append-only mode. If you never cleanup, it works well.

As soon as you want to actually remove data, it goes bad. For Salsa clean up of old data is actually a necessity, as the registry currently grows about 20GB per day.

Next steps

Sadly there is not much that can be done using the existing Docker container registry. Maybe GitLab or someone else would like to contribute a new implementation of a Docker registry, either integrated into GitLab itself or stand-alone?

Debian celebrates 26 years, Happy DebianDay!

0 no comments yet No one following this article yet. 8 views

26 years ago today in a single post to the comp.os.linux.development newsgroup, Ian Murdock announced the completion of a brand new Linux release named #Debian.

Since that day we’ve been into outer space, typed over 1,288,688,830 lines of code, spawned over 300 derivatives, were enhanced with 6,155 known contributors, and filed over 975,619 bug reports.

We are home to a community of thousands of users around the globe, we gather to host our annual Debian Developers Conference #DebConf which spans the world in a different country each year, and of course today's many celebrations held around the world.

It's not too late to throw an impromptu #DebianDay celebration or to go and join one of the many celebrations already underway.

As we celebrate our own anniversary, we also want to celebrate our many contributors, developers, teams, groups, maintainers, and users. It is all of your effort, support, and drive that continue to make Debian truly: The universal operating system.

Happy #DebianDay!

New Debian Developers and Maintainers (May and June 2019)

0 no comments yet No one following this article yet. 10 views

The following contributors got their Debian Developer accounts in the last two months:

  • Jean-Philippe Mengual (jpmengual)
  • Taowa Munene-Tardif (taowa)
  • Georg Faerber (georg)
  • Kyle Robbertze (paddatrapper)
  • Andy Li (andyli)
  • Michal Arbet (kevko)
  • Sruthi Chandran (srud)
  • Alban Vidal (zordhak)
  • Denis Briand (denis)
  • Jakob Haufe (sur5r)

The following contributors were added as Debian Maintainers in the last two months:

  • Bobby de Vos
  • Jongmin Kim
  • Bastian Germann
  • Francesco Poli


DebConf19 closes in Curitiba and DebConf20 dates announced

0 no comments yet No one following this article yet. 7 views

DebConf19 group photo - click to enlarge

Today, Saturday 27 July 2019, the annual Debian Developers and Contributors Conference came to a close. Hosting more than 380 attendees from 50 different countries over a combined 145 event talks, discussion sessions, Birds of a Feather (BoF) gatherings, workshops, and activities, DebConf19 was a large success.

The conference was preceded by the annual DebCamp held 14 July to 19 July which focused on individual work and team sprints for in-person collaboration toward developing Debian and host to a 3-day packaging workshop where new contributors were able to start on Debian packaging.

The Open Day held on July 20, with over 250 attendees, enjoyed presentations and workshops of interest to the wider audience, a Job Fair with booths from several of the DebConf19 sponsors and a Debian install fest.

The actual Debain Developers Conference started on Sunday 21 July 2019. Together with plenaries such as the the traditional 'Bits from the DPL', lightning talks, live demos and the announcement of next year's DebConf (DebConf20 in Haifa, Israel), there were several sessions related to the recent release of Debian 10 buster and some of its new features, as well as news updates on several projects and internal Debian teams, discussion sessions (BoFs) from the language, ports, infrastructure, and community teams, along with many other events of interest regarding Debian and free software.

The schedule was updated each day with planned and ad-hoc activities introduced by attendees over the course of the entire conference.

For those who were not able to attend, most of the talks and sessions were recorded for live streams with videos made, available through the Debian meetings archive website. Almost all of the sessions facilitated remote participation via IRC messaging apps or online collaborative text documents.

The DebConf19 website will remain active for archival purposes and will continue to offer links to the presentations and videos of talks and events.

Next year, DebConf20 will be held in Haifa, Israel, from 23 August to 29 August 2020. As tradition follows before the next DebConf the local organizers in Israel will start the conference activites with DebCamp (16 August to 22 August), with particular focus on individual and team work toward improving the distribution.

DebConf is committed to a safe and welcome environment for all participants. During the conference, several teams (Front Desk, Welcome team and Anti-Harassment team) are available to help so both on-site and remote participants get their best experience in the conference, and find solutions to any issue that may arise. See the web page about the Code of Conduct in DebConf19 website for more details on this.

Debian thanks the commitment of numerous sponsors to support DebConf19, particularly our Platinum Sponsors: Infomaniak, Google and Lenovo.

About Debian

The Debian Project was founded in 1993 by Ian Murdock to be a truly free community project. Since then the project has grown to be one of the largest and most influential open source projects. Thousands of volunteers from all over the world work together to create and maintain Debian software. Available in 70 languages, and supporting a huge range of computer types, Debian calls itself the universal operating system.

About DebConf

DebConf is the Debian Project's developer conference. In addition to a full schedule of technical, social and policy talks, DebConf provides an opportunity for developers, contributors and other interested people to meet in person and work together more closely. It has taken place annually since 2000 in locations as varied as Scotland, Argentina, and Bosnia and Herzegovina. More information about DebConf is available from https://debconf.org/.

About Infomaniak

Infomaniak is Switzerland's largest web-hosting company, also offering backup and storage services, solutions for event organizers, live-streaming and video on demand services. It wholly owns its datacenters and all elements critical to the functioning of the services and products provided by the company (both software and hardware).

About Google

Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products such as online advertising technologies, search, cloud computing, software, and hardware.

Google has been supporting Debian by sponsoring DebConf for more than ten years, and is also a Debian partner sponsoring parts of Salsa's continuous integration infrastructure within Google Cloud Platform.

About Lenovo

As a global technology leader manufacturing a wide portfolio of connected products, including smartphones, tablets, PCs and workstations as well as AR/VR devices, smart home/office and data center solutions, Lenovo understands how critical open systems and platforms are to a connected world.

Contact Information

For further information, please visit the DebConf19 web page at https://debconf19.debconf.org/ or send mail to press@debian.org.

DebConf19 starts today in Curitiba

0 no comments yet No one following this article yet. 7 views

DebConf19 logo

DebConf19, the 20th annual Debian Conference, is taking place in Curitiba, Brazil from from July 21 to 28, 2019.

Debian contributors from all over the world have come together at Federal University of Technology - Paraná (UTFPR) in Curitiba, Brazil, to participate and work in a conference exclusively run by volunteers.

Today the main conference starts with over 350 attendants expected and 121 activities scheduled, including 45- and 20-minute talks and team meetings ("BoF"), workshops, a job fair as well as a variety of other events.

The full schedule at https://debconf19.debconf.org/schedule/ is updated every day, including activities planned ad-hoc by attendees during the whole conference.

If you want to engage remotely, you can follow the video streaming available from the DebConf19 website of the events happening in the three talk rooms: Auditório (the main auditorium), Miniauditório and Sala de Videoconferencia. Or you can join the conversation about what is happening in the talk rooms: #debconf-auditorio, #debconf-miniauditorio and #debconf-videoconferencia (all those channels in the OFTC IRC network).

You can also follow the live coverage of news about DebConf19 on https://micronews.debian.org or the @debian profile in your favorite social network.

DebConf is committed to a safe and welcome environment for all participants. During the conference, several teams (Front Desk, Welcome team and Anti-Harassment team) are available to help so both on-site and remote participants get their best experience in the conference, and find solutions to any issue that may arise. See the web page about the Code of Conduct in DebConf19 website for more details on this.

Debian thanks the commitment of numerous sponsors to support DebConf19, particularly our Platinum Sponsors: Infomaniak, Google and Lenovo.

DebConf19 invites you to Debian Open Day at the Federal University of Technology - Paraná (UTFPR), in Curitiba

0 no comments yet No one following this article yet. 8 views

DebConf19 logo

DebConf, the annual conference for Debian contributors and users interested in improving the Debian operating system, will be held in Federal University of Technology - Paraná (UTFPR) in Curitiba, Brazil, from July 21 to 28, 2019. The conference is preceded by DebCamp from July 14 to 19, and the DebConf19 Open Day on July 20.

The Open Day, Saturday, 20 July, is targeted at the general public. Events of interest to a wider audience will be offered, ranging from topics specific to Debian to the greater Free Software community and maker movement.

The event is a perfect opportunity for interested users to meet the Debian community, for Debian to broaden its community, and for the DebConf sponsors to increase their visibility.

Less purely technical than the main conference schedule, the events on Open Day will cover a large range of topics from social and cultural issues to workshops and introductions to Debian.

The detailed schedule of the Open Day's events includes events in English and Portuguese. Some of the talks are:

  • "The metaverse, gaming and the metabolism of cities" by Bernelle Verster
  • "O Projeto Debian quer você!" by Paulo Henrique de Lima Santana
  • "Protecting Your Web Privacy with Free Software" by Pedro Barcha
  • "Bastidores Debian - Entenda como a distribuição funciona" by Joao Eriberto Mota Filho
  • "Caninos Loucos: a plataforma nacional de Single Board Computers para IoT" by geonnave
  • "Debian na vida de uma Operadora de Telecom" by Marcelo Gondim
  • "Who's afraid of Spectre and Meltdown?" by Alexandre Oliva
  • "New to DebConf BoF" by Rhonda D'Vine

During the Open Day, there will also be a Job Fair with booths from our several of our sponsors, a workshop about the Git version control system and a Debian installfest, for attendees who would like to get help installing Debian on their machines.

Everyone is welcome to attend. As the rest of the conference, attendance is free of charge, but registration in the DebConf19 website is highly recommended.

The full schedule for the Open Day's events and the rest of the conference is at https://debconf19.debconf.org/schedule and the video streaming will be available at the DebConf19 website

DebConf is committed to a safe and welcome environment for all participants. See the DebConf Code of Conduct and the Debian Code of Conduct for more details on this.

Debian thanks the numerous sponsors for their commitment to DebConf19, particularly its Platinum Sponsors: Infomaniak, Google and Lenovo.

Debian 10 "buster" has been released!

0 no comments yet No one following this article yet. 13 views

Alt Buster has been released

You've always dreamt of a faithful pet? He is here, and his name is Buster! We're happy to announce the release of Debian 10, codenamed buster.

Want to install it? Choose your favourite installation media and read the installation manual. You can also use an official cloud image directly on your cloud provider, or try Debian prior to installing it using our "live" images.

Already a happy Debian user and you only want to upgrade? You can easily upgrade from your current Debian 9 "stretch" installation; please read the release notes.

Do you want to celebrate the release? We provide some buster artwork that you can share or use as base for your own creations. Follow the conversation about buster in social media via the #ReleasingDebianBuster and #Debian10Buster hashtags or join an in-person or online Release Party!

Upcoming Debian 10 "buster"!

0 no comments yet No one following this article yet. 11 views

Alt Buster is coming on 2019-07-06

The Debian Release Team in coordination with several other teams are preparing the last bits needed for releasing Debian 10 "buster" on Saturday 6 July 2019. Please, be patient! Lots of steps are involved and some of them take some time, such as building the images, propagating the release through the mirror network, and rebuilding the Debian website so that "stable" points to Debian 10.

If you are considering create some artwork on the occasion of buster release, feel free to send us links to your creations to the (publicly archived) debian-publicity mailing list, so that we can disseminate them throughout our community.

Follow the live coverage of the release on https://micronews.debian.org or the @debian profile in your favorite social network! We'll spread the word about what's new in this version of Debian 10, how the release process is progressing during the weekend and facts about Debian and the wide community of volunteer contributors that make it possible.

If you want to celebrate the release of Debian 10 buster, join one of the many release parties or consider organizing one in your city! Celebration will also happen online on the Debian Party Line.

DebConf19 Cheese and Wine Party

0 no comments yet No one following this article yet. 6 views

In less than one month we will be in Curitiba to start DebCamp and DebConf19 \o/

This C&W is the 15th official DebConf Cheese and Wine party. The first C&W was improvised in Helsinki during DebConf 5, in the so-called "French" room. Cheese and Wine parties are now a tradition for DebConf.

The event is very simple: bring good edible stuff from your country. We like cheese and wine, but we love the surprising stuff that people bring from all around the world or regions of Brazil. So, you can bring non-alcoholic drinks or a typical food that you would like to share as well. Even if you don't bring anything, feel free to participate: our priorities are our attendants and free cheese.

We have to organize for a great party. An important part is planning - We want to know what you are bringing, in order to prepare the labels and organizing other things.

So, please go to our wiki page and add what you will bring!

If you don't have time to buy before travel, we list some places where you can buy cheese and wine in Curitiba. There are more information about C&W, what you can bring, vegan cheese, Brazil customs regulations and non-alcoholic drinks at our site.

C&W will happen on July 22nd, 2019 (Monday) after 19h30min.

We are looking forward to seeing you all here!

DebConf19 logo

Diversity and inclusion in Debian: small actions and large impacts

0 no comments yet No one following this article yet. 9 views

The Debian Project always has and always will welcome contributions from people who are willing to work on a constructive level with each other, without discrimination.

The Diversity Statement and the Code of Conduct are genuinely important parts of our community, and over recent years some other things have been done to make it clear that they aren't just words.

One of those things is the creation of the Debian Diversity Team: it was announced in April 2019, although it had already been working for several months before as a welcoming space for, and a way of increasing visibility of, underrepresented groups within the Debian project.

During DebConf19 in Curitiba there will be a dedicated Diversity and Welcoming Team. It will consist of people from the Debian community to offer a contact point when you feel lost or uneasy. The DebConf team is also in contact with a local LGBTIQA+ support group for exchange of safety concerns and information with respect to Brazil in general.

Today Debian also recognizes the impact LGBTIQA+ people have had in the world and within the Debian project, joining the worldwide Pride celebrations. We show it by changing our logo for this time to the Debian Diversity logo, and encourage all Debian members and contributors to show their support of a diverse and inclusive community.

Debian Diversity logo