Open-source News

How I troubleshoot swappiness and startup time on Linux David Both Tue, 09/13/2022 - 03:00

I recently experienced another interesting problem in the Linux startup sequence that has a circumvention–not a solution. It started quite unexpectedly.

I was writing a couple of articles while making some updates to my personal copy of my series of books, "Using and Administering Linux: Zero to SysAdmin." I had four instances of LibreOffice Write open to doing all that. I had three VMs running with VirtualBox to test some of the things I was writing about. I also had LibreOffice Impress open to work on an unrelated presentation. I like to listen to music, so I had one of several tabs in Firefox open to Pandora, my music streaming service of choice. I had multiple Bash shells open using Konsole with numerous tabs and the Alpine text-mode email client in one. Then there were the various tabs in the Thunar file manager.

So I had a lot going on. Just like I do now as I write this article.

The symptoms

As I used these open sessions, I noticed that things slowed down considerably while waiting for the system to write a document to the M.3 SSD–a process that should have been really fast. I also noticed that the music was choppy and dropped out completely every few minutes. Overall performance was generally poor. I began to think that Fedora had a serious problem.

My primary workstation, the one I was working on at the time, has 64GB of RAM and an Intel Core i9 Extreme with 16 cores and Hyperthreading (32 CPUs) that can run as fast as 4.1 GHz using my configured overclocking. So I should not have experienced any slowdowns–or so I thought at the time.

Determine the problem

It did not take long to find the problem because I have experienced similar symptoms before on systems with far less memory. The issue looked like delays due to page swapping. But why?

I started with one of my go-to tools for problem determination, htop. It showed that the system was using 13.6GB of memory for programs, and most of the rest of the RAM was in cache and buffers. It also showed that swapping was actively occurring and that about 253MB of data was stored in the swap partitions.

Date & Time: 2022-08-12 10:53:08
Uptime: 2 days, 23:47:15
Tasks: 200, 1559 thr, 371 kthr; 4 running
Load average: 3.97 3.05 2.08
   
Disk IO: 202.6% read: 687M write: 188K
Network: rx: 0KiB/s tx: 0KiB/s (0/0 packets)
Systemd: running (0/662 failed) (0/7912 jobs)    
Mem[|||||||##*@@@@@@@@@@@@@@@@@@@@@@@@@@    13.6G/62.5G]
Swp[||#                                      253M/18.0G]

But that meant I still had lots of memory left the system could use directly for programs and data and more that it could recover from cache and buffers. So why was this system even swapping at all?

I remembered hearing about the "swappiness" factor in one of my Red Hat training classes. But that was a long time ago. I did some searches on "swappiness" to learn about the kernel setting vm.swappiness.

The default value for this kernel parameter is 60. That represents the percent of free memory not yet in use. When the system reaches that 60% trigger point, it begins to swap, no matter how much free memory is available. My system started swapping when about 0.6 * 62.5GB = 37.5GB of unused memory remained.

Based on my online reading, I discovered that 10% is a better setting for many Linux systems. With that setting, swapping starts when only 10% of RAM is free.

I checked the current swappiness setting on my system, and it was set to the default.

# sysctl vm.swappiness
vm.swappiness = 60

Time to change this kernel setting.

More for sysadmins Enable Sysadmin blog The Automated Enterprise: A guide to managing IT with automation eBook: Ansible automation for Sysadmins Tales from the field: A system administrator's guide to IT automation eBook: A guide to Kubernetes for SREs and sysadmins Latest sysadmin articles Fix the issue

I won't dive into the gory details, but the bottom line is that either of the following commands, run as root, will instantly do the job on a running Linux computer without a reboot.

# sysctl -w vm.swappiness=10

You could also use this next command to do the same thing.

# echo 10 > /proc/vm/swappiness

Tecmint has an excellent article about setting kernel parameters.

Both commands change the live kernel setting in the /proc filesystem. After running either of those commands, you should run the sysctl vm.swappiness command to verify that the kernel setting has changed.

But those commands only change the swappiness value for the currently running system. A reboot returns the value to its default. I needed to ensure that this change is made persistent across reboots.

But first, the failure

To permanently change the kernel vm.swappiness variable, I used the procedure described in my previous article, How I disabled IPv6 on Linux, to add the following line to the end of the /etc/default/grub file:

GRUB_CMDLINE_LINUX="vm.swappiness=1"

I then ran the grub2-mkconfig command as root to rebuild the /boot/grub2/grub.cfg file. However, testing with VMs and real hardware showed that it did not work, and the swappiness value did not change. So I tried another approach.

And the success

Between this failure at startup time, the one I describe in the How I disabled IPv6 on Linux article, and other startup issues I explored due to encountering those two, I decided that this was a Linux startup timing problem. In other words, some required services, one of which might be the network itself, were not up and running, which prevented these kernel option changes from being committed to the /proc filesystem, or they were committed and then overwritten when the service started.

I could make all of these work as they should by adding them to a new file, /etc/sysctl.d/local-sysctl.conf with the following content, which includes all of my local kernel option changes:

###############################################
#            local-sysctl.conf                #
#                                             #
# Local kernel option settings.               #
# Install this file in the /etc/sysctl.d      #
# directory.                                  #
#                                             #
# Use the command:                            #
# sysctl -p /etc/sysctl.d/local-sysctl.conf   #
# to activate.                                #
#                                             #
###############################################
###############################################
# Local Network settings                      #
# Specifically to disable IPV6                #
###############################################
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

###############################################
# Virtual Memory                              #
###############################################
# Set swappiness
vm.swappiness = 1

I then ran the following command, which activated only the kernel options in the specified file:

# sysctl -p /etc/sysctl.d/local-sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
vm.swappiness = 13

This is a more targeted approach to setting kernel options than I used in my article about disabling IPv6.

Reporting the bug

At the time of this writing, there is no true fix for the root cause of this problem–whatever the cause. There is a way to temporarily circumvent the issue until a fix is provided. I used the /etc/sysctl.d/local-sysctl.conf file that I had created for testing and added a systemd service to run at the end of the startup sequence, wait for a few seconds, and run sysctl on that new file. The details of how to do that are in the How I disabled IPv6 on Linux article.

I had already reported this as bug 2103517 using Red Hat's Bugzilla when trying to disable IPv6. I added this new information to that bug to ensure that my latest findings were available to the kernel developers.

You can follow the link to view the bug report. You do not need an account to view bug reports.

Final thoughts

After experimenting to see how well I could reproduce the symptoms, along with many others, I have determined that the vm.swappiness setting of 60% is far too aggressive for many large-memory Linux systems. Without a lot more data points than those of my own computers, all I can tentatively conclude is that systems with huge amounts of RAM that get used only infrequently are the primary victims of this problem.

The immediate solution to the problem of local kernel option settings not working is to set them after startup. The automation I implemented is a good example of how to use systemd to replace the old SystemV startup file rc.local.

This bug had not been previously reported. It took a few days of experimenting to verify that the general problem in which locally-set kernel options were not being set or retained at startup time was easily repeatable on multiple physical and virtual systems. At that point, I felt it important to report the bug to ensure it gets fixed. Reporting it is another way I can give back to the Linux community.

I recently experienced another interesting problem in the Linux startup sequence that has a circumvention–not a solution. It started quite unexpectedly.

Image by:

opensource.com

Linux Sysadmin What to read next How I recovered my Linux system using a Live USB device This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Open source blockchain development: Get started with Hyperledger FireFly Nicko Guyer Tue, 09/13/2022 - 03:00

It takes more than a blockchain node to build enterprise-grade applications at scale. As a result, developers often find themselves building plumbing from scratch to make their business logic work. The release of Hyperledger FireFly changed blockchain development, offering developers a full stack of tools to build and scale secure web applications using familiar APIs. FireFly’s next-gen platform simplifies development, making it easy to connect across multiple public and private chains while running many use cases simultaneously. Whether you want to build on permissioned chains like Hyperledger Fabric, Corda, or Enterprise Ethereum, or public chains like Ethereum, Polygon, Avalanche, Optimism, BNB Chain, Arbitrum, Moonbeam, or Fantom, FireFly has you covered.

In this article I'll walk you through where to download Hyperledger FireFly, how to set up a local development environment, and introduce you to the FireFly Sandbox. But first, a quick introduction to the Supernode.

What is a Supernode?

Hyperledger FireFly is an open source project that was contributed to the Hyperledger Foundation by Kaleido, a blockchain and digital asset platform provider. To make FireFly a reality, the Raleigh, NC-based company collaborated with the blockchain community to fit vital technology components into an enterprise-grade, pluggable development and runtime stack called a Supernode.

Image by:

(Nicko Guyer, CC BY-SA 4.0)

This development stack offers three key advantages to blockchain developers, especially those looking to build enterprise-grade applications with scale.

• Accelerate: Hyperledger FireFly helps developers create applications on the blockchain protocol of their choice, and build quickly with familiar REST APIs. Users can leverage pre-built services for tokens, wallets, storage, and identity to reach production faster.
• Orchestrate: Hyperledger FireFly makes it easier to manage data end-to-end from blockchain to back-office. APIs allow developers to trigger business processes based on blockchain activities, and off-chain storage and messaging to protect sensitive data.
• Support: Hyperledger FireFly supports high-volume workloads, integrates with existing IT systems and databases, and communicates with network participants.

Getting Started with Hyperledger

FireFly makes it super easy to build powerful blockchain applications. Installing the stack on your machine is a simple process, too. Below I'm going to walk you through the three step process to get up and running so you can start testing the FireFly functionality today.

Image by:

(Nicko Guyer, CC BY-SA 4.0)

Install FireFly CLI

The FireFly command-line interface (CLI) creates local FireFly stacks for offline development of blockchain apps. Having FireFly locally allows developers to test and iterate on ideas without worrying about setting up extra infrastructure.

The easiest way to install the FireFly CLI is to download a pre-compiled binary of the latest release. To do this, visit the release page.

Next, extract the binary and move it to /usr/bin/local. Assuming you downloaded the package from GitHub into your Downloads directory:

$ sudo tar -zxf ~/Downloads/firefly-cli_*.tar.gz -C /usr/local/bin ff

This places the ff executable in /usr/local/bin.

If you downloaded the package from GitHub to a different directory, change the tar command above to wherever the firefly-cli_*.tar.gz file is located.

Alternatively, you can install the FireFly CLI using Go. If you have a local Go development environment, and you have included ${GOPATH}/bin in your path, you can use Go to install the FireFly CLI by running:

$ go install github.com/hyperledger/firefly-cli/ff@latest

Finally, verify the installation by running ff version. This prints the current version:

{ "Version": "v1.1.0", "License": "Apache-2.0" }

With the FireFly CLI installed, you are ready to run some Supernodes on your machine.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Start Your Environment

A FireFly stack is a collection of Supernodes that work together on a single development machine. A stack has multiple members (also referred to as organizations). Every member has their own Supernode within the stack. This allows developers to build and test data flows with a mix of public and private data between various parties, all within a single development environment.

Image by:

(Nicko Guyer, CC BY-SA 4.0)

Creating a new FireFly stack is relatively easy. The ff init command creates a new stack for you, and prompts you for a few details such as the name, and how many members you want in your stack.

There are also some settings you can change. The defaults are the simplest way to get going, but you can see a full list of options by running ff init --help.

Once you've created your stack, use the command ff start dev to run your environment.

After your stack has started, it prints the links to each member's UI, and the Sandbox for that node.

Use the FireFly Sandbox Image by:

(Nicko Guyer, CC BY-SA 4.0)

Each member gets an instance of the FireFly Sandbox as well. The Sandbox is like an example application. It can be used to test, iterate, and practice using FireFly features. It provides code snippets as examples of how to build those features into your own application backend.

There are a couple of things in the Sandbox you may want to check out to experience the full capabilities of Hyperledger FireFly.

The Messages tab helps you send messages, and view the data payload, in every member's FireFly Explorer, or send a private message to one member, and verify that the data payload is not visible in a third member's FireFly Explorer. You can send an image file, and download it from another member's FireFly Explorer.

The Tokens tab creates a non-fungible token pool, and allows you to transfer an NFT to another member and verify the account balances in FireFly Explorer.

The Contracts tab can create a contract interface and API, then view the Swagger UI for your new API, or create an event listener. You can also use the Swagger UI to call a smart contract function that emits an event. Any event received in the Sandbox also shows up in FireFly Explorer.

Build your app

Hyperledger FireFly brings a complete open source stack for developers who want to build and scale secure, enterprise-grade applications with access to blockchain technology. It's simple to install on your machine, and the Sandbox allows developers to view code snippets and test ideas—all so blockchain applications reach production faster. Read more about Hyperledger FireFly's capabilities in the project documentation and give it a try yourself.

Hyperledger FireFly brings a complete open source stack for developers who want to build and scale secure, enterprise-grade applications with access to blockchain technology.

Image by:

Opensource.com

Blockchain What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Remixing Linux for blind and visually impaired users Vojtech Polasek Tue, 09/13/2022 - 03:00

When I was around 5 years old, my father brought home our first computer. From that moment on, I knew I wanted to pursue a career in computers. I haven't stopped hanging around them since. During high school, when considering which specific area I wanted to focus on, I started experimenting with hacking, and that was the moment I decided to pursue a career as a security engineer.

I'm now a software engineer on the security compliance team. I've been at Red Hat for over two years, and I work remotely in the Czech Republic. I've used Linux for about 12 years, mainly Arch Linux and Fedora, but I've also administered Debian, Gentoo, and Ubuntu in the past.

Image by:

(Vojtech Polasek, CC BY-SA 4.0)

Photo description: Black and white image of a smiling Vojtech, with a red frame around it and an illustrated paper airplane in the background.

Outside of my day job, I play blind football, and I'm involved in various projects connecting visually impaired and sighted people together, including working in a small NGO that runs activities for blind and visually impaired people. I'm also working on an accessible Fedora project currently called Fegora, an unofficial Linux distribution aimed at visually impaired users.

The assistive technology stack

When I use a smart device, I need several pieces of assistive technology. The first and most essential is called a screen reader. This is software that presents what's on the screen to blind or visually impaired people, either through speech or through braille (basically, it tries to serve as our eyes). It can read out notifications and tell me which button or page element I'm focusing on, allowing me to interact with graphical user interfaces.

Screen readers use speech synthesis to speak aloud what appears on the screen. There are a variety of speech synthesizers, and some voices are more "natural-sounding" than others. The one I use, Espeak, is not very natural-sounding, but it's lightweight and fast. It also supports almost all languages, including Czech (which I use).

Finally, I use a Braille display, a device that represents a line of text in Braille. I use this a lot, especially when I'm coding or doing code reviews. It's easier to grasp the structure of code when I can freely move from one code element to another by touch. I can also use its buttons to move the cursor to the character or area of the screen I'm interested in, and it has a Braille keyboard too if I want to use it.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles How I use assistive technology on a daily basis

When using a computer as a blind or visually impaired person, there are a couple of things that are relatively straightforward to do using the tech above. Personally, these are a few of the things I do every day:

• The text console is pretty much my favorite application. As a general rule, when something's in text, then blind people can read it with a screen reader (this doesn't hold true in all cases, but in most.) I mainly use the console for system management, text editing, and working with guidance and documentation.
• I browse the web and interact with websites.
• I code and do code reviews using VSCode and Eclipse.
• I send emails and instant messages.
• I can use word processing software, like Google Docs (which is not open source, but common in the modern office) and LibreOffice. Google Docs developers have added a lot of keyboard shortcuts, which I can use to move around documents, jump to headings or into comments, and so on.
• I can play multimedia, usually. It depends on how the application is written. Some media players are more accessible than others.

Possible but painful

This brings me to tasks that aren't so easy. I like to call these "possible but painful".

PDF files can be difficult. Sometimes I end up needing to use optical character recognition (OCR) software to convert images to text. For example, recently I needed to read a menu for a restaurant. They had the PDF of their menu on their website, but it had been flattened, and didn't have a text layer. For me, this shows up as a blank screen. I had to use an OCR application from my smartphone to extract the text for me. Not only is this an extra step, but the resulting "translation" of the text isn't always entirely accurate.

Viewing and creating a presentation can be problematic. To work around this, I create slides in HTML, using software such as Pandoc, which can process markdown and convert it into slides. I've been using this for many years and it works well—it allows me total control of the resulting slides, because the markdown is just simple text.

Video games can be made more accessible by basing them on sound or text. However, playing games can be doubly challenging on Linux as not only would you need to find an accessible game, but most PC games are also native to Windows so you would be dealing with some compatibility issues as well.

Some websites and interfaces are more difficult to navigate than others. These issues are often quite easy to solve just by setting some attributes correctly. In general, lots of web content comes in the form of images, especially today. One of the easiest ways to make web content more accessible is to make sure that alternative text is added to images so that screen readers can read it out, and people who cannot distinguish the image have some idea what's there. Another thing I experience a lot is unlabeled controls: you know there's a button or a check box but you don't know what it does.

The Fegora project optimises Linux for accessibility

Developers don't intentionally set out to build applications that aren't accessible. The problem is that they usually don't know how to test them. There aren't many blind Linux users, so there aren't many people testing the accessibility of applications and providing feedback. Therefore, developers don't produce accessible applications, and they don't get many users. And so the cycle continues.

This is one thing we hope to tackle with the Fegora project. We want to create a Fedora remix that's user-friendly for visually impaired and blind users. We hope it will attract more users, and that those users start discovering issues to report, which will hopefully be solved by other developers in the open source community.

So why are we doing this? Well, it's important to point out that Fedora is not an inaccessible distribution by design. It does have many accessibility tools available in the form of packages. But these aren't always present from the beginning, and there are a lot of small things which need to be configured before it can be proficiently used. This is something that can be discouraging to a beginner Fedora user.

We want Fegora to be as friendly and predictable for a blind user as possible. When a user launches a live image, the screen immediately starts being read as soon as a graphical user interface appears. All environment variables needed for accessibility are loaded and configured correctly.

Fegora brings the following changes, among others:

• Environment variables for accessibility are configured from the start.
• The Orca screen reader starts as soon as the graphical interface loads.
• A custom repo is added with extra voice synthesis and packaged software.
• Many alternative keyboard shortcuts have been added.
• There's a special script that can turn your monitor on and off. Many users do not need the monitor at all and having it off is a great power saver!

So how can you help?

First, if you'd like to contribute to Fegora (or just spread the word), you can find out more on our repository.

Additionally, when working on a team with someone who has a visual impairment, there might be some additional considerations depending on the accessibility tech being used. For example, it's not easy for us to listen to someone and read at the same time, because we are basically getting both things through audio, unless someone is very proficient with the Braille display.

Lastly, bear in mind that blind and visually impaired users consume the same end products as you do, whether that's presentation slides or websites or PDFs. When building products or creating content, your choices have a huge effect on accessibility and how easy it is for us to engage with the end result. Know that we are here, we love to use computers and technology, and we're often willing to help you test it, too.

Image by:

(Vojtech Polasek, CC BY-SA 4.0)

 Image description: Vojtech holding a football. He is wearing a football uniform and protective goggles.

Fegora, a Fedora project, is an unofficial Linux distribution aimed at visually impaired users.

Image by:

Opensource.com

Linux Accessibility Diversity and inclusion What to read next Use this open source screen reader on Windows Usability and accessibility start with open communication This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

IBM upgrades Linux mainframe, boosting availability and AI performance  VentureBeat

How to set up Jellyfin on Linux in Docker  AddictiveTips

LPC 2022: Rust Linux Drivers Capable Of Achieving Performance Comparable To C Code  Phoronix

Held today during the first day of Linux Plumbers Conference 2022 in Dublin was a Rust mini-conference about the ongoing work on making Rust a suitable systems programming language and integrating support for Rust within the mainline Linux kernel. There were many interesting talks from the status of the Rust integration from the Linux kernel to a Rust-written NVMe driver that can perform as well as the C written driver...

Google for quite some time now has been working on "Ghost" as a means of controlling the Linux kernel scheduler from user-space and/or eBPF programs. Ghost provides an extensive API so developers can alter the kernel's scheduler behavior from user-space or eBPF and fine-tune the scheduling behavior based on system preferences...

3 Ways to Screenshot Right-Click Context Menus on Linux  MUO - MakeUseOf

LinkedIn Donates Feature Store to Linux Foundation  Datanami

Since NTFS3 was mainlined last year in the Linux kernel as a modern NTFS read/write file-system driver developed by Paragon Software, it's mostly just been some fixes since then and other minor updates. A new NTFS3 patch series sent out today is at least preparing a new feature for this kernel driver...

Linux's Modern NTFS Driver Preparing A "hidedotfiles" Option  Phoronix

One of many changes to find with next month's Ubuntu 22.10 release is Debuginfod integration...

Ubuntu 22.10 Adds Debuginfod Integration  Phoronix

This fearsome new Linux malware will send a shudder down the spines of IT professionals  TechRadar

AMD Utilizes User-Space Hinting To Achieve Faster EPYC CPU Performance in Linux  Wccftech

PyTorch gets lit under The Linux Foundation  The Register

Looking to switch to Linux from Windows? The Feren OS Transfer Tool makes it easy  ZDNet

Building a Retro Linux Gaming Computer - Part 18: Run Away and Join the Circus  GamingOnLinux

Building a Retro Linux Gaming Computer - Part 18: Run Away and Join the Circus  GamingOnLinux