opensource.com

Our favorite Linux replacements for antiquated open source tools Opensource.com Wed, 08/10/2022 - 03:00 1 reader likes this 1 reader likes this

Here at Opensource.com, we thought it would be interesting to survey some of our authors to get a feel for what tools they feel are antiquated (but perhaps still useful!) and what they think of the replacement utilities. What follows is a series of responses and a bit of fun, too.

We sent out the following prompt:

• Have you discovered some of your favorite tools have become outdated or deprecated? Or maybe you just switched it up for something new?
• What do you use now? Tell us a little about how you feel it is helpful to have made the switch.

Firewalls

A biggie for me is iptables. I sweated blood learning how to use iptables, and ebtables, and arptables, and how to manipulate MAC addresses, and more. I built dozens of firewalls around scripts to set up rulesets, and I eventually got pretty good at it. Now nftables makes all that obsolete. The fun never stops. I still think somebody with marketing clout could make software-defined boundaries work. —Greg Scott

+1 on iptables

I have been using iptables since I first learned Linux 25+ years ago. The newest tool is firewalld, but that and all other firewall tools I have seen for Red Hat-based distros are still based on and wrap around the kernel-level netfilter modules. I find the firewalld tool creates huge sets of rules that don't do anything more for me than the older iptables. I am sure some large environments need those complex rulesets, but they could also be implemented using iptables or scripts like Greg's.

I do like nmcli, but it is taking me some time to learn it. In fact, I prefer it to the old ifcfg and ip commands. It feels more integrated into the system than the older ones. But I do like the older ifcfg- interface configuration files. Those are easy to create and understand. They don't require the INI-style format that requires section headers. In fact, the old-style files are not even sequence sensitive. —David Both

ipchains?

To further underscore this example, are you sure you weren't using ipchains back then? (The ipfirewall and ipfwadm successor, ipchains wasn't supplanted by iptables until around 2001.) —Jeremy Stanley

In response to Jeremy. ^^

My very first firewall was ipchains, circa late 1999. Everything after that was iptables. Back then, I had to build my own kernel to get all the netfilter modules I needed. Modern conveniences like flat-panel monitors and DSL were science fiction in those days. And don't even think about fiber. I had to ride a horse uphill through blizzards every day to visit customers. And then it was uphill back home, too. —Greg Scott

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Text editing

I just have to ask—who's still using troff (groff) and who has moved on to... hmm, shall we say, LibreOffice or AsciiDoctor or...?

I have a dear friend who continues with a troff-based product on his Sun SPARCStation V. —Chris Hermansen

[ Related read Old-school technical writing with groff ]

Editing man pages

^^ In response to Chris

Anyone maintaining man pages! Though lots of people are probably generating those from other markup these days. Some folks (like me) do still compose or edit the troff files directly instead. —Jeremy Stanley

Markup stacks

There are always people who use older things, but there are superior tools nowadays. I wouldn't use LibreOffice for the kind of stuff you'd use troff/groff for—if you are writing at that level, you probably depend on a text editor you know well, source-control for managing your inputs, and you are comfortable with markup languages.

That means you want to use a markup stack. There are many, including:

• Sphinx + ReST + GitHub Actions + GitHub Pages
• MkDocs + Markdown + GitLab CI + GitLab Pages
• Nikola + Jupyter Notebooks + Jenkins + (AWS S3 + CloudFront)

What is common to all the stacks is:

• A thing that pulls different input files into one coherent whole (Sphinx/MkDocs/Nikola)
• A reasonably high-level text markup language (ReST/Markdown/MD embedded in Jupyter Notebooks)
• A CI pipeline to transform those into the output format (usually a tree of HTML files, but sometimes a PDF or something)
• A place where people can download the published version (GitHub Pages, GitLab Pages, AWS S3 + CloudFront)

I'll note that these are pretty much orthogonal choices. Any reasonable generator can take any input (even MkDocs, for which it is least true, has the mkdocs-gen-files plugin so you can use Pandoc to convert stuff to Markdown). Any reasonable CI can run any generator and push to any target.

So even with the list above, there are 81 stacks available.

(Sphinx/MkDocs/Nikola) x (ReST/Markdown/Jupyter Notebooks) x (GHA/GitLab CI/Jenkins) x (GHP/GLP/S3+CF)

Because Pandoc understands troff (ms/man), you can plug troff+ms or troff+man into the "markup" slot if you really want to. You can probably install Jenkins on the Sun SPARCStation V and keep using the same machine and format. But why? :)

There's probably an article for OSDC there: "How I converted troff docs to a modern stack using mkdocs+mkdocs-gen-files and GitLab CI." —Moshe Zadka

Other groff examples

Actually, I'm writing an article right now about "Old school technical writing with groff" (part of a larger series I'm writing about tech writing tools). I don't use groff for serious tech writing, but it's in my toolkit of things I learned and will probably never forget. And I review groff when I teach "Writing with Digital Technologies."

While writing the article, I recalled that when I installed Linux in 1993, there weren't any writing apps on Linux. No word processors, just groff and LaTeX. I used LaTeX to write my physics lab reports (because it could do math easily) and groff to write papers for other classes (because I could opt to print to a line printer instead, which I thought was a clever way to make my paper look longer). If I wanted to write with a word processor, I had to dual-boot back to DOS to run WordPerfect or Galaxy Write. StarOffice came out for Linux in 1996. I bought StarOffice.

Interestingly, Brian Kernighan still writes all his books in groff. "Unix: A History and a Memoir" (2020) and "Understanding the Digital World" (2021) were both completely processed in groff. —Jim Hall

Revisiting the fmt command

I use the fmt command a lot these days. It's really handy for a ton of stuff. If you write Readme documentation (or other docs) in plain text, you know the pain when you insert some new text in the middle of a line, and then the lines don't end at the same column. You can run fmt to clean that up.

More commonly, I'm on an email list where list members prefer to receive emails in plain text, so my email client is set for plain text most of the time. If I need to reply to someone's list email (and they didn't send it in plain text), a paragraph is usually just one long line, and my email client doesn't correctly line-wrap when I reply. It's just > at the start of a long sentence.

So I do this:

$ fmt -p '>' > /tmp/f
{copy & paste ">" quoted text}
^D

And then:

$ cat /tmp/f

And then copy and paste the result into my email. —Jim Hall

Changes to bootloaders

Just when your foo is sufficiently sharp, there are reasonable odds the tools will be replaced.

LILO to GRUB was painful until my GRUB-foo reached a sufficient level. GRUB2 is awesome, but a new learning curve.

Muscle memory is also an issue — ipconfig, nslookup, and netstat are on auto-pilot. Plus, if you're using other Linux environments, like Tiny Core Linux, you might not always have the latest and greatest tools.

Switching from if-cfg-style scripts to nmcli is the new learning curve, so this never really ends. —Steven Ellis

[ Related read 6 deprecated Linux commands and the tools you should be using instead ]

Quick FIPS set up

Often things change for the better; my two cents. The question was asked, Have you discovered some of your favorite tools have become outdated or deprecated? Or maybe you just switched it up for something new?

A colleague recently asked me how to enable FIPS on Linux, and it's something I had not done in a while. I remember how arcane this process was, which involved enabling a repo, installing a package (dracut-fips), running commands (dracut) to regenerate initramfs, modifying the GRUB bootloader config file (fips=1), etc.

Also, What do you use now? Tell us a little about how you feel it is helpful to have made the switch.

Luckily on RHEL9, the above has been replaced by the fips-mode-setup command with two handy flags, --check and --setup. That's it! Run those commands, reboot the system, and your machine boots up with FIPS enabled. Super easy! —Gaurav Kamathe

Old and comfortable

Clearly, both the fun of open source and the strong opinions are still present, as is the variety of tools and the freedom to choose what works best for you. Perhaps these tools and others like them are old—even antiquated—but they may still serve a purpose. Some of these older utilities inspired more modern solutions without losing their own inherent value. Finally, there's something to be said for user comfort and familiarity. With open source, all those hours spent developing your foo need not be lost just because some vendor decided it was time for a new release.

We asked our community of contributors what open source tools they are using in place of those that feel outdated or antiquated.

Image by:

Opensource.com

Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Create beautiful PDFs in LaTeX Jim Hall Wed, 08/10/2022 - 03:00 1 reader likes this 1 reader likes this

The LaTeX document preparation system has an interesting history. When programmer Don Knuth wrote his first book, The Art of Computer Programming, in 1968, it was produced using an old-style printing press method. When he published the second edition in 1976, the publisher had moved to modern phototypesetting.

Knuth was unhappy with how the new edition looked. Addressing the problem from a programmer's perspective, Knuth decided to create his own text processing system so his future books could be formatted to look the same way, for every book in the series. And so it was that Don Knuth wrote the first version of TeX in 1978.

A few years later, Leslie Lamport created a set of macros that help authors write complex documents more easily. Lamport's macro extensions, LaTeX, essentially extends TeX to easily produce all kinds of documents. For example, many academic organizations use LaTeX to publish journals and proceedings.

Writing documents in LaTeX

It's easy to learn the basics of LaTeX by writing a short article. Let's start by borrowing from the About Opensource.com page to create this sample input file:

$ cat about.tex
\documentclass{article}
\begin{document}

Opensource.com is a premier, daily publication focused on
open source and Linux tutorials, stories, and resources.

We're a diverse and inviting group, made up of staff
editors, Correspondents, contributors, and readers. We
value differences in skills, talents, backgrounds, and
experiences. There are a few different ways to get involved
as a reader or a writer.

\end{document}

Like other document formatting programs, LaTeX collects words and fills paragraphs. That means you can add new text in the middle of a paragraph and not worry about how the final document will look. As long as you don't add a blank line in the middle of a paragraph, LaTeX creates fully justified paragraphs. When it finds a blank line, LaTeX starts a new paragraph.

LaTeX needs a few control statements to define the document. Every LaTeX document should start with a declaration of the document's class. LaTeX supports several kinds of documents, including letters, books, and articles. For this example, I used \documentclass{article} to set the article class.

Tell LaTeX where the text begins and ends with the \begin{document} and \end{document} statements. If you add text before the \begin{document}, LaTeX generates an error. Any text after \end{document} is ignored.

Process this document using LaTeX with the latex command:

$ latex about.tex
This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=latex)
 restricted \write18 enabled.
entering extended mode
(./about.tex
LaTeX2e <2020-10-01> patch level 4
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2020/04/10 v1.4m Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))
(/usr/share/texlive/texmf-dist/tex/latex/l3backend/l3backend-dvips.def)
No file about.aux.
[1] (./about.aux) )
Output written on about.dvi (1 page, 736 bytes).
Transcript written on about.log.

LaTeX produces a lot of text so you can see what it is doing. If your document contains errors, LaTeX prints a message, and possibly prompt for what it should do. In most cases, you can type exit at the prompt to force LaTeX to quit.

If LaTeX was successful in generating a document, it produces a file with a .dvi extension. The DVI stands for Device Independent because you can use a variety of tools to create other kinds of output. For example, the dvipdf program converts the DVI file to a PDF file.

$ dvipdf about.dvi Image by:

(Jim Hall, CC BY-SA 4.0)

Adding lists

LaTeX supports two kinds of lists: an enumerated list where each item starts with a number, and an itemized or "bullet" list. Add a short enumerated list after the second paragraph to list the ways that folks can get involved with Opensource.com:

\begin{enumerate}
\item Be a writer
\item Be a reader
\end{enumerate}

Similar to how you need to provide \begin and \end statements around a document definition, you also need to provide \begin and \end statements around a list. Within the list, start each new item with an \item command. When you process this new file with LaTeX and convert it to PDF format, you see your list formatted as a numbered list:

Image by:

(Jim Hall, CC BY-SA 4.0)

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles

You can also add lists within a list. This is a neat feature if you need to provide a list with several options for each item. For example, you can add a few different resources for folks who want to become writers at Opensource.com. The embedded list uses its own \begin and \end statements. I'll add some extra space around this example so it's easier to see, but LaTeX doesn't really care about the blank lines and extra spaces:

\begin{enumerate}
\item Be a writer

  \begin{itemize}
  \item Resources for writers
  \item Contributor Club
  \item Correspondent Program
  \end{itemize}

\item Be a reader
\end{enumerate}

The new list is inserted as an embedded list inside item number 1 because you added the list between the two original \item statements. You could have instead inserted this list after item number 2 by adding the new list before the \end{enumerate} statement.

Image by:

(Jim Hall, CC BY-SA 4.0)

Sections and subsections

You can make a long document easier to read by breaking it up into sections. To add a section title to a LaTeX document, use the \section{...} statement, and write the section's title inside the braces. For example, you can add a new section titled "About Opensource.com" to the top of the document with this:

$ head about.tex
\documentclass{article}
\begin{document}

\section{About Opensource.com}

Opensource.com is a premier, daily publication focused on
open source and Linux tutorials, stories, and resources.

We're a diverse and inviting group, made up of staff
editors, Correspondents, contributors, and readers. We

The article document class adds a number before each major section, and increases the font size so it stands out in the document.

Image by:

(Jim Hall, CC BY-SA 4.0)

For documents that require more organization, you can add subsections using the \subsection{...} command. Like the \section{...} command, enter the subsection's title between the curly braces.

$ head about.tex
\documentclass{article}
\begin{document}

\section{About Opensource.com}

Opensource.com is a premier, daily publication focused on
open source and Linux tutorials, stories, and resources.

\subsection{Welcome to the Opensource.com community} Image by:

(Jim Hall, CC BY-SA 4.0)

Title and author

Scientific articles meant for publication require a title, author, and publication date. LaTeX provides a method to add this information by inserting commands that define each, then generates the article's title with a separate \maketitle command.

Add "About Us" as the article's title, "Opensource.com Editors" for the author, and "July 10, 2022" as the publication date. You must enter this block after the \begin{document} and before the rest of the content, such as the first section:

\title{About Us}
\author{Opensource.com Editors}
\date{July 10, 2022}
\maketitle

When you process the document, LaTeX adds the title, author, and date to the top of the artcle:

Image by:

(Jim Hall, CC BY-SA 4.0)

Adding emphasis

Scientific and other technical documents often include terms and phrases that need to carry special emphasis. LaTeX provides several font effects you can use in technical documents, including emphasis text (usually displayed in italics), bold text, and small caps.

Update your LaTeX document to put the phrase "staff editors, Correspondents, contributors, and readers" in italics text, and the specific words "reader" and "writer" later in the paragraph in emphasis text. You can also put the phrase "skills, talents, backgrounds, and experiences" in bold. And while it's not the correct way to style it, you can use small caps to type "Linux."

$ head -20 about.tex
\documentclass{article}
\begin{document}

\title{About Us}
\author{Opensource.com Editors}
\date{July 10, 2022}
\maketitle

\section{About Opensource.com}

Opensource.com is a premier, daily publication focused on
open source and \textsc{Linux} tutorials, stories, and resources.

\subsection{Welcome to the Opensource.com community}

We're a diverse and inviting group, made up of \textit{staff
editors, Correspondents, contributors, and readers}. We
value differences in \textbf{skills, talents, backgrounds, and
experiences}. There are a few different ways to get involved
as a \emph{reader} or a \emph{writer}.

This sample shows different ways to apply different styles to text. When you need to add emphasis, use the \emph{...} command, with the word or phrase between the curly braces. To display text in italics, boldface, or small caps, use a variation of the \text command: \textit{...} for italics, \textbf{...} for boldface, and \textsc{...} for small caps. LaTeX supports lots of other ways to style text, but these styles get you pretty far in writing scientific documents.

Image by:

(Jim Hall, CC BY-SA 4.0)

Using LaTeX

I've only touched on a few ways to write scientific and technical documents in LaTeX. Depending on your needs, you can also use LaTeX to insert footnotes and typeset mathematical equations and expressions. To explore other ways to use LaTeX for scientific writing, also read A introduction to creating documents in LaTeX here on Opensource.com.

Use the LaTeX markup language to compose documents.

Image by:

Jonas Leupe on Unsplash

Linux Documentation What to read next How I use the Linux fmt command to format text How I use the Linux sed command to automate file edits Old-school technical writing with groff This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

A guide to JVM interpretation and compilation Jayashree Hutt… Tue, 08/09/2022 - 10:36 Register or Login to like Register or Login to like

Java is a platform-independent language. Programs are converted to bytecode after compilation. This bytecode gets converted to machine code at runtime. An interpreter emulates the execution of bytecode instructions for the abstract machine on a specific physical machine. Just-in-time (JIT) compilation happens at some point during execution, and ahead-of-time (AOT) compilation happens during build time. 

This article explains when an interpreter comes into play and when JIT and AOT will occur. I also discuss the trade-offs between JIT and AOT.

Source code, bytecode, machine code

Applications are generally written using a programming language like C, C++, or Java. The set of instructions written using high-level programming languages is called source code. Source code is human readable. To execute it on the target machine, source code needs to be converted to machine code, which is machine readable. Source code is typically converted into machine code by a compiler. 

In Java, however, the source code is first converted into an intermediate form called bytecode. This bytecode is platform independent, which is why Java is well known as a platform-independent programming language. The primary Java compiler javac converts the Java source code into bytecode. Then, the bytecode is interpreted by the interpreter.

Here is a small Hello.java program:

//Hello.java
public class Hello {

    public static void main(String[] args) {
         System.out.println("Inside Hello World!");
         }
}

Compile it using javac to generate a Hello.class file containing the bytecode. 

$ javac Hello.java
$ ls
Hello.class  Hello.java

Now, use javap to disassemble the content of the Hello.class file. The output of javap depends on the options used. If you don't choose any options, it prints basic information, including which source file this class file is compiled from, the package name, public and protected fields, and methods of the class.

$ javap Hello.class
Compiled from "Hello.java"
public class Hello {
  public Hello();
  public static void main(java.lang.String[]);
}

To see the bytecode content in the .class file, use the -c option:

$ javap -c Hello.class
Compiled from "Hello.java"
public class Hello {
  public Hello();
        Code:
           0: aload_0
           1: invokespecial #1                      // Method java/lang/Object."":()V
           4: return

  public static void main(java.lang.String[]);
        Code:
           0: getstatic         #2                      // Field java/lang/System.out:Ljava/io/PrintStream;
           3: ldc               #3                      // String Inside Hello World!
           5: invokevirtual #4                      // Method    
java/io/PrintStream.println:(Ljava/lang/String;)V
           8: return
}

To get more detailed information, use the -v option:

$ javap -v Hello.classInterpreter, JIT, AOT

The interpreter is responsible for emulating the execution of bytecode instructions for the abstract machine on a specific physical machine. When compiling source code using javac and executing using the java command, the interpreter operates during runtime and serves its purpose.

$ javac Hello.java
$ java Hello
Inside Hello World!

The JIT compiler also operates at runtime. When the interpreter interprets a Java program, another component, called a runtime profiler, is silently monitoring the program's execution to observe which portion of the code is getting interpreted and how many times. These statistics help detect the hotspots of the program, that is, those portions of code frequently being interpreted. Once they're interpreted above a set threshold, they are eligible to be converted into machine code directly by the JIT compiler. The JIT compiler is also known as a profile-guided compiler. Conversion of bytecode to native code happens on the fly, hence the name just-in-time. JIT reduces overhead of the interpreter emulating the same set of instructions to machine code.

The AOT compiler compiles code during build time. Generating frequently interpreted and JIT-compiled code at build time improves the warm-up time of the Java Virtual Machine (JVM). This compiler was introduced in Java 9 as an experimental feature. The jaotc tool uses the Graal compiler, which is itself written in Java, for AOT compilation. 

Here's a sample use case for a Hello program:

//Hello.java
public class Hello {


    public static void main(String[] args) {
            System.out.println("Inside Hello World!");
            }
}


$ javac Hello.java
$ jaotc --output libHello.so Hello.class
$ java -XX:+UnlockExperimentalVMOptions -XX:AOTLibrary=./libHello.so Hello
Inside Hello World!When do interpreting and compiling come into play: an example

This example illustrates when Java uses an interpreter and when JIT and AOT pitch in. Consider a simple Java program, Demo.java:

//Demo.java
public class Demo {
  public int square(int i) throws Exception {
        return(i*i);
  }


  public static void main(String[] args) throws Exception {
        for (int i = 1; i <= 10; i++) {
          System.out.println("call " + Integer.valueOf(i));
          long a = System.nanoTime();
          Int r = new Demo().square(i);
        System.out.println("Square(i) = " + r);
          long b = System.nanoTime();
          System.out.println("elapsed= " + (b-a));
          System.out.println("--------------------------------");
        }
  }
}

This simple program has a main method that creates a Demo object instance, and calls the method square, which displays the square root of the for loop iteration value. Now, compile and run the code:

$ javac Demo.java
$ java Demo
1 iteration
Square(i) = 1
Time taken= 8432439
--------------------------------
2 iteration
Square(i) = 4
Time taken= 54631
--------------------------------
.
.
.
--------------------------------
10 iteration
Square(i) = 100
Time taken= 66498
--------------------------------

More on Java What is enterprise Java programming? Red Hat build of OpenJDK Java cheat sheet Free online course: Developing cloud-native applications with microservices Fresh Java articles

The question now is whether the output is a result of the interpreter, JIT, or AOT. In this case, it's wholly interpreted. How did I conclude that? Well, to get JIT to contribute to the compilation, the hotspots of the code must be interpreted above a defined threshold. Then and only then are those pieces of code queued for JIT compilation. To find the threshold for JDK 11:

$ java -XX:+PrintFlagsFinal -version | grep CompileThreshold
 intx CompileThreshold     = 10000                                      {pd product} {default}
[...]
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

The above output demonstrates that a particular piece of code should be interpreted 10,000 times to be eligible for JIT compilation. Can this threshold be manually tuned, and is there some JVM flag that indicates whether a method is JIT compiled? Yes, there are multiple options to serve this purpose. 

One option for learning whether a method is JIT compiled is -XX:+PrintCompilation. Along with this option, the flag -Xbatch provides the output in a more readable way. If both interpretation and JIT are happening in parallel, the -Xbatch flag helps distinguish the output of both. Use these flags as follows:

$ java -Xbatch  -XX:+PrintCompilation  Demo
         34        1        b  3           java.util.concurrent.ConcurrentHashMap::tabAt (22 bytes)
         35        2         n 0           jdk.internal.misc.Unsafe::getObjectVolatile (native)   
         35        3        b  3           java.lang.Object::<init> (1 bytes)
[...]
210  269         n 0           java.lang.reflect.Array::newArray (native)   (static)
        211  270        b  3           java.lang.String::substring (58 bytes)
[...]
--------------------------------
10 iteration
Square(i) = 100
Time taken= 50150
-------------------------------- 

The output of the above command is too lengthy, so I've truncated the middle portion. Note that along with the Demo program code, the JDKs internal class functions are also getting compiled. This is why the output is so lengthy. Because my focus is Demo.java code, I'll use an option that can minimize the output by excluding the internal package functions. The command -XX:CompileCommandFile disables JIT for internal classes:

$ java -Xbatch -XX:+PrintCompilation -XX:CompileCommandFile=hotspot_compiler Demo

The file hotspot_compiler referenced by -XX:CompileCommandFile contains this code to exclude specific packages:

$ cat hotspot_compiler
quiet
exclude java/* *
exclude jdk/* *
exclude sun/* *

In the first line, quiet instructs the JVM not to write anything about excluded classes. To tune the JIT threshold, use -XX:CompileThreshold with the value set to 5, meaning that after interpreting five times, it's time for JIT:

$ java -Xbatch -XX:+PrintCompilation -XX:CompileCommandFile=hotspot_compiler \
-XX:CompileThreshold=5 Demo
        47      1       n 0     java.lang.invoke.MethodHandle::linkToStatic(LLLLLL)L (native)  
           (static)
        47      2       n 0     java.lang.invoke.MethodHandle::invokeBasic(LLLLL)L (native)  
        47      3       n 0     java.lang.invoke.MethodHandle::linkToSpecial(LLLLLLL)L (native)  
           (static)
        48      4       n 0     java.lang.invoke.MethodHandle::linkToStatic(L)I (native)   (static)
        48      5       n 0     java.lang.invoke.MethodHandle::invokeBasic()I (native)  
        48      6       n 0     java.lang.invoke.MethodHandle::linkToSpecial(LL)I (native)  
           (static)
[...]
        1 iteration
        69   40         n 0     java.lang.invoke.MethodHandle::linkToStatic(ILIIL)I (native)  
           (static)
[...]
Square(i) = 1
        78   48         n 0     java.lang.invoke.MethodHandle::linkToStatic(ILIJL)I (native)  
(static)
        79   49         n 0     java.lang.invoke.MethodHandle::invokeBasic(ILIJ)I (native)  
[...]
        86   54         n 0     java.lang.invoke.MethodHandle::invokeBasic(J)L (native)  
        87   55         n 0     java.lang.invoke.MethodHandle::linkToSpecial(LJL)L (native)  
(static)
Time taken= 8962738
--------------------------------
2 iteration
Square(i) = 4
Time taken= 26759
--------------------------------

10 iteration
Square(i) = 100
Time taken= 26492
--------------------------------

The output is still not different from interpreted output! This is because, as per Oracle's documentation, the -XX:CompileThreshold flag is effective only when TieredCompilation is disabled:

$ java -Xbatch -XX:+PrintCompilation -XX:CompileCommandFile=hotspot_compiler \
-XX:-TieredCompilation -XX:CompileThreshold=5 Demo
124     1       n       java.lang.invoke.MethodHandle::linkToStatic(LLLLLL)L (native)   (static)
127     2       n       java.lang.invoke.MethodHandle::invokeBasic(LLLLL)L (native)  
[...]
1 iteration
        187   40        n       java.lang.invoke.MethodHandle::linkToStatic(ILIIL)I (native)   (static)
[...]
(native)   (static)
        212   54        n       java.lang.invoke.MethodHandle::invokeBasic(J)L (native)  
        212   55        n       java.lang.invoke.MethodHandle::linkToSpecial(LJL)L (native)   (static)
Time taken= 12337415
[...]
--------------------------------
4 iteration
Square(i) = 16
Time taken= 37183
--------------------------------
5 iteration
        214   56        b       Demo::<init> (5 bytes)
        215   57        b       Demo::square (16 bytes)
Square(i) = 25
Time taken= 983002
--------------------------------
6 iteration
Square(i) = 36
Time taken= 81589
[...]
10 iteration
Square(i) = 100
Time taken= 52393

This section of code is now JIT compiled after the fifth interpretation:

--------------------------------
5 iteration
        214   56        b       Demo::<init> (5 bytes)
        215   57        b       Demo::square (16 bytes)
Square(i) = 25
Time taken= 983002
--------------------------------

Along with the square() method, the constructor is also getting JIT compiled because there is a Demo instance inside the for loop before calling square(). Hence, it will also reach the threshold and be JIT compiled. This example illustrates when JIT comes into play after interpretation. 

To see the compiled version of the code, use the -XX:+PrintAssembly flag, which works only if there is a disassembler in the library path. For OpenJDK, use the hsdis disassembler. Download a suitable disassembler library— in this case, hsdis-amd64.so— and place it under Java_HOME/lib/server. Make sure to use -XX:+UnlockDiagnosticVMOptions before -XX:+PrintAssembly. Otherwise, JVM will give you a warning. 

The entire command is as follows:

$ java -Xbatch -XX:+PrintCompilation -XX:CompileCommandFile=hotspot_compiler \ -XX:-TieredCompilation -XX:CompileThreshold=5 -XX:+UnlockDiagnosticVMOptions \ -XX:+PrintAssembly Demo
[...]
5 iteration
        178   56        b       Demo::<init> (5 bytes)
Compiled method (c2)    178   56                Demo::<init> (5 bytes)
 total in heap  [0x00007fd4d08dad10,0x00007fd4d08dafe0] = 720
 relocation     [0x00007fd4d08dae88,0x00007fd4d08daea0] = 24
[...]
 handler table  [0x00007fd4d08dafc8,0x00007fd4d08dafe0] = 24
[...]
 dependencies   [0x00007fd4d08db3c0,0x00007fd4d08db3c8] = 8
 handler table  [0x00007fd4d08db3c8,0x00007fd4d08db3f8] = 48
----------------------------------------------------------------------
Demo.square(I)I  [0x00007fd4d08db1c0, 0x00007fd4d08db2b8]  248 bytes
[Entry Point]
[Constants]
  # {method} {0x00007fd4b841f4b0} 'square' '(I)I' in 'Demo'
  # this:       rsi:rsi   = 'Demo'
  # parm0:      rdx     = int
  #             [sp+0x20]  (sp of caller)
[...]
[Stub Code]
  0x00007fd4d08db280: movabs $0x0,%rbx          ;   {no_reloc}
  0x00007fd4d08db28a: jmpq   0x00007fd4d08db28a  ;   {runtime_call}
  0x00007fd4d08db28f: movabs $0x0,%rbx          ;   {static_stub}
  0x00007fd4d08db299: jmpq   0x00007fd4d08db299  ;   {runtime_call}
[Exception Handler]
  0x00007fd4d08db29e: jmpq   0x00007fd4d08bb880  ;   {runtime_call ExceptionBlob}
[Deopt Handler Code]
  0x00007fd4d08db2a3: callq  0x00007fd4d08db2a8
  0x00007fd4d08db2a8: subq   $0x5,(%rsp)
  0x00007fd4d08db2ad: jmpq   0x00007fd4d08a01a0  ;   {runtime_call DeoptimizationBlob}
  0x00007fd4d08db2b2: hlt    
  0x00007fd4d08db2b3: hlt    
  0x00007fd4d08db2b4: hlt    
  0x00007fd4d08db2b5: hlt    
  0x00007fd4d08db2b6: hlt    
  0x00007fd4d08db2b7: hlt    
ImmutableOopMap{rbp=NarrowOop }pc offsets: 96
ImmutableOopMap{}pc offsets: 112
ImmutableOopMap{rbp=Oop }pc offsets: 148 Square(i) = 25
Time taken= 2567698
--------------------------------
6 iteration
Square(i) = 36
Time taken= 76752
[...]
--------------------------------
10 iteration
Square(i) = 100
Time taken= 52888

The output is lengthy, so I've included only the output related to Demo.java.

Now it's time for AOT compilation. This option was introduced in JDK9. AOT is a static compiler to generate the .so library. With AOT, the interested classes can be compiled to create an .so library that can be directly executed instead of interpreting or JIT compiling. If JVM doesn't find any AOT-compiled code, the usual interpretation and JIT compilation takes place. 

The command used for AOT compilation is as follows:

$ jaotc --output=libDemo.so Demo.class

To see the symbols in the shared library, use the following:

$ nm libDemo.so

To use the generated .so library, use -XX:AOTLibrary along with -XX:+UnlockExperimentalVMOptions as follows:

$ java -XX:+UnlockExperimentalVMOptions -XX:AOTLibrary=./libDemo.so Demo
1 iteration
Square(i) = 1
Time taken= 7831139
--------------------------------
2 iteration
Square(i) = 4
Time taken= 36619
[...]
10 iteration
Square(i) = 100
Time taken= 42085

This output looks as if it is an interpreted version itself. To make sure that the AOT compiled code is utilized, use -XX:+PrintAOT:

$ java -XX:+UnlockExperimentalVMOptions -XX:AOTLibrary=./libDemo.so -XX:+PrintAOT Demo
         28        1         loaded        ./libDemo.so  aot library
         80        1         aot[ 1]   Demo.main([Ljava/lang/String;)V
         80        2         aot[ 1]   Demo.square(I)I
         80        3         aot[ 1]   Demo.<init>()V
1 iteration
Square(i) = 1
Time taken= 7252921
--------------------------------
2 iteration
Square(i) = 4
Time taken= 57443
[...]
10 iteration
Square(i) = 100
Time taken= 53586

Just to make sure that JIT compilation hasn't happened, use the following:

$ java -XX:+UnlockExperimentalVMOptions -Xbatch -XX:+PrintCompilation \ -XX:CompileCommandFile=hotspot_compiler -XX:-TieredCompilation \ -XX:CompileThreshold=3 -XX:AOTLibrary=./libDemo.so -XX:+PrintAOT Demo
         19        1         loaded        ./libDemo.so  aot library
         77        1         aot[ 1]   Demo.square(I)I
         77        2         aot[ 1]   Demo.main([Ljava/lang/String;)V
         77        3         aot[ 1]   Demo.<init>()V
         77        2         aot[ 1]   Demo.main([Ljava/lang/String;)V   made not entrant
[...]
4 iteration
Square(i) = 16
Time taken= 43366
[...]
10 iteration
Square(i) = 100
Time taken= 59554

If any small change is made to the source code subjected to AOT, it's important to ensure that the corresponding .so is created again. Otherwise, the stale AOT-compiled .so won't have any effect. For example, make a small change to the square function such that now it's calculating cube:

//Demo.java
public class Demo {

  public int square(int i) throws Exception {
        return(i*i*i);
  }

  public static void main(String[] args) throws Exception {
        for (int i = 1; i <= 10; i++) {
          System.out.println("" + Integer.valueOf(i)+" iteration");
          long start = System.nanoTime();
          int r= new Demo().square(i);
          System.out.println("Square(i) = " + r);
          long end = System.nanoTime();
          System.out.println("Time taken= " + (end-start));
          System.out.println("--------------------------------");
        }
  }
}

Now, compile Demo.java again:

$ java Demo.java

But, don't create libDemo.so using jaotc. Instead, use this command:

$ java -XX:+UnlockExperimentalVMOptions -Xbatch -XX:+PrintCompilation -XX:CompileCommandFile=hotspot_compiler -XX:-TieredCompilation -XX:CompileThreshold=3 -XX:AOTLibrary=./libDemo.so -XX:+PrintAOT Demo
         20        1         loaded        ./libDemo.so  aot library
         74        1         n           java.lang.invoke.MethodHandle::linkToStatic(LLLLLL)L (native)   (static)
2 iteration
sqrt(i) = 8
Time taken= 43838
--------------------------------
3 iteration
        137   56        b            Demo::<init> (5 bytes)
        138   57        b            Demo::square (6 bytes)
sqrt(i) = 27
Time taken= 534649
--------------------------------
4 iteration
sqrt(i) = 64
Time taken= 51916
[...]
10 iteration
sqrt(i) = 1000
Time taken= 47132

Though the old version of libDemo.so is loaded, JVM detected it as a stale one. Every time a .class file is created, a fingerprint goes into the class file, and a class fingerprint is kept in the AOT library. Because the class fingerprint is different from the one in the AOT library, AOT-compiled native code is not used. Instead, the method is now JIT compiled, because the -XX:CompileThreshold is set to 3.

AOT or JIT?

If you are aiming to reduce the warm-up time of the JVM, use AOT, which reduces the burden during runtime. The catch is that AOT will not have enough data to decide which piece of code needs to be precompiled to native code.  By contrast, JIT pitches in during runtime and impacts the warm-up time. However, it will have enough profiling data to compile and decompile the code more efficiently.

Use interpretation, just-in-time compilation, and ahead-of-time compilation efficiently by understanding the differences among them.

Image by:

Image by WOCinTech Chat, CC BY 2.0

Java What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Fix file permission errors on Linux Seth Kenlon Mon, 08/08/2022 - 03:00 Register or Login to like Register or Login to like

If you're sharing files between two users over the network or "sneaker net" (saving a file to a hard drive and copying it to a computer), you may encounter permission errors when you try to read or write the file. Even if you understand the concept of file permissions, you may not know exactly how to diagnose the problem or solve it. I used to perform data migration as a service, so I've run into my fair share of permission errors and ownership conflicts. Here's how I fix them fast.

1. Determine the correct user

Before you can fix a permission error, you must determine who requires permission. You might think you already know that, but what you may not realize is that the user name isn't the most definitive attribute of user identity. Your computer doesn't see you as a person so much as it sees you as a number. To learn your number, take a look at your user ID:

$ id --user
10052. Get the current owner

Next, determine the owner of the file you're unable to interact with. Because there's a file permission problem happening, you may need to use the sudo command to see the information about the file:

$ sudo ls --numeric-uid-gid
-rw------- 1 1000 100  23041 Aug  2 05:26 bar
-rw------- 1 1000 100  54281 Aug  2 04:58 baz
-rw------- 1 1000 100    822 Aug  2 08:19 foo

In this example, the user owning the files is identified as user ID 1000, and that's why user ID 1005 can't interact with them. Worse yet, the files are marked as readable and writable only by the user that owns them, so not even members of the same group can interact with the files.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles 3. Change permissions to match

You know the user requiring permission, so you can change the current owner to match your current user:

$ sudo chown 1005 foo

You can also grant members of your group, and possibly other users on the system, access to the files by changing the file mode. For instance, to maintain read and write permissions (7) while granting read permissions (4) to the group and any other user:

$ sudo chmod 744 fooLearn more

File permissions can seem tricky when you're not comfortable with them. For more information on how file ownership works, read Introduction to chown. For more information on how file permissions work, read Introduction to chmod.

Don't let file permissions slow you down. Here's how to manage them on Linux and macOS.

Image by:

Opensource.com

Linux Mac What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How open organizations can harness energy disruptions Ron McFarland Mon, 08/08/2022 - 03:00 Register or Login to like Register or Login to like

Many people talk a lot about the values of Open Organization Principles, but in many ways, they require people to change how they do things, which can be difficult. That is true for businesses and industries as well. Disruption in many sectors is coming. How do we use Open Principles to address them? This article looks at what's happening in industries related to energy and transportation when it comes to drastic costing changes that will lead to industrial disruption.

Business disruption is happening through new technology or methods, which will slash costs. This is forcing industrial change. Consider the oil, coal, natural gas, nuclear, petroleum, biofuels, and charcoal (the primary energy in many developing countries) industries. All these industries are grouped in the fossil fuel-burning energy-generating industry. Imagine them all becoming obsolete and totally replaced by the solar and wind industries in the next decade or so because of costs. That is industrial disruption.

As a resource, I have read, Clean Disruption of Energy and Transportation, by Tony Seba.

The book was written before 2014, but many of his concepts are valid today. In his book, Seba presents an energy-usage case for electric vehicles (EV) replacing internal combustion engine vehicles (ICE), and the entire automotive industry shrinking with autonomous (self-driving) car disruption strictly on cost factors. If this is true, adapting to those changes will be required, and Open Organization Principles can be helpful.

He also builds a case for current electrical power generation being completely replaced in the years ahead by solar and wind with battery storage based strictly on cost competitive advantages. I discussed this in my article on Jeremy Rifkin's book The Zero Marginal Cost Society: The Internet of Things, the Collaborative Commons, and the Eclipse of Capitalism.

Seba's book reminds me too of Michael E Porter's five competitive forces the force of substitutes.

For example, automobiles replaced the horse and carriage in about a decade in the early 1900s, and digital cameras replaced all film cameras on the market. This replacement was due to:

1. A cost advantage with prices coming down at exponential rates (from dollars to pennies like computer sizes to computing capacity) through increased innovation, scale, and competition.
2. Superior product features.
3. Added new product features,
4. Central control replaced by distributed/shared control.

For these reasons, in the past, products became obsolete by something innovative. They were disrupted.

Imagine wanting to make an open organization community that can produce its own electricity instead of being forced to buy electricity at much higher prices. Could such a community be formed? Throughout his book, Seba makes a case that this is possible using the four above factors.

This article is the first of a two-part series on this topic. This article gives strategies for open organization communities on the use of electric vehicles over internal combustion engined (ICE) cars, the use of self-driving vehicles over human-driving vehicles, and the use of solar power generation over nuclear power generation. I'll give more in the second part of this series.

Learn about open organizations Download resources Join the community What is an open organization? How open is your organization? A question of power

Seba assumes that electrical power is a commodity with no distinct quality difference and that there is no such thing as high-quality and low-quality electricity to the user. The only difference is the price of electricity at a given time.

The price of electricity fluctuates greatly. He also assumes that the automotive industry only has one quality, the movement of goods and people. Therefore, Seba's whole book looks at the cost (particularly end-user price) of both electricity (in kilowatt/hours) and movement (miles/kilometers over the life of a vehicle). He does a wonderful job of breaking down indirect and direct costs in great detail, including many costs I had never considered. This seems to be the same conclusion Jeremy Rifkin came to, calling it marginal costs in his book, The Zero Marginal Cost Society (cited above).

By providing general cost breakdowns, Seba enables readers to seriously consider how they and their community might use solar and wind electric power generation. One could build an open organization community based on the information this book makes available.

Coincidentally, I've just read Bill Gates' book, How to Avoid a Climate Disaster. Gates' book has one primary goal: To remove 51 billion tons of greenhouse gases from the atmosphere by 2050. Gates would love it if Seba's prediction comes true.

Seba believes that by 2030, all electricity generation and vehicles on the road will be carbon-free. Seba believes that over a 10-15 year period disruptive forces will penetrate industries in an "S" shaped growth. They will start slowly while the users adjust to this new purchasing option.

Then, once several success stories are in operation, disruption will take off like a rocket, making all old technology and processes obsolete. After several decades, the demand will flatten when the disruptive technology fully serves the market.

Image by:

(Ronald McFarland, CC BY-SA 4.0)

Here is how the "S" demand works for electrical energy according to Seba:

1. Some solar and wind systems are accepted in the market, making capital more available and less costly.
2. More creative financing plans are developed and offered.
3. Local, distributed energy generation increases.
4. The business structure of energy flips from centralized to distributed.
5. Enabling technologies, such as sensors, artificial intelligence, big data, and mobile communications, improve exponentially. Sensors, smaller and more energy efficient than ever before, will have more information on energy usage than any conventional utility could ever obtain.
6. The increasing cost of resource-based energy will push more people away from them.
7. Complementary markets, such as wind power generation, electric vehicles, and self-driving cars will increase exponentially.
8. Investments in other storage technologies for shared solar facilities, wind operations, electric vehicles, and self-driving cars will increase.
9. The business environment of energy will become more and more distributed.
10. The conventional command-and-control energy business model will enter a vicious cycle of rising prices and obsolete assets.

Seba writes, "World demand for energy is expected to be 16.9 TW by 2030, according to the US Energy Information Agency." That may seem dire, but Seba adds, "Should solar continue on its exponential trajectory, the energy generation will be 100% solar by 2030."

I'm sure that Gates would be delighted if that turns out to be true, but he is taking no chances on just those predictions. Gates is investing in future nuclear power plants and carbon capture systems for current fossil fuel burning plants to reduce carbon dioxide emissions further.

Building a community

Assume we want to build an open organization community to introduce three things:

• The use of electric vehicles (EV) over internal combustion engined (ICE) vehicles.
• Self-driving vehicles over human-driven vehicles.
• Solar power generation over nuclear power generation.

Here is additional detail on Seba's arguments for each.

1. Use of electric over internal combustion engined vehicles

Consider the following attributes of EV over ICE vehicles:

• Seba presents that an electric vehicle is inexpensive to operate. An EV is five times more efficient in energy use than ICE vehicles. That makes an EV 10 times cheaper than ICE.
• An electric vehicle is cheaper to maintain because it contains few moving parts.
• In the future, there will be wireless charging stations along the road, so vehicles need not stop to charge. ICE vehicles will always have to stop at filling stations.
• Electric vehicles are safer, with more safety sensors.
• By 2025, EVs will likely be cheaper to purchase than ICE, with battery costs coming down rapidly.
• Battery performance and range are increasing with new developments.
• Electric vehicles can gather driving data in real-time for added safety, performance analytics, and future improvements.
• Electric vehicles can use excess electricity in the owners' home solar system or sell the excess to the local utility company.
• An EV company can offer free fuel and maintenance for five years if it wants to. No ICE vehicle manufacturer can do this.

2. Use of self-driving vehicles over human-driving vehicles

Young people are moving away from owning vehicles in favor of Uber or Lyft services. It is becoming cheaper in the long term. Ride-sharing offers many advantages, including:

• Each Uber/Lyft vehicle on the road is estimated to replace around 15 cars purchased. This result shrinks the overall vehicle numbers, impacting the need for parking lots and vehicle insurance.
• Owning an asset used less than 10% of the time is not practical. Most cars are parked most of the time, except for commercial vehicles that operate during work hours.

The next step is self-driving cars, according to Seba. When self-driving cars are fully in the market, there will be disruption in the automotive, transportation, logistics, and oil industries. Self-driving cars will be mainly purchased, owned, and operated by corporations with fleets, not individuals.

Self-driving cars provide many advantages, a few of which are listed here:

• Self-driving cars expand the transportation of people. Children, the elderly, and those with disabilities will be new customers.
• The benefits of self-driving cars include:
  • Save lives (no human error, advanced machine learning, and sensing).
  • Save time (less time on the road with less congestion or looking for parking spaces).
  • Save space (fewer roads, garages, and parking spaces required).
  • Save energy (lower fuel consumption).
  • Save money (in purchasing, operating, maintaining, and insuring for transportation).
• The cost of transportation will be cheaper, safer, and easier.

3. Use of solar power generation over nuclear power generation

Solar power generation pricing is falling rapidly and is projected to continue falling with more use. This is where the Internet of Things (IoT) comes in. While nuclear power costs will also come down, it's not competitive, particularly as decommissioning is extremely expensive.

Here are some considerations for solar power generation:

• Solar cost has improved by 1,540 times since 1970. Based on Seba's research, solar costs are expected to come down by another two-thirds by 2020 (at the time the book was written in 2014).
• Currently, operating nuclear power plants' costs are increasing and are continually over budget in cost. If their costs don't come down, they will phase out. Bill Gates is betting on improved low-cost designs.
• Solar panel costs have dropped by a factor of US$100/Wh (Watt-hour) to 65¢/Wh from 1970 to around 2012, far lower than nuclear prices.
• New nuclear power plants are expected to provide electricity at 25¢/kWh-30¢/kWh. That is not competitive, with solar being around 12.5¢/kWh. In the United States, residential customers pay around 12.5¢/kWh, and some solar power suppliers expect to reach 5.79¢/kWh soon.
• Nuclear power generation uses twice as much water as natural gas, which is 10,000 times more than solar. This places added stress on the freshwater supply.
• There are now solar manufacturing companies providing power 24 hours/day. Torresol Gemasolar is a company in Spain putting one of these facilities into operation. They use safer and cheaper molten salt thermal energy storage. This energy storage method is a tenth of the cost of Li-on batteries.
• Seba says, "Large, centralized, top-down, supplier-centric energy is on the way out. It is being replaced by modular, distributed, bottom-up, open, knowledge-based, consumer-centric energy." For solar, many users will also be producers and sharers of electricity and information.
• Solar now regularly provides 20-35% of Germany's power. That will continue with improved battery storage, especially as Germany moves away from Russian oil and gas due to the war in Ukraine. Both for residential and commercial solar plants, tens of thousands of power generation units are being installed in the United States, Europe, and Australia.
• In 2014, 430 reactors in 31 countries provided 370 GW of nuclear power capacity, mainly in France, Japan, Russia, and the United States. These most likely will be replaced with solar, wind, or 4th generation nuclear, which uses waste as fuel. Europe will shut down 80% of its 186 nuclear power plants by 2030 (Global Data). Many current nuclear power plants are close to their shutdown age.
• Nuclear power plants are not insurable at any premium, so only governments protect them.

Wrap up

In the second part of this series, I'll discuss ideas regarding the use of solar power generation over oil power generation, the use of solar power generation over natural gas (with methane) power generation, the use of solar power generation over biofuels power generation, the use of solar power generation over coal power generation, and distributed electricity generation (small and simple) over conventional large utilities.

Learn strategies for communities on the use of electric vehicles over internal combustion engined cars, the use of self-driving vehicles over human-driving vehicles, and the use of solar power generation over nuclear power generation.

Image by:

Photo by Jason Hibbets

The Open Organization Science What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Why we chose the Clojure programming language for Penpot Andrey Antukh Sun, 08/07/2022 - 03:00 Register or Login to like Register or Login to like

"Why Clojure?" is probably the question we've been asked the most at Penpot. We have a vague explanation on our FAQ page, so with this article, I'll explain the motivations and spirit behind our decision.

It all started in a PIWEEK. Of course!

During one of our Personal Innovation Weeks (PIWEEK) in 2015, a small team had the idea to create an open source prototyping tool. They started work immediately, and were able to release a working prototype after a week of hard work (and lots of fun). This was the first static prototype, without a backend.

I was not part of the initial team, but there were many reasons to choose ClojureScript back then. Building a prototype in just a one-week hackathon isn't easy and ClojureScript certainly helped with that, but I think the most important reason it was chosen was because it's fun. It offered a functional paradigm (in the team, there was a lot of interest in functional languages).

It also provided a fully interactive development environment. I’m not referring to post-compilation browser auto-refresh. I mean refreshing the code at runtime without doing a page refresh and without losing state! Technically, you could be developing a game, and change the behavior of the game while you are still playing it, just by touching a few lines in your editor. With ClojureScript (and Clojure) you don't need anything fancy. The language constructs are already designed with hot reload in mind from the ground up.

Image by:

(Andrey Antukh, CC BY-SA 4.0)

I know, today (in 2022), you can also have something similar with React on plain JavaScript, and probably with other frameworks that I’m not familiar with. Then again, it's also probable that this capability support is limited and fragile, because of intrinsic limitations in the language, sealed modules, the lack of a proper REPL.

About REPL

In other dynamic languages, like JavaScript, Python, Groovy (and so on), REPL features are added as an afterthought. As a result, they often have issues with hot reloading. The language patterns are fine in real code, but they aren't suitable in the REPL (for example, a const in JavaScript evaluates the same code twice in a REPL).

These REPLs are usually used to test code snippets made for the REPL. In contrast, REPL usage in Clojure rarely involves typing or copying into the REPL directly, and it is much more common to evaluate small code snippets from the actual source files. These are frequently left in the code base as comment blocks, so you can use the snippets in the REPL again when you change the code in the future.

In the Clojure REPL, you can develop an entire application without any limitations. The Clojure REPL doesn't behave differently from the compiler itself. You're able to do all kinds of runtime introspection and hot replacement of specific functions in any namespace in an already running application. In fact, it's not uncommon to find backend applications in a production environment exposing REPL on a local socket to be able to inspect the runtime and, when necessary, patch specific functions without even having to restart the service.

Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java From prototype to the usable application

After PIWEEK in 2015, Juan de la Cruz (a designer at Penpot, and the original author of the project idea) and I started working on the project in our spare time. We rewrote the entire project using all the lessons learned from the first prototype. At the beginning of 2017, we internally released what could be called the second functional prototype, this time with a backend. And the thing is, we were still using Clojure and ClojureScript!

The initial reasons were still valid and relevant, but the motivation for such an important time investment reveals other reasons. It's a very long list, but I think the most important features of all were: stability, backwards compatibility, and syntactic abstraction (in the form of macros).

Image by:

(Andrey Antukh, CC BY-SA 4.0)

Stability and backwards compatibility

Stability and backwards compatibility are one of the most important goals of the Clojure language. There's usually not much of a rush to include all the trendy stuff into the language without having tested its real usefulness. It's not uncommon to see people running production on top of an alpha version of the Clojure compiler, because it's rare to have instability issues even on alpha releases.

In Clojure or ClojureScript, if a library doesn't have commits for some time, it's most likely fine as is. It needs no further development. It works perfectly, and there's no use in changing something that functions as intended. Contrarily, in the JavaScript world, when you see a library that hasn't had commits in several months, you tend to get the feeling that the library is abandoned or unmaintained.

There are numerous times when I've downloaded a JavaScript project that has not been touched in 6 months only to find that more than half of the code is already deprecated and unmaintained. On other occasions, it doesn’t even compile because some dependencies have not respected semantic versioning.

This is why each dependency of Penpot is carefully chosen, with continuity, stability, and backwards compatibility in mind. Many of them have been developed in-house. We delegate to third party libraries only when they have proven to have the same properties, or when the effort to time ratio of doing it in-house wasn't worth it.

I think a good summary is that we try to have the minimum necessary external dependencies. React is probably a good example of a big external dependency. Over time, it has shown that they have a real concern with backwards compatibility. Each major release incorporates changes gradually and with a clear path for migration, allowing for old and new code to coexist.

Syntactic abstractions

Another reason I like Clojure is its clear syntactic abstractions (macros). It's one of those characteristics that, as a general rule, can be a double-edged sword. You must use it carefully and not abuse it. But with the complexity of Penpot as a project, having the ability to extract certain common or verbose constructs has helped us simplify the code. These statements cannot be generalized, and the possible value they provide must be seen on a case-by-case basis. Here are a couple of important instances that have made a significant difference for Penpot:

• When we began building Penpot, React only had components as a class. But those components were modeled as functions and decorators in a rumext library. When React released versions with hooks that greatly enhanced the functional components, we only had to change the implementation of the macro and 90% of Penpot's components could be kept unmodified. Subsequently, we've gradually moved from decorators to hooks completely without the need for a laborious migration. This reinforces the same idea of the previous paragraphs: stability and backwards compatibility.
• The second most important case is the ease of using native language constructs (vectors and maps) to define the structure of the virtual DOM, instead of using a JSX-like custom DSL. Using those native language constructs would make a macro end up generating the corresponding calls to React.createElement at compile time, still leaving room for additional optimizations. Obviously, the fact that the language is expression-oriented makes it all more idiomatic.

Here's a simple example in JavaScript, based on examples from React documentation:

function MyComponent({isAuth, options}) {
    let button;
    if (isAuth) {
        button = <LogoutButton />;
    } else {
        button = <LoginButton />;
    }

    return (
        <div>
          {button}
          <ul>
            {Array(props.total).fill(1).map((el, i) =>
              <li key={i}>{{item + i}}li>
            )}
          ul>
        div>
    );
}

Here's the equivalent in ClojureScript:

(defn my-component [{:keys [auth? options]}]
  [:div
   (if auth?
     [:& logout-button {}]
     [:& login-button {}])
   [:ul
    (for [[i item] (map-indexed vector options)]
      [:li {:key i} item])]])

All these data structures used to represent the virtual DOM are converted into the appropriate React.createElement calls at compile time.

The fact that Clojure is so data-oriented made using the same native data structures of the language to represent the virtual DOM a natural and logical process. Clojure is a dialect of LISP, where the syntax and AST of the language use the same data structures and can be processed with the same mechanisms.

For me, working with React through ClojureScript feels more natural than working with it in JavaScript. All the extra tools added to React to use it comfortably, such as JSX, immutable data structures, or tools to work with data transformations, and state handling, are just a part of the ClojureScript language.

Guest Language

Finally, one of the fundamentals of Clojure and ClojureScript ​​is that they were built as Guest Languages. That is, they work on top of an existing platform or runtime. In this case, Clojure is built on top of the JVM and ClojureScript on top of JavaScript, which means that interoperability between the language and the runtime is very efficient. This allowed us to take advantage of the entire ecosystem of both Clojure plus everything that's done in Java (the same is true for ClojureScript and JavaScript).

There are also pieces of code that are easier to write when they're written in imperative languages, like Java or JavaScript. Clojure can coexist with them in the same code base without any problems.

There's also an ease of sharing code between frontend and backend, even though each one can be running in a completely different runtime (JavaScript and JVM). For Penpot, almost all the most important logic for managing a file's data is written in code and executed both in the frontend and in the backend.

Perhaps you could say we have chosen what some people call a "boring" technology, but without it actually being boring at all.

Trade-offs

Obviously, every decision has trade-offs. The choice to use Clojure and ClojureScript is not an exception. From a business perspective, the choice of Clojure could be seen as risky because it's not a mainstream language, it has a relatively small community compared to Java or JavaScript, and finding developers is inherently more complicated.

But in my opinion, the learning curve is much lower than it might seem at first glance. Once you get rid of the it's different fear (or as I jokingly call it: fear of the parentheses), you start to gain fluency with the language very quickly. There are tons of learning resources, including books and training courses.

The real obstacle I have noticed is the paradigm shift, rather than the language itself. With Penpot, the necessary and inherent complexity of the project makes the programming language the least of our problems when facing development: building a design platform is no small feat.

This article originally appeared on the Kaleidos blog and has been republished with permission. 

Though it is not a mainstream language, Clojure was the right language for Penpot to choose because of its key features: stability, backwards compatibility, and syntactic abstraction.

Art and design Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Old-school technical writing with groff Jim Hall Sat, 08/06/2022 - 03:00 Register or Login to like Register or Login to like

One of my favorite stories about Unix is how it turned into a text processing system. Brian Kernighan tells the story in his book Unix: A History and a Memoir (chapter 3) but to summarize: The Unix team at Bell Labs ran the original Unix on a PDP-7 computer, but it was a tiny system and didn't have sufficient resources to support new work. So Ken Thompson and others lobbied to purchase a new PDP-11 computer. Management denied the request. Around the same time, the Patents department planned to buy a new computer platform to produce patent applications using proprietary document formatting software. The Unix group proposed that the Patents department instead buy a new PDP-11 for the Unix team, and the Unix team would create formatting software for them.

That new formatting system was called nroff, short for "New Roff," an updated version of a text formatting program called Roff from a 1960s computer system. The name Roff came from the old expression, "I'll run off a document."

Basic formatting with nroff

By default, nroff collects words and fills paragraphs. When nroff encounters a blank line, it starts a new paragraph. For example, start with this article's introduction, which is only a few paragraphs long:

$ cat intro
Old-school technical writing with groff
Jim Hall
 
One of my favorite stories about Unix is how it turned
into a text processing system. Brian Kernighan tells the
story in his book Unix: A History and a Memoir (chapter 3)
but to summarize:
The Unix team at Bell Labs ran the original Unix on
a PDP-7 computer, but it was a tiny system and didn't
have sufficient resources to support new work. So Ken
Thompson and others lobbied to purchase a new PDP-11
computer. Management denied the request. Around the same
time, the Patents department planned to buy a new computer
platform to produce patent applications using proprietary
document formatting software. The Unix group proposed
that the Patents department instead buy a new PDP-11 for
the Unix team, and the Unix team would create formatting
software for them.
 
That new formatting system was called nroff, short for
"New Roff," an updated version of a text formatting program
called Roff from a 1960s computer system. The name Roff
came from the old expression, "I'll run off a document."

If you process this file with nroff, lines are "glued" together so the output is paragraphs with full justification. Using nroff also hyphenates words, if that helps balance lines in the text:

$ nroff intro | head
Old‐school technical writing with groff Jim Hall
 
One  of  my  favorite  stories about Unix is how it turned into a
text processing system. Brian Kernighan tells the  story  in  his
book  Unix:  A History and a Memoir (chapter 3) but to summarize:
The Unix team at Bell Labs ran the original Unix on a PDP‐7  com‐
puter,  but  it  was a tiny system and didn’t have sufficient re‐
sources to support new work. So Ken Thompson and  others  lobbied
to purchase a new PDP‐11 computer. Management denied the request.
Around the same time, the Patents department planned to buy a new

Original Unix systems used a typewriter-style printer that used 66 lines of 80 columns on a US Letter page, and nroff makes the same assumptions. It also adds empty lines so each page of output is 66 lines per page, but I've used the head command to show just the first few lines of output because my sample text isn't very long.

Breaking lines and centering text

The first two lines were meant to be separate lines of text. You can insert a formatting instruction to tell nroff to add a line break. All nroff instructions start with a dot, followed by a brief command. To add a line break, use the .br instruction between the first and second line:

Old-school technical writing with groff .br Jim Hall

When you process this new file, nroff prints the title and author on separate lines:

$ nroff intro | head
Old‐school technical writing with groff
Jim Hall
 
One  of  my  favorite  stories about Unix is how it turned into a
text processing system. Brian Kernighan tells the  story  in  his
book  Unix:  A History and a Memoir (chapter 3) but to summarize:
The Unix team at Bell Labs ran the original Unix on a PDP‐7  com‐
puter,  but  it  was a tiny system and didn’t have sufficient re‐
sources to support new work. So Ken Thompson and  others  lobbied
to purchase a new PDP‐11 computer. Management denied the request.

You can add other formatting to make this document look better. To center the top two lines, use the .ce formatting request. This takes a number argument, to indicate how many lines nroff should center. Here, you can center the top two output lines with the .ce 2 request:

.ce 2 Old-school technical writing with groff .br Jim Hall

With this added instruction, nroff correctly centers the first two lines:

$ nroff intro | head
             Old‐school technical writing with groff
                            Jim Hall
 
One  of  my  favorite  stories about Unix is how it turned into a
text processing system. Brian Kernighan tells the  story  in  his
book  Unix:  A History and a Memoir (chapter 3) but to summarize:
The Unix team at Bell Labs ran the original Unix on a PDP‐7  com‐
puter,  but  it  was a tiny system and didn’t have sufficient re‐
sources to support new work. So Ken Thompson and  others  lobbied
to purchase a new PDP‐11 computer. Management denied the request.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Adding page margins

Printing this to a printer results in text starting on the first line of the page, and against the left edge. To add a few lines of extra space from the top of the page, use the .sp request, with the number of blank lines to add:

.sp 5
.ce 2
Old-school technical writing with groff
.br
Jim Hall

By default, nroff formats the output so each line is 65 columns wide. Printing to an 80 column US Letter page leaves 15 empty columns. Adding 7 spaces on the left side neatly balances the output with equal left and right page margins. You can create this page offset using the .po 7 request:

.po 7
.sp 5
.ce 2
Old-school technical writing with groff
.br
Jim Hall

Processing the new file with nroff produces a plain text page that's ready to print:

$ nroff intro | head Old‐school technical writing with groff Jim Hall One of my favorite stories about Unix is how it turned into a text processing system. Brian Kernighan tells the story in hisPrinting to a laser printer

Later, the Unix team at Bell Labs acquired a phototypesetting machine, capable of producing printed text similar to a laser printer. To support the typesetter's new capabilities, the Unix team updated nroff to become the typesetter-specific troff program, and a few years later updated it again to become ditroff, the device-independent version of troff.

Linux systems provide modern versions of nroff and troff using the GNU groff program. You can still use the old nroff program name to generate plain text output, or troff to produce ditroffcompatible output. Using the groff program, you can also prepare documents for other kinds of output files, such as Postscript.

You can process the same input file using groffto print on a Postscript-compatible laser printer by selecting a suitable output type using the -T option, such as -Tps to generate a Postscript file. For example, I can print to a printer with the lpr command and the HP_LaserJet_CP1525nw device, because that's how my Linux system recognizes my laser printer:

$ groff -Tps intro | lpr -P "HP_LaserJet_CP1525nw"Generating other kinds of output

If you instead want to save the output as a PDF file, you can convert the Postscript using the ps2pdf tool:

$ groff -Tps intro | ps2pdf - > intro.pdf

To generate a web page from the same file, use -Thtml to set the output type to HTML:

$ groff -Thtml intro > index.html

The groff command supports lots of other built-in formatting requests to provide other kinds of document formatting. If you want to learn the other default formatting requests available to you in the GNU groff implementations of nroff and troff, refer to chapter 5 in the The GNU Troff Manual.

Formatting documents using these built-in commands takes a lot of effort to keep everything looking the same. Technical writers who use groff instead use a collection of formatting requests called macros, which provide their own commands to generate section headings, paragraphs, block quotes, footnotes, lists, and other useful document formatting. To learn more about one popular macro package, read How to format academic papers on Linux with groff -me on Opensource.com.

Take a trip back in time to experience text formatting from a bygone era.

Image by:

LSE Library. Modified by Opensource.com. CC BY-SA 4.0

Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Scalable storage for the masses to debut in ownCloud Infinite Scale Markus Feilner Fri, 08/05/2022 - 03:00 Register or Login to like Register or Login to like

So far, ownCloud users have only had the opportunity to choose between simple local file storage based on a POSIX-compatible file system or EOS Open Storage (EOS). The latter causes massive complexity during setup. More recent versions of ownCloud feature a functionality called Decomposed FS. This file system is supposed to bring oCIS to arbitrary storage backends and even scalable ones.

During the rewrite of ownCloud from PHP to Go, the company behind the private cloud solution is introducing massive changes to the backend of the software. This time, the developers do away with a tedious requirement of the first versions of ownCloud Infinite Scale (oCIS). By implementing a file system abstraction layer, oCIS can use arbitrary backend-storages (and even such storage mechanisms that can seamlessly scale out). This article explains the reasons behind ownCloud's decision, the details behind the Decomposed FS component, and what it means for ownCloud users and future deployments.

To understand the struggle that oCIS faces, it's important to have a quick look back at the genesis of ownCloud Infinite Scale. What started as an attempt to gain more performance resulted in a complete rewrite of the tool in a programming language that was different from the previous one. ownCloud did, however, partner with several companies and institutions when rewriting its core product. Amongst the organizations that ownCloud partnered with was a university in Australia. Said university used EOS, a storage technology developed by CERN, the European physics research institute notorious for running projects like the Large Hadron Collider (LHC). For the partnership between ownCloud and the Australian University to make any sense, oCIS needed to be tied to the existing on-premises storage solution of the institute.

File storage at CERN is vastly different from what the average user usually uses. While not the only experiment at CERN, the LHC certainly belongs to the experiments generating the largest amounts of data. Although CERN uses technology such as OpenStack to crunch data resulting from an active LHC, terabytes of material for later use still need to be saved somewhere daily. Accordingly, one of CERN's biggest factors regarding their own storage solution is seamless scalability. The institute's answer to that demand is a storage solution called "EOS Open Storage" (EOS). At the time of writing this article, EOS at CERN had a capacity of over 500 petabytes, making it one of the most extensive storage facilities in use at experimental institutes in the whole world.

EOS is not a file system or even anything close to it. Instead, it's a combination of numerous services with various interfaces. It implicitly takes care of redundancy, and its layout may remind some readers of solutions such as LustreFS, with central services for file metadata and file placement. EOS was obviously designed with massive amounts of data in mind. It came with the ability to scale out to dimensions that average admins will hardly ever encounter.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources EOS doesn't work for end-users

The immense scale is precisely why the close ties between oCIS and EOS soon became a challenge for the vendor and its software. The question arose, Was it also possible to run oCIS with a simple driver for local storage, just like ownCloud 10? That basic driver did not deliver all the features that ownCloud wanted to implement. On the other hand, even the most optimistic oCIS developers knew right from the start that average end users would definitely not run their own EOS instance. This was true for several reasons. First, the required hardware doesn't make sense for admins and will likely exceed budgets by several orders of magnitude. Furthermore, running a storage solution such as EOS solely for storing pictures of Aunt Sally's birthday party doesn't make much sense. In fact, EOS's complexity would soon scare most users away from oCIS. To make oCIS a success even for the average users, ownCloud had to devise a plan adapted to the storage facilities usually found in smaller environments. And that is where the Decomposed FS comes into play.

The challenges of storing files

To understand what the Decomposed FS actually is, it is necessary to remember how typical POSIX file systems work and why that might be a challenge for solutions such as oCIS. A file system is a data structure allowing the use of arbitrary storage devices. Usually, devices based on storage blocks (block devices) are used for this task. Without a file system, it would be possible to write files to a device but not to read the files in a targeted manner at a later point in time. Instead, the entire content of the storage device would need to be walked through until the specific information is found, which would, of course, be highly ineffective and very slow.

UNIX- and UNIX-like file systems nowadays mostly use file systems meeting the requirements of the POSIX standard for file systems (Figure 1). Hence they use a hierarchy-based structure, which is why you will often read about "file system trees." Accordingly, there will usually be a root directory (/) and subdirectories containing the actual files with their payload. Directories are, strictly speaking, handled as a special kind of file internally by the file system. File access is path-based. Users will implicitly or explicitly specify a full path to a file they want to open, and the respective kernel's file system driver reads the file from the storage device and presents it to the user. This process itself is already a complicated mechanism.

Image by:

Figure 1: POSIX-compatible filesystems allow users to access files based on their path only. This is slow and keeps several relevant features from being implemented properly (Markus Feilner and Martin Loschwitz, CC BY-SA 4.0).

Matters get worse, though. A path is not the only property of a file in modern operating systems. Consider file permissions, for instance. Limiting access to specific files to a certain group of people is desirable. Imagine a mail server where users only get access to their mailbox after successfully logging in with a combination of a username and password. Admins definitely do not want user A to be able to open user's B mailbox. To accommodate this need, file systems implement a permission scheme. For every arbitrary file in a file system, information on who is allowed to access it must be stored somewhere. This is where inodes come into play in POSIX-based file systems. Inodes store the so-called file metadata and have an ID associated by the file system with the actual path to the file on the storage device. Inodes and the associated files can not be accessed by the inode-id by the user. Also, inodes use recyclable IDs, which means a certain inode ID may contain the metadata of another file as compared to the situation a week ago.

"Just use this!" is not enough for oCIS

The bad news is still not over. As the above text pointed out, files on a POSIX file system cannot be accessed by a static ID at all. The traditional POSIX scheme allows for path-based access only. Of course, the kernel driver for a specific file system will internally perform the inode lookup based on the inode's ID–but that ID is not unique and intransparent to the user.

What sounds like a technical nuance is a major deal breaker for solutions such as ownCloud Infinite Scale. For it, the traditional UNIX way of handling files doesn't quite cut it for many reasons. Of course, it would be tempting to program oCIS in a way that solely relies on the underlying file system's file handling. That would make it impossible to use different drivers for different storage backends. Instead, the general assumption would need to be the availability of POSIX schematics in the storage backend behaving identically. Solutions like EOS would already be out of the game, as would solutions based on Amazon's S3 or aspiring technologies such as Ceph and its backing storage, the RADOS object store. Those interested in upstream details about the oCIS and S3NG oCIS storage should look here.

Users want features that POSIX doesn't provide

What users generally expect nowadays from solutions such as ownCloud far exceeds the abilities of the POSIX standard. Take, for instance, a function as simple as the trash bin. Users generally want a storage solution to provide such a trash bin as a place to store files temporarily before deleting them permanently. Mistakes happen, and users are often glad because they could collect files from the bin they had already considered lost. POSIX simply does not offer a trash bin-like feature. Implementing a trash bin in a typical POSIX file system would mean iterating through the whole tree and moving the files within the file system. That would be highly inefficient, consume loads of resources, and take quite a while to complete if, for instance, a user moved a directory with thousands of files to the bin.

If in place, kernel caching helps manage performance issues associated with such operations. It's a poison in disguise, though. Relying on it adds another constraint on the file system driver on the storage device used by oCIS.

From ownCloud's point of view, moving the files around on the file system wouldn't even make sense. After all, ownCloud would only need to know that certain files are marked as "trash," and it wouldn't really matter where the actual data resides on the physical storage device.

Metadata: "Here be dragons" from an oCIS point of view

Again, directly editing metadata in inodes by an inode-ID is out of the question as inode-IDs are not unique for the recently mentioned reasons. This is problematic from another perspective: Attaching additional metadata efficiently for solutions such as ownCloud is challenging. Modern file systems support user-extended attributes, an arbitrary set of properties to be defined in the user landscape of a system. Because there is no secure way to reliably access the metadata of any given file in the tree without traversing it, the theoretically available features remain mostly unused.

Long story short: Traditional POSIX semantics comes with some problems for solutions like oCIS. The two major issues are the inability to access files with a unique ID and the inability to access file metadata without traversing the whole file system tree.

What the oCIS developers call Decomposed FS nowadays is their attempt to decouple ownCloud's own file handling from the handling of files on the underlying storage device. Decomposed FS got its name because the feature partially implements certain aspects of a POSIX file system on top of arbitrary storage facilities. In stark and direct contrast to existing solutions, everything in Decomposed FS is centered around the idea of file access based on unique IDs–and more precisely, UUIDs (Figure 2).

Image by:

Figure 2: The first incarnation of Decomposed FS uses unique IDs (UUIDs) to name files. A structure of symbolical links is additionally established to allow access by traversing a tree of files (Markus Feilner and Martin Loschwitz, CC BY-SA 4.0)

How does that work in practice?

Like normal POSIX file systems, Decomposed FS uses a hierarchy model to handle its files. A root directory even serves as an entry to the complete set of files stored in ownCloud. oCIS will create individual files with unique IDs (UUIDs) as names on the file system layer below that root directory. Within those files, the actual payload (blob) of the files will reside.

Now obviously, ownCloud will still need a way to organize its files in directories from a user's point of view. Users will still assume they can implement a hierarchical structure for their own files ("Pictures / Holidays / Bahamas 2021 / ...") independently of how the software organizes its files internally. To imitate such a structure, ownCloud developers resort to symbolic links. They create a separate tree with the "nodes " that represent the files and folders of the user-facing tree. The file metadata is stored in the extended attributes of that node. This ensures that accessing files by simply traversing over them (more precisely, the tree of symbolic links) remains possible, along with the ability to access individual files based on their unique ID directly. Another advantage is that iterating over the tree just needs to load the nodes which contain no file contents. The file contents are only loaded when the user downloads the file. Even moving files within the tree doesn't touch the file contents anymore; it just changes the nodes tree.

Image by:

Figure 3: Functions like a trash bin are not provided by the underlying file system, they need to be provided by software and Decomposed FS makes that considerably easier and less resource intensive (Markus Feilner and Martin Loschwitz, CC BY-SA 4.0).

This mode of operation brings massive advantages that become obvious quickly when looking at the earlier example of moving many files to the trash bin (Figure 3). Rather than shuffling the actual files around on the backend storage, when Decomposed FS moves files to the trash, it makes oCIS mark the inodes as "trashed" and not the actual files.

Here is a basic technical explanation. In the Decomposed FS, we only create, create-symlink, and move, which are atomic operations on POSIX. In addition, we implement locking when changing file attributes to avoid conflicts.

One factor the oCIS developers considered during the creation of Decomposed FS was to be as agnostic of the underlying storage device as possible. At this time, any given POSIX file system will do. The users' home directories are not touched. All blobs and nodes are stored inside the ocis data directory.

That said, it is safe to assume that oCIS will reliably work atop almost every file system currently supported by Linux or Windows. It will most definitely work with the wide range of file systems used as the standard ones for these operating systems.

The road ahead

oCIS people explicitly refer to Decomposed FS as a "testbed for future development." Against all odds, the oCIS file system driver for ownCloud Infinite Scale already implements most components of the Decomposed FS design as documented by ownCloud. And the developers are anything but out of ideas to improve the concept further. The only instance taking control over a Decomposed FS instance created by ownCloud is, well, ownCloud. The vendor has the luxury of some freedom to experiment and extend the Decomposed FS feature set.

Accordingly, new features are added to decomposed FS regularly, such as the ability to decouple the Decomposed FS namespace completely from the actual payload of data stored in files in oCIS. The feature has meanwhile landed in an extension of the oCIS storage driver named after the Amazon S3 protocol because it enables file storage on separate and distinct storage devices, even on devices that are not locally present. The local blobstore, which just stores the file's contents, can be replaced by a remote object store. This is where S3 makes its debut in the setup. If only the metadata needs to be stored locally and S3 is the backing storage, the user is, in fact, already equipped with seamlessly and endlessly scalable storage.

oCIS, with the oCIS driver being faster and more flexible, is not enough yet for the vendor to rest on its laurels. New features are already planned for the driver, including one designed by the ownCloud folks that is nothing short of spectacular. In future releases, ownCloud intends to do away with the requirement of storing files on a POSIX-compatible backend.

To understand why that makes sense, another look at POSIX helps. POSIX was designed to offer various guarantees when accessing the file system. These include consistency guarantees on reads and writes when renaming files and overwriting existing ones. However, many of these operations do not even occur in ownCloud setups. With ownCloud maintaining its own file system on the storage backend, overwriting files doesn't mean replacing contents on the storage device with newer content. Instead, a new blob is added, and the existing symlink is changed to point to the new file. POSIX guarantees would not be an issue if they weren't consuming performance that would otherwise be available to the user.

ownCloud folks are not the first to run into this issue. The developers of the scalable object store RADOS, the heart of Ceph, have long used XFS as the on-disk format before giving up on it and creating BlueStore (Figure 4). BlueStore does away with the need to have a POSIX file system at all on the physical storage device. Instead, Ceph simply stores the name of its objects and the physical storage address in a huge key-value store based on RocksDB. ownCloud has already expressed intentions to steer Decomposed FS in a similar direction. In this setup, there is no longer a need for a local file system. Instead, details on existing files and their paths are also stored in a large RocksDB key-value store. Even the need for a local symlink tree to iterate through is removed in such an environment.

Image by:

Figure 4: Ceph has long done away with using POSIX for its on-disk file scheme. It resorts to RocksDB and a minimalistic self-written file system instead. The Decomposed FS architecture of ownCloud will soon allow it to follow suit (Markus Feilner and Martin Loschwitz, CC BY-SA 4.0).

In addition, oCIS maintaining its own file system namespace brings several major advantages compared to a simple, POSIX-compatible file system and will allow for additional features to be implemented. For instance, ownCloud could handle its files in a Git-like manner with different versions of the same file being present. With access realized through a key-value store, information about previous versions of the same file are easy to store.

Summary: oCIS for the masses

The introduction of Decomposed FS and the ability to decouple the actual file storage from the file system metadata are huge game changers from the oCIS point of view. Many home users will likely still use a single disk as a local oCIS storage backend and simply benefit from the increased performance of Decomposed FS. In stark contrast, institutions and companies will love that running oCIS in scalable storage solutions has already become much easier thanks to the new feature. Additional improvements to be implemented in the near future will, if they make it into the stable oCIS distribution, increase the versatility of oCIS even further. An ownCloud instance that writes its file blobs directly into Ceph could be very interesting for SMBs–after all, the times when even small Ceph clusters were hard to build have long gone. One way or another, the development of Decomposed FS strengthens oCIS and makes it available to a much broader audience.

ownCloud rethinks file systems.

OwnCloud What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 155 points Vienna, Austria

Martin is a jack of many trades. He started his Open Source career in the early 2000s as an active contributor to Debian GNU/Linux. He has been an active advocate of Open Source principles since then in numerous positions primarily in the areas of High Availability, Cloud Computing (OpenStack & Ceph) and process automation. He works as a journalist with a primary focus on Open Source software in his free time.

| Follow mloschwitz Open Minded Author Contributor Club Register or Login to post a comment.

Delete the local reference to a remote branch in Git Agil Antony Fri, 08/05/2022 - 03:00 Register or Login to like Register or Login to like

After you merge a GitLab or GitHub pull request, you usually delete the topic branch in the remote repository to maintain repository hygiene. However, this action deletes the topic branch only in the remote repository. Your local Git repository also benefits from routine cleanup.

To synchronize the information in your local repository with the remote repository, you can execute the git prune command to delete the local reference to a remote branch in your local repository.

More on Git What is Git? Git cheat sheet Markdown cheat sheet New Git articles

Follow these three simple steps:

1. Checkout the central branch of your repository (such as main or master) $ git checkout <central_branch_name>2. List all the remote and local branches $ git branch -a

Example output:

  4.10.z
* master
  remotes/mydata/4.9-stage
  remotes/mydata/4.9.z
  remotes/mydata/test-branch

In this example, test-branch is the name of the topic branch that you deleted in the remote repository.

3. Delete the local reference to the remote branch

First, list all the branches that you can delete or prune on your local repository:

$ git remote prune origin --dry-run

Example output:

Pruning origin
URL: git@example.com:myorg/mydata-4.10.git
* [would prune] origin/test-branch

Next, prune the local reference to the remote branch:

$ git remote prune origin

Example output:

Pruning origin
URL: git@example.com:myorg/mydata-4.10.git
* [pruned] origin/test-branch

That's it!

Maintaining your Git repository

Keeping your Git repository tidy may not seem urgent at first, but the more a repository grows, the more important it becomes to prune unnecessary data. Don't slow yourself down by forcing yourself to sift through data you no longer need.

Regularly deleting local references to remote branches is a good practice for maintaining a usable Git repository.

Follow a few simple steps to keep your Git repository tidy.

Image by:

Opensource.com

Git What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

3 ways to take screenshots on Linux Jim Hall Thu, 08/04/2022 - 03:00 Register or Login to like Register or Login to like

When writing about open source software, I prefer to show a few screenshots to help demonstrate what I'm talking about. As the old saying goes, a picture is worth a thousand words. If you can show a thing, that's often better than merely trying to describe it.

There are a few ways you can take screenshots in Linux. Here are three methods I use to capture screenshots on Linux:

1. GNOME

GNOME has a great built-in screenshot tool. Just hit the PrtScr key on your keyboard, and GNOME displays a screenshot dialog:

Image by:

(Jim Hall, CC BY-SA 40)

The default action is to grab a screenshot of a region. This is an incredibly useful way to crop a screenshot as you make it. Just move the highlight box to where you need it, and use the "grab" corners to change the size. Or select one of the other icons to take a screenshot of the entire screen, or just a single window on your system. Click the circle icon to take the screenshot, similar to the "take photo" button on mobile phones. The GNOME screenshot tool saves screenshots in a Screenshots folder inside your Pictures folder.

2. GIMP

If you need more options for screenshots, you can grab a screenshot using GIMP, the popular image editor. To take a screenshot, go to File and choose the Create submenu, and then choose Screenshot.

Image by:

(Jim Hall, CC BY-SA 40)

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles

The dialog allows you to take a screenshot of a single window, the entire screen, or just a region. I like that this tool lets you set a delay: how long until you select the window, and how long after that to take the screenshot. I use this feature a lot when I want to grab a screenshot of a menu action, so I have enough time to go to the window and open the menu.

GIMP opens the screenshot as a new image, which you can edit and save to your preferred location.

3. Firefox

If you need to take a screenshot of a website, try Firefox's built-in screenshot utility. Right-click anywhere in the web page body, and select Take Screenshot from the menu:

Image by:

(Jim Hall, CC BY-SA 40)

​

Firefox switches to a modal display and prompts you to click or drag on the page to select a region, or use one of the icons to save a copy of the full page or just what's visible in the browser:

Image by:

(Jim Hall, CC BY-SA 40)

As you move your mouse around the screen, you may notice that Firefox highlights certain areas. These are block elements on the page, such as a  or another block element. Click on the element to take a screenshot of it. Firefox saves the screenshot to your Downloads folder, or wherever you have set as your "download" location.

If you're trying to document a process, a screenshot can save you a lot of time. Try using one of these methods to take a screenshot on Linux.

Save time by taking screenshots on Linux with one of my favorite tools.

Tools Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Lengthen the life of your hardware with Linux Don Watkins Thu, 08/04/2022 - 03:00 Register or Login to like Register or Login to like

Sustainability is an increasingly important problem when it comes to computing. Reduce, reuse, recycle is a popular motto for environmentally responsible consumption, but applying that to your computer hardware can be challenging.

Many proprietary operating systems essentially force a hardware upgrade upon you long before your old hardware is used up. If you own a computer with Windows, you've probably needed to purchase a new one to upgrade because your old one didn't meet the hardware requirements of the latest OS. Apple doesn't do any better, either. A MacBook Air I owned was essentially rendered obsolete by an upgrade to macOS Mojave in 2019.

By contrast, I run Linux on my three-and-a-half-year-old laptop, and it still runs like new. Because the Linux kernel is more efficient with resources than either Windows or macOS, it can run successfully on older hardware. I've never been forced to purchase new hardware in order to upgrade Linux.

The advantage of Linux is that it is free and open source. With a few notable exceptions, most Linux distributions are available free of charge, and they are not the product of a large technology company with profit in mind. Even businesses that offer Linux products know that profitability doesn't lie in selling software and forcing updates but in stellar support of what their customers are trying to do with that software.

Simply put, Linux is the best bet for a sustainable operating system.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Making Linux accessible

There was a time when Linux users were required to be more technologically savvy than the average computer user, but those days are a thing of the past. Most Linux distributions are as plug-and-play as their proprietary counterparts. Better yet, computers that are 10 or more years old can easily run any of the popular Linux distributions without modification. Instead of defining a computer's lifespan with the arbitrary benchmark of operating system support, you can measure it instead by the life of the hardware itself. That's how it should be. Hardware, like everything else, eventually fails, but software we can control.

This improved sustainability doesn't impede my workflow, either. I have access to a wide range of free and open source software and cloud-based systems that afford me ample opportunities to be creative while keeping my aging hardware out of the landfill.

Reuse hardware, reduce electronic waste

When deciding to refurbish an older computer, first determine how it will be used. You don't need lots of processing power if you are just surfing the web and writing with a word processor. But if you are working from home and using your computer for video conferencing with Jitsi or one of the proprietary solutions, you will need more RAM and processing power. In that case, I suggest (based on what's available in 2022) looking for an Intel i5 or AMD Ryzen 5 with at least 8 GB of RAM.

I put this concept to the test with that old MacBook Air I mentioned earlier. That system was given new life when I installed Linux in January 2020.

Hardware purchases

If you're not refurbishing and just need a computer, using Linux frees you from purchasing the latest hardware. It's easy to find serviceable laptops and desktops on FreeGeek and elsewhere that provide good hosts for Linux distributions. If you buy a computer from FreeGeek, it comes with Linux preinstalled and tested, and you're contributing to a community whose mission is "to sustainably reuse technology, enable digital access, and provide education to create a community that empowers people to realize their potential."

However you get to Linux, it's well worth learning and supporting. It's a sustainable OS that puts you in control of your data, your purchases, and the way you affect the environment.

Keep hardware out of landfills by using an operating system that stretches the usable life of your devices.

Image by:

Opensource.com

Linux Hardware What to read next Restore an old MacBook with Linux How I gave my old laptop new life with the Linux Xfce desktop This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

A sysadmin's guide to network interface configuration files David Both Wed, 08/03/2022 - 03:00 Register or Login to like Register or Login to like

In the first article of this series, Get started with NetworkManager on Linux, I looked at what NetworkManager does and some of the tools it provides for viewing network connections and devices. I discussed using the nmcli command to view the status of network devices and connections on a host. I also mentioned that NetworkManager does not need interface configuration files for most hosts. However, it can create its own INI-style connection configuration files, and it recognizes the older and now deprecated network interface configuration files.

This article explores three questions:

• Why do I not use interface configuration files?
• Why do I use interface configuration files?
• Why do I use the old network interface configuration files?

Sound confusing? Read on.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles My network philosophy

I talk about philosophy a lot. I even wrote a book, The Linux Philosophy for SysAdmins, which has tenets that apply to the design and structure of computers, operating systems, and networks. I won't bore you with all the details, but there are some things to consider when designing–or redesigning–a network.

As a "Lazy SysAdmin," I like to "Find the Simplicity"–yes, those are two of the tenets–and create an elegant network design. This is not just about the physical design and layout of the network components and wiring, although the best, most elegant, and easiest networks to work on are well laid out physically and look good. However, this discussion is about the logical structure of the network.

Why I don't use interface configuration files?

I don't use interface configuration files on my network mostly because each host is dynamically configured at boot time using the Dynamic Host Configuration Protocol (DHCP) server. This allows centralized configuration and management of a few computers up to hundreds or even thousands of systems. The bottom line is that all the configuration data necessary for each host is stored in the DHCP configuration file, /etc/dhcp/dhcpd.conf, where it is centrally managed.

I impose simplicity on my network by using tools that provide central management for most of the connected hosts–all except for hosts that work as routers and provide server services. The idea is that using DHCP to provide all of the network configuration data needed by most of the network hosts simplifies my job and allows me to be the "Lazy SysAdmin." Suppose something changes on the network, such as the IP address to the default gateway or the primary name server? Changing that information in a single location, the dhcpd.conf file, is much less work than changing a static configuration on ten or a thousand hosts.

NetworkManager does not need local configuration files when DHCP provides the network configuration information. By default, all Fedora and Red Hat hosts obtain their network configuration from a DHCP server. This makes installing new hosts on a network easy and simple. All you need is a DHCP server.

For most networks with a single host, such as in a home office with one or two laptops and a few other devices, the wireless router provided by the ISP contains the DHCP server required to offer a complete set of configuration data to all your devices. The router's DHCP server provides the network configuration data even if you use the 4-port wired switch on the back of most wireless routers to connect a wired desktop computer.

Why I do use interface configuration files?

Most of my network hosts don't need static network configurations and use DHCP.

However, there are two hosts on which I do use static network configuration: My network server–the one that runs the DHCP server–and the Linux host I use for my network/firewall. These two hosts are best configured using static setups that don't depend upon external configurations.

Think about this for a minute. If the DHCP server must have an IP address to send network configuration information, including an IP address, to itself… Well, that just won't work—sort of the network equivalent of the chicken and egg situation.

DHCP clients request network configuration using a broadcast on the network, and the server responds to that request using the MAC address of the requesting client. The DHCP server cannot also be a DHCP client, so this just won't work.

Even if using the DHCP server to set its own IP address–and other network configuration attributes–could be made to work, all of the recommendations I see on the Internet suggest that it would be a really terrible idea and that no good administrator would even consider doing such a thing.

The Linux host I use for a router and firewall has four network interfaces, three of which are currently active, and one on the motherboard, which is defective. It also has a set of forwarding and routing rules that must always be consistent. This configuration is best dealt with using static network settings.

For example, one interface on my router connects to the WAN side of my wireless router. The wireless router provides an internal DHCP server for hosts connected to its LAN and WiFi side but depends upon either static or DHCP configuration on the WAN side. So I configure both the WAN side of the wireless router and the NIC that connects it to my Linux router using a static setup.

Another interface on that Linux router connects to the outside world via a static IP address provided by my ISP. If I set that interface to be configured by DHCP, the ISP's router would serve it one of the remaining other IP addresses in the eight-address block I have been assigned.

Any type of static network configuration, as opposed to DHCP, requires network configuration files.

Why I still use the old style ifcfg-<interface-name> files

The answer to this is really simple. I just have not gotten around to making the switch. These files are located in the /etc/sysconfig/network-scripts directory, and–fortunately for me–NetworkManager will still search for and use these if it has none of its own network connection files. There won't be any network connection files because they do not get created automatically, and I have not needed to create them.

I intend to make the switch in Part 3 of this series and bring my network up to current configuration practices. For now, however, it is still a good idea to know about the old-style network configuration files because there are still a lot of them around.

What I have now

I'll review the current state of the network on my router. Aside from the local loop (lo)–which is always present on Unix and Linux hosts–this host currently has three active network interface cards (NICs). Due to the problems with the on-board NIC described in Part 1 of this series, I deactivated it in UEFI/BIOS of this host so that it no longer shows up. I have also disabled IPv6 on my network as I don't need it.

The following is the nmcli command showing the state of the NICs on my router/firewall host:

[root@wally ~]# nmcli
np4s0: connected to enp4s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:04:44:03, hw, mtu 1500
        ip4 default
        inet4 45.20.209.41/29
        route4 45.20.209.40/29 metric 102
        route4 default via 45.20.209.46 metric 102
 
enp1s0: connected to enp1s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:03:E9:89, hw, mtu 1500
        inet4 192.168.10.1/24
        route4 192.168.10.0/24 metric 101
 
enp2s0: connected to enp2s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:03:FD:85, hw, mtu 1500
        inet4 192.168.0.254/24
        route4 192.168.0.0/24 metric 100
 
lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
 
DNS configuration:
        servers: 192.168.0.52 8.8.8.8 8.8.4.4
        interface: enp4s0
 
        servers: 192.168.0.52 8.8.8.8
        interface: enp2s0
 
        servers: 192.168.0.52 8.8.8.8
        interface: enp1s0

Each of these NICs has an interface configuration file in the /etc/sysconfig/network-scripts/ directory. This is because they were originally installed at a time when NetworkManager—or the earlier network service—created those files automatically during installation. Since NetworkManager continues to recognize these files, there is no pressing need for me to do anything different.

Interface configuration file naming

Fortunately, most of you missed out on some of the fun all us old-time sysadmins used to have when adding, removing, or just moving the NIC hardware in hosts with multiple NICs. It seemed like all of the NICs would get renamed whenever anything changed. That meant I would need to determine which name each NIC was given and modify the interface configuration files to match the correct names.

There are now very consistent NIC naming conventions based on the NIC's logical position on the PCIe or USB data busses. This convention was created around 2009 to eliminate those problems.

How it works–sort of

The udev device manager detects when a new device has been added to the system, such as a new NIC, and creates a rule to identify and name it if one does not already exist. During the early part of the startup phase, the Linux kernel uses udev to identify connected devices, including network interface controllers. At this stage, the devices are still known by their traditional names of ethX. Shortly after that, systemd renames the devices according to a series of hierarchical naming schemes.

I used my firewall system as an example of a system with multiple network connections. You can also do this on your own Linux hosts.

[root@wally ~]# dmesg | grep eth
[    2.081738] r8169 0000:01:00.0 eth0: RTL8168e/8111e, 84:16:f9:03:e9:89, XID 2c2, IRQ 126
[    2.081830] r8169 0000:01:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    2.089218] r8169 0000:02:00.0 eth1: RTL8168e/8111e, 84:16:f9:03:fd:85, XID 2c2, IRQ 127
[    2.089303] r8169 0000:02:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    2.094383] r8169 0000:04:00.0 eth2: RTL8168e/8111e, 84:16:f9:04:44:03, XID 2c2, IRQ 128
[    2.094467] r8169 0000:04:00.0 eth2: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    2.142068] r8169 0000:01:00.0 enp1s0: renamed from eth0
[    2.152128] r8169 0000:04:00.0 enp4s0: renamed from eth2
[    2.161346] r8169 0000:02:00.0 enp2s0: renamed from eth1

[root@wally ~]#

This example shows that a little over two seconds into the Linux startup sequence, the ethX network devices were located, and less than a second later, they were renamed enpXs0.

All current releases of RHEL, CentOS, and Fedora use the newest NIC naming conventions. Most other distros also use this naming convention.

The NIC naming convention for these distributions is described in detail in the RHEL 7 document, "Networking Guide," with an explanation of how the names are derived. Using the NetworkManager tools to manage networking is covered in the RHEL 8 document, "Configuring and Managing Networking."

The following is an excerpt from Chapter 11 of the RHEL 7 "Networking Guide":

• Scheme 1: Names incorporating Firmware or BIOS provided index numbers for on-board devices (example: eno1), are applied if that information from the firmware or BIOS is applicable and available, else falling back to scheme 2.
• Scheme 2: Names incorporating Firmware or BIOS provided PCI Express hotplug slot index numbers (example: ens1) are applied if that information from the firmware or BIOS is applicable and available, else falling back to scheme 3.
• Scheme 3: Names incorporating physical location of the connector of the hardware (example: enp2s0), are applied if applicable, else falling directly back to scheme 5 in all other cases.
• Scheme 4: Names incorporating interface's MAC address (example: enx78e7d1ea46da), is not used by default, but is available if the user chooses.
• Scheme 5: The traditional unpredictable kernel naming scheme, is used if all other methods fail (example: eth0).

The primary function of the revised naming schemes is to provide a consistent set of NIC names so that installing a new NIC or even just a reboot would not cause the NIC names to change. This by itself is well worth the changes. I have had plenty of opportunities to fight with the apparently random renaming of multiple ethX devices on a single host. That was much less fun than learning the revised naming schemes.

Understanding interface configuration files

These interface configuration files are easy to create and modify. The following configuration files on my firewall/router host are all located in the /etc/sysconfig/network-scripts directory. This directory previously contained all of the scripts used to manage the network connections, but NetworkManager has made them obsolete. Only the deprecated interface configuration files might remain in this directory.

-rw-r--r-- 1 root root 381 Jan 11
2021 ifcfg-enp1s0
-rw-r--r-- 1 root root 507 Jul 27
2020 ifcfg-enp2s0
-rw-r--r-- 1 root root 924 Mar 31 14:24 ifcfg-enp4s0

This is the configuration file for the interface that connects the firewall host to my home network, as you can see from the comment:

[root@wally network-scripts]# cat ifcfg-enp2s0
# Interface configuration file for enp2s0 / 192.168.0.254
# This interface is for the internal network
# Correct as of 20220711
HWADDR=84:16:f9:03:fd:85
NAME="enp2s0"
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.0.254
PREFIX=24
DNS1=192.168.0.52
DNS2=8.8.8.8
DEFROUTE=no
PEERDNS=yes
PEERROUTES=no
IPV4_FAILURE_FATAL=no

This file is for a fairly simple static configuration that provides the IP address, the CIDR prefix, and two DNS server IP addresses. No default route (gateway) IP address is specified as that is configured in one of the other interface configuration files.

The code block below shows the interface configuration file for the connection from my Linux router to the ISP's router. It uses one of the static IP addresses provided to me by my ISP.

##############################################################################
# Interface configuration file for enp4s0 / 45.20.209.41
# This NIC was installed to circumvent problems with motherboard NIC, eno1.
------------------------------------------------------------------------------
# This interface is for the WAN - the AT&T fiber optic external network
# Correct as of 20220711
##############################################################################
TYPE= "Ethernet"
BOOTPROTO="static"
NM_CONTROLLED= "yes"
DEFROUTE= "yes"
NAME=enp4s0
UUID="fa2117dd-6c7a-44e0-9c9d-9c662716a352"
ONBOOT= "yes"
HWADDR=84:16:f9:04:44:03
IPADDR=45.20.209.41
PREFIX=29
GATEWAY=45.20.209.46
DNS1=192.168.0.52
DNS2=8.8.8.8
DNS3=8.8.4.4
PEERDNS=no
IPv6INIT=no
IPv6_AUTOCONF=no
IPv6_DEFROUTE=no
         

Since I don't use IPv6 and have disabled it, I could delete the IPv6 statement in both files.

Network configuration variables

The table below lists the most common network configuration variables, along with some brief explanations for each. Many IPv6 options are functionally equivalent to the similarly named IPv4 ones. Local configuration variable settings override those provided by a DHCP server. You can use DHCP for configuring a host but use an interface configuration file to override one or more of the DHCP configuration variables.

Some of the more common configuration variables found in the old-style network interface configuration files. Configuration variable Description TYPE Type of network such as Ethernet or token ring. PROXY_METHOD Proxy configuration method. "none" means no proxy is in use. BROWSER_ONLY Whether a proxy configuration is for browsers only. BOOTPROTO Options are dhcp, bootp, none, and static. The "none" option implies DHCP. DEFROUTE This interface is the default route for this host to the outside world. IPv4_FAILURE_FATAL If this is set to "no" failure to obtain an IPv4 connection will not affect any attempt to make an IPv6 connection. NAME The interface name, such as enp0s3. This should match the interface name in the interface configuration file name. UUID A Universally Unique Identifier for the interface. It is created with a hash of the interface name. The HWADDR is an older means of bonding the file to the hardware interface, and I have found that the UUID can be commented out or deleted without issues. DEVICE The name of the interface to which this configuration file is bound. ONBOOT If yes, this starts the interface at boot (really startup time). If set to no, the interface is not started until a user logs in at the GUI or manually starts the interface. HWADDR The MAC address of the interface. This is one of the more important fields in the file as it is used to bond the file to the correct hardware interface. The UUID is a more recent addition and can also be used, but the HWADDR was first and is more widely used. DNS1, DNS2 Up to two name servers may be specified. USERCTL Specifies whether non-privileged users may start and stop this interface. Options are yes/no. IPADDR The IP Address assigned to this NIC. BROADCAST The broadcast address for this network such as 10.0.2.255. NETMASK The netmask for this subnet such as 255.255.255.0. Use either NETMASK or PREFIX but not both. PREFIX The CIDR prefix for the network such as 24. Use either NETMASK or PREFIX but not both. NETWORK The network ID for this subnet such as 10.0.2.0. SEARCH The DNS domain name to search when doing lookups on unqualified hostnames such as using studentvm1 instead of studentvm1.example.com. GATEWAY The network router or default gateway for this subnet, such as 10.0.2.1. PEERDNS The yes value indicates that /etc/resolv.conf is to be modified by inserting the DNS server entries specified by DNS1 and DNS2 options in this file. No means do not alter the resolv.conf file. Yes is the default when DHCP is specified in the BOOTPROTO line. IPv6INIT Whether to initialize IPv6 or not. The default is yes. IPv6_AUTOCONF Yes means use DHCP for configuration of IPv6 on this interface. IPv6_DEFROUTE This interface is the IPv6 default route for this host to the outside world. IPv6_FAILURE_FATAL If this is set to "no" failure to obtain an IPv6 connection will not affect any attempt to make an IPv4 connection. IPv6_ADDR_GEN_MODE Configure IPv6 Stable Privacy addressing.

There are many more configuration variables than are listed here, but these are the ones that are most frequently used.

Final thoughts

There are still plenty of Linux hosts around that use the interface configuration files described in this article. Despite being deprecated, NetworkManager still recognizes these files and can use them to configure network interfaces. However, most modern Linux systems use NetworkManager, so no configuration files are needed unless they serve a special use case, like a server or a router.

I have a few hosts that require more than just the standard NetworkManager configuration. For me, it has just not been a priority to change from the old interface configuration files to the current connection configuration files used by NetworkManager. To prevent future problems with my network, I need to switch to the NetworkManager network connection files. The next article in this series will describe how I make that switch.

Simplify the complex world of interface configuration files with this handy tutorial.

Image by:

Internet Archive Book Images. Modified by Opensource.com. CC BY-SA 4.0

Linux Networking Sysadmin What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How I use the Linux sed command to automate file edits Jim Hall Tue, 08/02/2022 - 03:00 1 reader likes this 1 reader likes this

When I use the Linux command line, whether I'm writing a new program on my desktop computer or managing a website on my web server, I often need to process text files. Linux provides powerful tools that I leverage to get my work done. I frequently use sed, an editor that can modify text according to a pattern.

sed stands for stream editor, and it edits text in a file and prints the results. One way to use sed is to identify several occurrences of one string in a file and replace them with a different string. You can use sed to process text files to a seemingly endless degree, but I'd like to share a few ways I use sed to help me manage files.

Search and replace text in a file on Linux

To use sed, you need to use a regular expression. A regular expression is a set of special characters that define a pattern. My most frequent example of using sed is replacing text in a file. The syntax for replacing text looks like this: s/originaltext/newtext/. The s tells sed to perform text replacement or swap occurrences of text. Provide the original text and new text between slashes.

This syntax will only replace the first occurrence of originaltext on each line. To replace every occurrence, even if the original text appears more than once on a line, append g to the end of the expression. Here is an example: s/originaltext/newtext/g.

To use this with sed, specify this regular expression with the -e option:

$ sed -e 's/originaltext/newtext/g'

For example, let's say I have a Makefile for a program called game, which simulates Conway's Game of Life:

.PHONY: all run clean

all: game

game: game.o
        $(CC) $(CFLAGS) -o game game.o $(LDFLAGS)

run: game
        ./game

clean:
        $(RM) *~
        $(RM) *.o
        $(RM) game

The name game isn't very descriptive, so I might choose to rename it life. Renaming the game.c source file to life.c is easy enough, but now I need to modify the Makefile to use the new name. I can use sed to change every occurrence of game to life:

$ sed -e 's/game/life/g' Makefile
.PHONY: all run clean

all: life

life: life.o
        $(CC) $(CFLAGS) -o life life.o $(LDFLAGS)

run: life
        ./life

clean:
        $(RM) *~
        $(RM) *.o
        $(RM) life

This prints the sed output to the screen, which is a good way to check if the text replacement will do what you want. To make these changes to the Makefile, first, make a backup of the file, then run sed and save the output to the original filename:

$ cp Makefile Makefile.old
$ sed -e 's/game/life/g' Makefile.old > Makefile

If you are confident that your changes are exactly what you want, use the -i or --in-place option to edit the file in place. However, I recommend adding a backup filename suffix like --in-place=.old to save a copy of the original file in case you need to restore it later. It looks like this:

$ sed --in-place=.old -e 's/game/life/g' Makefile
$ ls Makefile*
Makefile  Makefile.old

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Quoting files with sed on Linux

You can use other features of regular expressions to match specific instances of text. For example, you might need to replace text that occurs at the start of a line. With sed, you can match the beginning of a line with ^, the caret character.

One way I use "start of line" in replacing text is when I need to quote a file in an email. Let's say I want to share my Makefile in an email, but I don't want to include it as a file attachment. Instead, I prefer to "quote" the file in the body of an email, using > before each line. I can use the following sed command to print out an edited version to my terminal, which I can copy and paste into a new email:

$ sed -e 's/^/>/' Makefile
>.PHONY: all run clean
>
>all: life
>
>life: life.o
>       $(CC) $(CFLAGS) -o life life.o $(LDFLAGS)
>
>run: life
>       ./life
>
>clean:
>       $(RM) *~
>       $(RM) *.o
>       $(RM) life

The s/^/>/ regular expression matches the start of each line (^) and places a > there. Effectively, this starts each line with the > symbol.

The tabs might not show up correctly in an email, but I can replace all tabs in the Makefile with a few spaces by adding another regular expression:

$ sed -e 's/^/>/' -e 's/\t/  /g' Makefile
>.PHONY: all run clean
>
>all: life
>
>life: life.o
>  $(CC) $(CFLAGS) -o life life.o $(LDFLAGS)
>
>run: life
>  ./life
>
>clean:
>  $(RM) *~
>  $(RM) *.o
>  $(RM) life

The \t indicates a literal tab, so s/\t/ /g tells sed to replace all tabs in the input with two spaces in the output.

If you need to apply lots of edits to files, you can save your -e commands in a file and use -f to tell sed to use that file as a "script." This approach is especially useful if you need to make the same edits frequently. I might have prepared the Makefile for quoting in email using a script file called quotemail.sed:

$ cat quotemail.sed
s/^/>/
s/\t/  /g
$ sed -f quotemail.sed Makefile
>.PHONY: all run clean
>
>all: life
>
>life: life.o
>  $(CC) $(CFLAGS) -o life life.o $(LDFLAGS)
>
>run: life
>  ./life
>
>clean:
>  $(RM) *~
>  $(RM) *.o
>  $(RM) lifeLearn to work with sed on Linux

sed is a great tool to keep in your Linux command-line toolkit. Explore the sed manual page and learn more about how to use it. Type man sed at the command line to get complete documentation about the different command line options and how to use sed to process text files.

Here are some tips and tricks to automating file edits from the Linux command line.

Image by:

Opensource.com

Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How I disabled IPv6 on Linux David Both Mon, 08/01/2022 - 03:00 Register or Login to like Register or Login to like

IPv6 is a good thing for the Internet in general, but I find it unnecessarily complex for use in most home and small- to medium-size businesses. Like many others, I continue to use private IPv4 address ranges for my own internal networks and those for which I have some level of responsibility. My ISP only provides IPv4 addresses anyway, so it makes no sense to use IPv6 internally when all external packets are IPv4. Besides, IPv4 is much simpler, and one of my Linux Philosophy tenets is "Find the Simplicity."

As a result, I disabled IPv6 on all my hosts. It seemed easy—at first. Here is how I did it.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Testing for IPv6

My hosts run the Fedora 36 Xfce spin along with many packages and configuration changes I perform after the initial default installation. All of my hosts have the most recent updates installed. One of those hosts is my firewall/router, which is the first host on which I disabled IPv6.

You can check to see whether IPv6 is currently active on your Linux host. The nmcli command results below are from my router/firewall host. All the active NICs have both IPv4 and IPv6 addresses.

# nmcli
enp4s0: connected to enp4s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:04:44:03, hw, mtu 1500
        ip4 default
        inet4 45.20.209.41/29
        route4 45.20.209.40/29 metric 102
        route4 default via 45.20.209.46 metric 102
        inet6 2600:1700:7c0:860:8616:f9ff:fe04:4403/64
        inet6 fe80::8616:f9ff:fe04:4403/64
        route6 fe80::/64 metric 256
        route6 default via fe80::a698:13ff:fee5:fa10 metric 1024
        route6 2600:1700:7c0:860::/64 metric 256

enp1s0: connected to enp1s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:03:E9:89, hw, mtu 1500
        inet4 192.168.10.1/24
        route4 192.168.10.0/24 metric 101
        inet6 fe80::8616:f9ff:fe03:e989/64
        route6 fe80::/64 metric 256

enp2s0: connected to enp2s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:03:FD:85, hw, mtu 1500
        inet4 192.168.0.254/24
        route4 192.168.0.0/24 metric 100
        inet6 fe80::8616:f9ff:fe03:fd85/64
        route6 fe80::/64 metric 256

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 192.168.0.52 8.8.8.8 8.8.4.4
        interface: enp4s0

        servers: 192.168.0.52 8.8.8.8
        interface: enp2s0

        servers: 192.168.0.52 8.8.8.8
        interface: enp1s0Use GRUB to configure the kernel

GRUB–the Grand Unified Bootloader–loads the kernel into memory and sets many kernel parameters during the Linux boot process. The current bootloader is actually GRUB2, the second major version of GRUB, but it is easier just to call it GRUB.

The GRUB configuration file /boot/grub2/grub.cfg provides the startup configuration for the kernel, but you should not modify it directly. The location of the UEFI GRUB configuration file may vary depending on your distribution. However, its location is irrelevant for this purpose. Linux startup kernel configuration can be easily modified using the /etc/default/grub configuration file, which configures the kernel properly regardless of whether it boots using UEFI.

There are two command line arguments to add to the /etc/default/grub file to configure the Linux kernel to disable IPv6. The first, GRUB_CMDLINE_LINUX="ipv6.disable=1" adds "ipv6.disable=1" to the kernel command line.

The second, GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 quiet splash" is a little more complicated. When you set GRUB_DISABLE_RECOVERY to true, one menu entry will be generated for each Linux kernel. When set to false, two menu entries are created for each kernel: One default entry and one entry for recovery mode. This option lists command-line arguments to be added only to the default menu entry, after those listed in GRUB_CMDLINE_LINUX.

The other two arguments define how the kernel startup looks on the display. The "quiet" argument disables most of the informational messages emitted by the kernel during its startup and self-configuration. The "splash" argument causes the splash screen to be displayed during kernel startup. The splash screen can be a graphic such as a logo or an animation.

My /etc/default/grub file looks like this after making those additions:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 quiet splash"
GRUB_CMDLINE_LINUX="ipv6.disable=1"

After making these (or any) additions to the /etc/default/grub file, run the grub2-mkconfig command, which adds these arguments to the kernel command line in the grub.cfg configuration file for each installed kernel. These changes do not take effect immediately. You must reboot the hosts for IPv6 to be disabled.

Automate this solution

Automating these tasks using Ansible is straightforward. Just create a playbook with the following settings and run it against the list of hosts that need this change. The first two tasks in this playbook append the desired configuration:

- name: Play 1 - Modify /etc/default/grub to configure kernel boot parameters.
  hosts: f36vm

  tasks:
    - name: Append GRUB_CMDLINE_LINUX_DEFAULT variable to /etc/default/grub
      lineinfile:
        path: /etc/default/grub
        insertafter: EOF
        line: 'GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 quiet splash"'

    - name: Append GRUB_CMDLINE_LINUX variable to /etc/default/grub
      lineinfile:
        path: /etc/default/grub
        insertafter: EOF
        line: 'GRUB_CMDLINE_LINUX="ipv6.disable=1"'

    - name: Run grub2-mkconfig to update /boot/grub/grub.cfg
      command:
        cmd: grub2-mkconfig

Once again, you must reboot the hosts for IPv6 to be disabled. You can add the reboot to the playbook so that the changes take effect immediately, or wait until the next time the hosts need to be rebooted for other reasons.

The other solution

I created another solution; it is just not the best and most elegant option. However, I want to show my thought process because there is some information here that might be useful.

Add a local file to sysctl.d

You can use the /etc/sysctl file to add the configuration settings necessary, but it is much better to add a local file to the /etc/sysctl.d directory so that it won't be overwritten during updates or upgrades. Note that there is already a file named 99-sysctl.conf. You can use that file to set the configuration, but I created a file named 99-local-network.conf with the following content for this purpose. That way, if the 99-sysctl.conf changes with future updates or upgrades, my file will remain untouched. This is not an executable file; it is a configuration file.

################################################################################
# Local NetworkManager settings - Specifically to disable IPV6                 #
################################################################################
#
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Note: Like many configuration files in configuration directories like this one, the files contained in the directory are read in natural sorted order. That means the value for a variable declared in a file with a later sort order will override an earlier declaration for the same variable.

A reboot is usually used to cause these changes to take effect, but I later learned how to do so without a reboot. I rebooted my system and reran the nmcli command with the following results:

[root@wally ~]# nmcli
enp4s0: connected to enp4s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:04:44:03, hw, mtu 1500
        ip4 default
        inet4 45.20.209.41/29
        route4 45.20.209.40/29 metric 101
        route4 default via 45.20.209.46 metric 101

enp1s0: connected to enp1s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:03:E9:89, hw, mtu 1500
        inet4 192.168.10.1/24
        route4 192.168.10.0/24 metric 102

enp2s0: connected to enp2s0
        "Realtek RTL8111/8168/8411"
        ethernet (r8169), 84:16:F9:03:FD:85, hw, mtu 1500
        inet4 192.168.0.254/24
        route4 192.168.0.0/24 metric 100

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 192.168.0.52 8.8.8.8 8.8.4.4
        interface: enp4s0

        servers: 192.168.0.52 8.8.8.8
        interface: enp2s0

        servers: 192.168.0.52 8.8.8.8
        interface: enp1s0

This shows that my simple fix worked.

Except...

Except that solution only works on one of my twelve Linux computers. After writing the above part of this article, I started installing this fix on all of my other hosts. I had only done one system when I realized that this approach did not always work. I then tried it on one of my VMs and it also failed there. As best I can tell, it only works on one host—the one I use as my firewall and router.

After some additional research on a VM, I discovered that these settings could also be issued as commands using sysctl so that the system would not need a reboot. I could enter those commands from the command line to deactivate IPv6. Here's an example:

[root@f36vm ~]# sysctl -w net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.all.disable_ipv6 = 1
[root@f36vm ~]# sysctl -w net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6 = 1
[root@f36vm ~]# nmcli
enp0s3: connected to Wired connection 1
        "Intel 82540EM"
        ethernet (e1000), 08:00:27:07:CD:FE, hw, mtu 1500
        ip4 default
        inet4 192.168.0.136/24
        route4 192.168.0.0/24 metric 100
        route4 default via 192.168.0.254 metric 100

lo: unmanaged
<SNIP>

It took some time to research this and determine that the file I created was not being read–or at least not processed–by sysctl during the Linux startup. Or, if it was, it was being ignored or the settings were being overwritten later by the same variables with different values. After this, I knew that there was no deeper problem preventing it.

About sysctl

At this point, I looked into the sysctl command in more detail. The purpose of the sysctl command is to set kernel parameters in the /proc directory. The root user can use it to set individual parameters from the command line. In addition–and this is the key to my solution–you can use it to view and set all of the kernel parameters stored in several locations, including /etc/sysctl.conf and in files located in /etc/sysctl.d. The sysctl command is used during Linux startup to set the kernel parameters.

I repeat: The system and the sysadmin can use the sysctl command to view and set kernel variables while the system is up and running and without a reboot. This command and the power it offers the Linux sysadmin is one of the most significant differences between Linux and other operating systems. It gives us a tool to do things impossible on other OSs.

I tested this by rebooting the VM and running the following command to set the variables in all locations listed on the sysctl man page:

[root@f36vm ~]# sysctl --system
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-coredump.conf ...
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
kernel.core_pipe_limit = 16
fs.suid_dumpable = 2
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 2
sysctl: setting key "net.ipv4.conf.all.rp_filter": Invalid argument
net.ipv4.conf.default.accept_source_route = 0
sysctl: setting key "net.ipv4.conf.all.accept_source_route": Invalid argument
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ...
net.core.optmem_max = 81920
* Applying /usr/lib/sysctl.d/50-libreswan.conf ...
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-local-network.conf ...
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...

[root@f36vm ~]#

Checking the status of the NIC with the nmcli command showed that IPv6 had been disabled and that IPv4 was still up and running. Yes, I do see the other errors in that data stream, but I am ignoring them for now. Be sure to read the man page for the sysctl command as it is quite interesting. It provides a method for processing all sysctl configuration files with the --service option.

The solution

Now that I understood the true nature of the problem, I could create a real solution–even if it might be only a temporary circumvention. I was able to do so in a way that is in keeping with the original intent of the Linux startup sequence and the sysctl.d method for configuring the kernel.

I left my new configuration file in place in /etc/sysctl.d. I created the simple Bash script shown below to run the sysctl --system command and stored it in /usr/local/bin.

#!/bin/bash
<SNIP – discarded a bunch of comments to save space>
sysctl --system

I tested this script multiple times before proceeding to ensure that it worked. If you do this, be sure to test it multiple times, both with and without rebooting, to return to the original kernel configuration.

Create the service

The real key to this solution was to create a new systemd service that would work similarly to the old rc.local SystemV script. In this case, I called it the MyStartup.service, and I renamed the script I created above and called it MyStartup.sh. To create the service itself, I created a new systemd unit file in the /usr/local/lib/systemd/system directory, which I also had to make. This service will run once at startup. I named this new file MyStartup.service and added the following content:

[Unit]
Description=Runs /usr/local/bin/MyStartup.sh

[Service]
ExecStart=/usr/local/bin/MyStartup.sh

[Install]
WantedBy=multi-user.target

Note that systemd unit files like this one don't need to be executable. It is owned by root.root with 664 permissions. It's pretty simple to create this service, and I can use it for so many local startup tasks that I plan to keep it even when there is a permanent fix to the extant problem.

The command below enables the new service. Note that the systemctl command searches the new directory by default without any options, arguments, or prodding from me to locate the new unit file.

[root@f36vm ~]# systemctl enable MyStartup.service
Created symlink /etc/systemd/system/multi-user.target.wants/MyStartup.service → /usr/local/lib/systemd/system/MyStartup.service.

Enabling the service does not run the MyStartup.sh script. This service will run the script at every reboot.

Testing

As soon as the VM rebooted, I logged in as root and ran the following command to check the status of IPv6, but it was still active. After more testing, I determined that the service was running the commands too soon, so I added a command to the MyStartup.sh script to sleep for 25 seconds before running the commands. Here is the edited script:

#!/bin/bash

# Wait a bit for things to start up and settle. It doesn't work without this.
sleep 25
# Run the sysctl command.
sysctl --system

I rebooted again and verified the status as soon as I could log in, with the result that the service was still sleeping.

[root@f36vm ~]# systemctl status MyStartup.service
● MyStartup.service - Runs /usr/local/bin/MyStartup.sh
     Loaded: loaded (/usr/local/lib/systemd/system/MyStartup.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2022-07-01 13:24:22 EDT; 14s ago
   Main PID: 667 (MyStartup.sh)
      Tasks: 2 (limit: 14129)
     Memory: 592.0K
        CPU: 1ms
     CGroup: /system.slice/MyStartup.service
             ├─ 667 /bin/bash /usr/local/bin/MyStartup.sh
             └─ 669 sleep 25

After waiting some additional time, I checked again, and the service had completed successfully, with the results clearly shown in the last few journal entries. Further testing showed that a delay of three seconds works reliably, while a delay of only one second always fails.

Jul 01 13:24:22 f36vm.both.org systemd[1]: Started MyStartup.service - Runs /usr/local/bin/MyStartup.sh.

[root@f36vm ~]# systemctl status MyStartup.service
○ MyStartup.service - Runs /usr/local/bin/MyStartup.sh
     Loaded: loaded (/usr/local/lib/systemd/system/MyStartup.service; enabled; vendor preset: disabled)
     Active: inactive (dead) since Fri 2022-07-01 13:24:47 EDT; 358ms ago
    Process: 667 ExecStart=/usr/local/bin/MyStartup.sh (code=exited, status=0/SUCCESS)
   Main PID: 667 (code=exited, status=0/SUCCESS)
        CPU: 9ms

Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: net.ipv4.conf.all.send_redirects = 0
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: net.ipv4.conf.all.accept_redirects = 0
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: * Applying /usr/lib/sysctl.d/50-pid-max.conf ...
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: kernel.pid_max = 4194304
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: * Applying /etc/sysctl.d/99-local-network.conf ...
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: net.ipv6.conf.all.disable_ipv6 = 1
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: net.ipv6.conf.default.disable_ipv6 = 1
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: * Applying /etc/sysctl.d/99-sysctl.conf ...
Jul 01 13:24:47 f36vm.both.org MyStartup.sh[1020]: * Applying /etc/sysctl.conf ...
Jul 01 13:24:47 f36vm.both.org systemd[1]: MyStartup.service: Deactivated successfully.

[root@f36vm ~]#

You can see in the data output above that the configuration statements in the 99-sysctl.conf file were applied. I also used the nmcli command to verify that IPv6 has been disabled.

Automate it

After figuring out how to consistently accomplish this solution, I added a set of tasks to do this to the Ansible playbook I use to distribute new and updated configuration files. That makes it easy for me to manage my 12 current hosts. I also added it to the playbook I use immediately after performing a basic installation on new hosts or ones that need a reinstallation for some reason. I added the following tasks to those playbooks.

This first set of tasks is for my solution to add a new service:

   - name: Install 99-local-network.conf file to disable IPV6
      copy:
        src: /root/ansible/system-scripts/files/99-local-network.conf
        dest: /etc/sysctl.d
        mode: 0644
        owner: root
        group: root

    - name: Install MyStartup.sh
      copy:
        src: /root/ansible/system-scripts/files/MyStartup.sh
        dest: /usr/local/bin
        mode: 0754
        owner: root
        group: root

    - name: create /root/ansible/system-scripts/files/ directory
      file:
        path: /root/ansible/system-scripts/files/
        state: directory
        mode: 0755
        owner: root
        group: root  
     
    - name: Install MyStartup.service
      copy:
        src: /root/ansible/system-scripts/files/MyStartup.service
        dest: /usr/local/lib/systemd/system/
        mode: 0664
        owner: root
        group: root

    - name: Enable the MyStartup.service
      systemd:
        name: MyStartup.service
        state: stopped
        enabled: yes

    - name: Run the raw command to disable IPV4 so a reboot is not required at this time
      command:
        cmd: sysctl --system

The advantage of this more complex solution of adding the configuration file to the sysctl.d directory and then running a playbook with this series of tasks is it disables IPv6 without requiring a reboot.

More for sysadmins Enable Sysadmin blog The Automated Enterprise: A guide to managing IT with automation eBook: Ansible automation for Sysadmins Tales from the field: A system administrator's guide to IT automation eBook: A guide to Kubernetes for SREs and sysadmins Latest sysadmin articles Final thoughts

Although just installing the file with the correct kernel parameters and rebooting worked fine on one of my hosts, it failed on all of the others I tried. I looked for differences that might explain why it failed on the others while working on my router/firewall but found nothing that provided a clue as to why this is so. I discovered this problem exists on hosts with newly installed Fedora 36 without any of the changes and post-installation configuration that I always perform and on those with my usual changes. I do plan to keep investigating. I intend to submit a bug report to Red Hat so that there might be an actual fix to this instead of either of these circumventions.

Neither circumvention depends upon the existence of any NIC configuration files—either the old-style interface configuration files that used to be located in /etc/sysconfig/network-scripts or the newer NetworkManager interface connection files located in the /etc/NetworkManager/system-connections directory. It works with or without those files. The result is global for all network interfaces on the host.

My own solution is a somewhat general approach that you can use in lieu of the old rc.local script of the old SystemV startup days.

Resources

The following resources are kept as current as possible but may not be completely accurate at any given time.

• The GRUB manual is available in multiple formats from the GNU website.
• The Fedora documentation website has an excellent page, Bootloading with GRUB2.
• The Red Hat RHEL 7 online documentation also has a good chapter about working with the GRUB2 boot loader.
• The kernel.org website has a long list of kernel parameters and a huge amount of other information about the Linux kernel.

Simplify your home network by disabling IPv6.

Sysadmin Networking Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Why program management matters in open source Ben Cotton Mon, 08/01/2022 - 03:00 Register or Login to like Register or Login to like

Everyone does program management. Some just do it poorly.

I've used that line for a laugh at the beginning of talks, but it's true. Program management is, at its core, the act of coordinating the interfaces between teams to produce something of value. In open source projects, the "something of value" is generally the software that the community produces. Most open source communities create software, and almost none of them have a formal program manager. So why have a program manager?

The difference lies in managing the software development with intent instead of by accident. The smaller the community, the easier it is to self-coordinate. The need for intentional coordination increases as the community grows or the software becomes more complex.

In The Mythical Man Month, Fred Brooks noted that the number of communication channels goes up dramatically faster than the number of people working on a project. A program manager can help simplify the communication overhead by serving as a centralized channel for information. By lurking on mailing lists and chat channels, the program manager sees what's going on in the project and communicates that broadly to the community and the public. This way, anyone who needs to know the high-level details can look at the program manager's summary instead of paying attention to every channel themselves.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources

A good program manager also acts as a guide in process development and improvement. Think of processes like trails. They develop organically over time as people find the best route from one place to another. A well-worn trail makes for a faster, more predictable trip. But if everyone has their own variations on the route, the trail won't be well-worn. The program manager's job isn't to blaze a trail and say, "This is the way everyone must go!" Instead, the program manager looks for the places where the individual paths are similar and helps the community find an optimal path to share. So a program manager doesn't say, "This is the process we must follow to develop our software." That wouldn't work well in an open source project. Instead, the program manager helps define a repeatable and predictable process based on what the community is already doing.

Many activities touch program management, even if they're handled mainly by other functions in the community. Examples include schedule development, feature planning, bug triage, release decisions, and more. With such a wide range of skills, you might be hard-pressed to find one person who can be great at all of those. And even if you found such a person, they might not have enough time to devote to doing everything. It's okay to split the work among several contributors so long as everyone clearly understands their responsibilities.

Ultimately, the program manager's job is to help the community avoid surprises. They do this by communicating status across teams and ensuring everyone knows what is needed—and when—to produce another great release. By necessity, you're doing this already, so you might as well do it on purpose.

This article is based on the author's book, Program Management for Open Source Projects.

As your open source community grows or your technology becomes more complex, a skilled program manager will coordinate teams and people with intention.

Image by:

Opensource.com

Community management What to read next 9 underrated responsibilities of an open source community manager This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Learn Rust by debugging Rust Gaurav Kamathe Fri, 07/29/2022 - 03:00 Register or Login to like Register or Login to like

In my previous article about rustup, I showed you how to install the Rust toolchain. Well, what good is the toolchain if you won’t be using it to get more hands-on with Rust? Learning any language involves reading existing code and writing a lot of sample programs. That's a good way to become proficient in a language. However, there's a third way: debugging code.

Learning through debugging involves trying to compile a pre-written (buggy) sample program, understanding the errors generated by the compiler, fixing the sample code, and then re-compiling it. Repeat that process until the code successfully compiles and runs.

Rustlings is an open source project by the Rust team that helps you learn Rust through the process of debugging. It also provides you with a lot of hints along the way. If you're a beginner to Rust and have either started or completed reading the Rust book, then rustlings is the ideal next step. Rustlings helps you apply what you've learned from the book, and move to working on bigger projects.

Installing rustlings

I'm using (and recommend) a Fedora machine to try rustlings, but any Linux distribution works. To install rustlings, you must download and run its install script. It's recommended that you do this as a normal user (not root) without any special privileges.

Remember, for you to be able to use rustlings, you need the Rust toolchain available on your system. If you don't have that already, please refer to my article on rustup.

Once you're ready, download the installation script:

$ curl -L https://raw.githubusercontent.com/rust-lang/rustlings/main/install.sh  > rustlings_install.sh
$ file rustlings_install.sh
rustlings_install.sh: Bourne-Again shell script, ASCII text executable

Inspect the script to learn what it does:

$ less rustlings_install.sh

And then run it to install:

$ bash rustlings_install.sh
[...]
Installing /home/tux/.cargo/bin/rustlings
Installed package `rustlings v4.8.0 (/home/tux/rustlings)` (executable `rustlings`)
All done!

Run 'rustlings' to get started.

Rustlings exercises

The installation provides you with the rustlings command. Run it along with the --help flag to see what options are available.

$ rustlings --help

The installation script also clones the rustlings Git repository, and installs all the dependencies required to run the sample programs. You can view the sample programs within the exercises directory under rustlings:

$ cd rustlings
$ pwd
/home/tux/rustlings
$ ls
AUTHORS.md  Cargo.toml        CONTRIBUTING.md  info.toml install.sh README.md  target Cargo.lock  CHANGELOG.md  exercises install.ps1  LICENSE src tests
$ ls -m exercises/
advanced_errors, clippy, collections, conversions, enums, error_handling, functions, generics, if, intro, macros, mod.rs,
modules, move_semantics, option, primitive_types, quiz1.rs, quiz2.rs, quiz3.rs, quiz4.rs, README.md,
standard_library_types, strings, structs, tests, threads, traits, variablesList all exercises from the command line

The rustlings command provides you with a list command which displays each sample program, its complete path, and the status (which defaults to pending.)

$ rustlings list
Name         Path                                 Status
intro1       exercises/intro/intro1.rs            Pending
intro2       exercises/intro/intro2.rs            Pending
variables1   exercises/variables/variables1.rs    Pending
variables2   exercises/variables/variables2.rs    Pending
variables3   exercises/variables/variables3.rs    Pending
[...]

Near the end of the output, you're given a progress report so you can track your work.

Progress: You completed 0 / 84 exercises (0.00 %).View sample programs

The rustings list command shows you what programs are available, so at any time you can view the code of those programs by copying the complete paths into your terminal as an argument for the cat or less commands:

$ cat exercises/intro/intro1.rsVerify your programs

Now you can start debugging programs. You can do that using the verify command. Notice that rustlings chooses the first program in the list (intro1.rs), tries to compile it, and succeeds:

$ rustlings verify
Progress: [-----------------------------------] 0/84
✅ Successfully ran exercises/intro/intro1.rs!

You can keep working on this exercise,
or jump into the next one by removing the `I AM NOT DONE` comment:

 6 |  // Execute the command `rustlings hint intro1` for a hint.
 7 |  
 8 |  // I AM NOT DONE
 9 |

As you can see from the output, even though the sample code compiles, there's work yet to be done. Each sample program has the following comment in its source file:

$ grep "NOT DONE" exercises/intro/intro1.rs
// I AM NOT DONE

Although the compilation of the first program worked fine, rustlings won't move to the next program until you remove the I AM NOT DONE comment.

Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java Moving to the next exercise

Once you have removed the comment from intro1.rs, you can move to the next exercise by running the rustlings verify command again. This time, you may notice that rustlings tries to compile the next program (intro2.rs) in the series, but runs into an error. You're expected to debug that issue, fix it, and then move forward. This is a critical step, allowing you to understand why Rust says a program has bugs.

$ rustlings verify
Progress: [>------------------------] 1/84
⚠️  Compiling of exercises/intro/intro2.rs failed! Please try again. Here's the output:
error: 1 positional argument in format string, but no arguments were given
 --> exercises/intro/intro2.rs:8:21
  |
8 |         println!("Hello {}!");
  |                         ^^

error: aborting due to previous errorGetting a hint

Rustlings has a handy hint argument, which tells you exactly what's wrong with the sample program, and how to fix it. You can think of it as an add-on help option, in addition to what the compiler error message tells you.

$ rustlings hint intro2
Add an argument after the format string.

Based on the above hint, fixing the program is easy. All you need to do is add an additional argument to the println statement. This diff should help you understand the changes:

< println!("Hello {}!", "world");
---
> println!("Hello {}!");

Once you make that change, and removed the NOT DONE comment from the source code, you can run rustlings verify again to compile and run the code.

$ rustlings verify
Progress: [>-------------------------------------] 1/84
✅ Successfully ran exercises/intro/intro2.rs!Track progress

You aren't going to finish all of the exercises in a day, and it's easy to lose track of where you left off. You can run the list command to see the status of your work.

$ rustlings list
Name         Path                                  Status
intro1       exercises/intro/intro1.rs             Done  
intro2       exercises/intro/intro2.rs             Done  
variables1   exercises/variables/variables1.rs     Pending
variables2   exercises/variables/variables2.rs     Pending
variables3   exercises/variables/variables3.rs     Pending
[...]Run specific exercises

If you don't want to start from the beginning and skip a few exercises, rustlings allows you to focus on specific exercises using the rustlings run command. This runs a specific program without requiring the previous lesson to be verified. For example:

$ rustlings run intro2
Hello world!
✅ Successfully ran exercises/intro/intro2.rs
$ rustlings run variables1

Typing exercise names can become tedious, but rustlings provides you with a handy next command so you can move to the next exercise in the series.

$ rustlings run nextAlternative watch command

If you don't want to keep typing verify after each modification you make, you can run the watch command in a terminal window, and then keep modifying the source code to fix issues. The watch command detects these modifications, and keeps re-compiling the program to see whether the issue has been fixed.

$ rustlings watchLearn by debugging

The Rust compiler is known to provide very meaningful error messages, which helps you understand issues in your code. This usually leads to faster debugging. Rustlings is a great way to practice Rust, to get used to reading error messages, and understand the Rust language. Check out the recent features of Rustlings 5.0.0 on GitHub.

[ Practice programming with Rust. Download our Rust cheat sheet. ]

Rustlings is an open source project by the Rust team that helps you learn Rust through the process of debugging.

Image by:

Opensource.com

Rust Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Fix bugs in Bash scripts by printing a stack trace Evan "Hippy" Slatis Fri, 07/29/2022 - 03:00 Register or Login to like Register or Login to like

No one wants to write bad code, but inevitably bugs will be created. Most modern languages like Java, JavaScript, Python, etc., automatically print a stack trace when they encounter an unhandled exception, but not shell scripts. It would make it much easier to find and fix bugs in shell scripts if you could print a stack trace, and, with a little work, you can.

Shell scripts can span multiple files, and well-written code is further broken down into functions. Tracing issues when something goes wrong in a shell script can be difficult when these scripts get large enough. A stack trace that walks the code backward from the error to the beginning can show you where your code failed and give you a better understanding of why so you can fix it properly.

To implement the stack trace, I use the trap in the following manner at the beginning of my script:

set -E

trap 'ERRO_LINENO=$LINENO' ERR
trap '_failure' EXIT

This example accomplishes a few things, but I'll address the second one, trap 'ERRO_LINENO=$LINENO' ERR, first. This line ensures the script traps all commands that exit with a non-zero exit code (i.e., an error), and saves the line number of the command in the file where the error was signaled. This is not captured on exit.

The first line above (set -E) ensures that the error trap is inherited throughout the script. Without this, whenever you drop into an if or until block, for example, you'd lose track of the correct line number.

The second trap captures the exit signal from the script and sends it to the _failure function, which I'll define in a moment. But why on exit and not error if you're trying to debug the script? In bash scripts, command failures are often used in control logic or can be ignored outright as unimportant by design. For example, say at the beginning of your script, you're looking to see if a particular program is already installed before asking the user whether they'd like you to install it for them:

if [[ ! -z $(command -v some_command) ]]
then
   # CAPTURE LOCATION OF some_command
   SOME_COMMAND_EXEC=$(which some_command)
else
   # echo $? would give us a non-zero value here; i.e. an error code
   # IGNORE ERR: ASK USER IF THEY WANT TO INSTALL some_command
fi

If you were to stop processing on every error and some_command is not installed, this would prematurely end the script, which is obviously not what you want to do here, so in general, you only want to log an error and stack trace when the script has exited unintentionally because of an error.

To force your script to exit whenever there's an unexpected error, use the set -e option:

set -e
# SCRIPT WILL EXIT IF ANY COMMAND RETURNS A NON-ZERO CODE
# WHILE set -e IS IN FORCE
set +e
# COMMANDS WITH ERRORS WILL NOT CAUSE THE SCRIPT TO EXIT HERE

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles

The next question is, what are some examples where you would probably like your script to exit and highlight a failure? Common examples include the following:

1. An unreachable remote system
2. Authentication to a remote system fail
3. Syntax errors in config or script files being sourced
4. Docker image builds
5. Compiler errors

Combing through many pages of logs after a script completes looking for any possible errors that may be hard to spot can be extremely frustrating. It's even more frustrating when you discover something is wrong long past the time you ran the script and now have to comb through multiple sets of logs to find what might have gone wrong and where. Worst is when the error has been around for a while, and you only discover it at the worst possible time. In any case, pinpointing the problem as quickly as possible and fixing it is always the priority.

Look at the sample stack trace code (available for download here):

# Sample code for generating a stack trace on catastrophic failure

set -E

trap 'ERRO_LINENO=$LINENO' ERR
trap '_failure' EXIT

_failure() {
  ERR_CODE=$? # capture last command exit code
  set +xv # turns off debug logging, just in case
  if [[  $- =~ e && ${ERR_CODE} != 0 ]]
  then
      # only log stack trace if requested (set -e)
      # and last command failed
      echo
      echo "========= CATASTROPHIC COMMAND FAIL ========="
      echo
      echo "SCRIPT EXITED ON ERROR CODE: ${ERR_CODE}"
      echo
      LEN=${#BASH_LINENO[@]}
      for (( INDEX=0; INDEX<$LEN-1; INDEX++ ))
      do
          echo '---'
          echo "FILE: $(basename ${BASH_SOURCE[${INDEX}+1]})"
          echo "  FUNCTION: ${FUNCNAME[${INDEX}+1]}"
          if [[ ${INDEX} > 0 ]]
          then
           # commands in stack trace
              echo "  COMMAND: ${FUNCNAME[${INDEX}]}"
              echo "  LINE: ${BASH_LINENO[${INDEX}]}"
          else
              # command that failed
              echo "  COMMAND: ${BASH_COMMAND}"
              echo "  LINE: ${ERRO_LINENO}"
          fi
      done
      echo
      echo "======= END CATASTROPHIC COMMAND FAIL ======="
      echo
  fi
}

# set working directory to this directory for duration of this test
cd "$(dirname ${0})"

echo 'Beginning stacktrace test'

set -e
source ./testfile1.sh
source ./testfile2.sh
set +e

_file1_function1

In the stacktrace.sh above, the first thing the _failure function does is capture the exit code of the last command using the built-in shell value $?. It then checks whether the exit was unexpected by checking the output of $-, a built-in shell value that holds the current bash shell settings, to see if set -e is in force. If the script exited on an error and the error was unexpected, the stack trace is output to the console.

The following built-in shell values are used to build the stack trace:

1. BASH_SOURCE: Array of filenames where each command was called back to the main script.
2. FUNCNAME: Array of line numbers matching each file in BASH_SOURCE.
3. BASH_LINENO: Array of line numbers per file matching BASH_SOURCE.
4. BASH_COMMAND: Last command executed with flags and arguments.

If the script exits with an error in an unexpected manner, it loops over the above variables and outputs each one in order so a stack trace can be built. The line number of the failed command is not held in the above arrays, but that's why you captured the line number each time a command failed with the first trap statement above.

Putting it all together

Create the following two files to support the test, so you can see how the information is gathered across multiple files. First, testfile1.sh:

_file1_function1() {
   echo
   echo "executing in _file1_function1"
   echo

   _file2_function1
}

# adsfadfaf

_file1_function2() {
   echo
   echo "executing in _file1_function2"
   echo
 
   set -e
   curl this_curl_will_fail_and_CAUSE_A_STACK_TRACE

   # function never called
   _file2_does_not_exist
}

And next, testfile2.sh:

_file2_function1() {
   echo
   echo "executing in _file2_function1"
   echo

   curl this_curl_will_simply_fail

   _file1_function2
}

NOTE: If you create these files yourself, make sure to make the stacktrace.sh file executable.

Executing stacktrace.sh will output the following:

~/shell-stack-trace-example$./stracktrace.sh
Beginning stacktrace test

executing in _file1_function1

executing in _file2_function1
curl: (6) Could not resolve host: this_curl_will_simply_fail

executing in _file1_function2
curl: (6) Could not resolve host: this_curl_will_fail_and_CAUSE_A_STACK_TRACE

========= CATASTROPHIC COMMAND FAIL =========

SCRIPT EXITED ON ERROR CODE: 6

---
FILE: testfile1.sh
  FUNCTION: _file1_function2
  COMMAND: curl this_curl_will_fail_and_CAUSE_A_STACK_TRACE
  LINE: 15
---
FILE: testfile2.sh
  FUNCTION: _file2_function1
  COMMAND: _file1_function2
  LINE: 7
---
FILE: testfile1.sh
  FUNCTION: _file1_function1
  COMMAND: _file2_function1
  LINE: 5
---
FILE: stracktrace.sh
  FUNCTION: main
  COMMAND: _file1_function1
  LINE: 53

======= END CATASTROPHIC COMMAND FAIL =======

For extra credit, try uncommenting the line in testfile1.sh and executing stacktrace.sh again:

# adsfadfaf

Then re-comment the line, and instead comment out the following line in testfile1.sh that caused a stack trace and run stacktrace.sh one last time:

curl this_curl_will_fail_and_CAUSE_A_STACK_TRACE

This exercise should give you an idea of the output and when it occurs if you have typos in your scripts.

Automatically printing a stack trace on unhandled errors in your scripts can make finding and fixing bugs in your code much easier.

Image by:

Pixabay, testbytes, CC0

Linux What to read next How to write functions in Bash Using variables in Bash This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Use this nifty Unix tool to process text on Linux Jim Hall Thu, 07/28/2022 - 03:00 1 reader likes this 1 reader likes this

Unix has always excelled at processing text, and Linux is no different. And the tools to work with and transform text files still exist on all Linux systems.

Like other computer systems, early Unix printed on paper, using a typewriter-style printing device. These printers provided limited formatting options, but with clever application of Unix tools, you could prepare professional-looking documents.

One such tool was the pr tool, to prepare text documents for printing. Let's explore how to use standard Unix tools, such as the pr processor and the fmt text formatter, to prepare text files for printing on a typewriter-style printer.

[ Read also: How I use the Linux fmt command to format text ]

Printing a plain text file

Let's say we wanted to print the MIT license, stored in a file called mit.txt. This file is already formatted for optimal screen display; lines are almost 80 columns wide, which fits well on a standard terminal.

$ cat mit.txt Copyright (c) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. Image by:

(Jim Hall, CC BY-SA 40)

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles

Printer paper is also 80 columns wide, at least on the classic printers. So we can also print the file to our printer using a command like lpr. But that's not a very interesting way to print the file, and it won't be very nice to read. For example, the document will start on the first line of the printed page, and immediately on the left edge of the paper.

We can make the document easier to read by using the pr command to add a top margin. By default, pr includes the date and time, the name of the file, and the page number in the top header. For example, the top of our file might look like this:

$ pr mit.txt | head 2022-06-24 18:27 mit.txt Page 1 Copyright (c) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights

In this example, I've used the head command to look at just the first ten lines of the pr output. The pr command also adds extra blank lines at the bottom of the page, to provide a bottom margin. The old-style typewriter printers used 66 lines per page, so the pr output assumes that too. But this file prints on one page, so I don't need to show the bottom of the file; it's just some blank lines.

Adding a left and right margin

Adding the top margin makes the document easier to read, but we can do better by adding space on the left and right of the printed page. This effectively adds a left and right margin to our document.

The first step is to use the fmt command to reformat the text file to a different width. Using fmt -w 70 reformats the text file to use lines that are 70 columns wide. We can add some blank space on the left of the document to create a left margin. Using pr -o 5 adds 5 spaces to the start of each line of the output. With the narrower text, we'll also have about 5 spaces in the right margin.

$ fmt -w 70 mit.txt | pr -o 5 | head 2022-06-24 18:35 Page 1 Copyright (c) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including

This is how Unix users printed plain text files. You can use the same set of commands to print text files on a modern laser printer, but your printer may expect a page feed command instead of using blank lines. To do that, add the -f option to the pr command, like this:

$ fmt -w 70 mit.txt | pr -f -o 5 | lpr

I'll omit the -f in the rest of this article, but remember to add -f to the pr command if you want to print documents to a modern laser printer.

Changing the header

You may notice that when we redirect the output of fmt to the pr command, the pr output no longer shows the name of the file. That's because when we chain several commands together like this, the pr command doesn't know the filename, so it's left blank. We can add the filename to the header by using the -h option:

$ fmt -w 70 mit.txt | pr -h 'mit.txt' -o 5 | head 2022-06-24 18:45 mit.txt Page 1 Copyright (c) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including

You can make other changes to the header, such as the -D option to change the date and time format, or replace it with new text.

$ fmt -w 70 mit.txt | pr -D '6/24/2022' -h 'mit.txt' -o 5 | head -30 6/24/2022 mit.txt Page 1 Copyright (c) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.Printing two columns

What if you wanted to make a text document look really fancy on the printed page? Certain documents such as technical articles might need to be printed in a two-column layout. The pr command can print text in multiple columns. For example, -2 prints in two columns and -3 will print in three columns.

However, be careful when printing text in multiple columns. If the lines are too long, pr may simply overlap one column with another, effectively losing text in the output. But we can leverage the fmt command to reformat the text to a narrower width, suitable for printing in two column format.

Let's do the math: If the printed page is 80 columns wide, and we've left 5 spaces on the left and right as page margins, that leaves 70 columns for our text. Using fmt -w 35 would cut the text neatly in half for two columns, but we may not leave much space between the columns. Instead, let's use fmt -w 33 to reformat the text width to 33 before sending the output to the pr command:

$ fmt -w 33 mit.txt | pr -2 -D '6/24/2022' -h 'mit.txt' -o 5 | head -30 6/24/2022 mit.txt Page 1 Copyright (c) substantial portions of the Software. Permission is hereby granted, free of charge, to any person THE SOFTWARE IS PROVIDED obtaining a copy of this "AS IS", WITHOUT WARRANTY OF software and associated ANY KIND, EXPRESS OR IMPLIED, documentation files (the INCLUDING BUT NOT LIMITED TO THE "Software"), to deal in the WARRANTIES OF MERCHANTABILITY, Software without restriction, FITNESS FOR A PARTICULAR PURPOSE including without limitation the AND NONINFRINGEMENT. IN NO rights to use, copy, modify, EVENT SHALL THE AUTHORS OR merge, publish, distribute, COPYRIGHT HOLDERS BE LIABLE sublicense, and/or sell copies FOR ANY CLAIM, DAMAGES OR OTHER of the Software, and to permit LIABILITY, WHETHER IN AN ACTION persons to whom the Software is OF CONTRACT, TORT OR OTHERWISE, furnished to do so, subject to ARISING FROM, OUT OF OR IN the following conditions: CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN The above copyright notice and THE SOFTWARE. this permission notice shall $

Unix is a great platform for processing text. While we use other tools today, including HTML in web browsers and PDF for printable content, it's nice to know how to use the existing Unix tools to create professional plain text documents.

The pr tool prepares text documents for printing.

Image by:

Internet Archive Book Images. Modified by Opensource.com. CC BY-SA 4.0

Linux What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.